Senior Data Engineer

LTM; Sheffield; On-site; GBP 76,000 - 103,000; Full time

Job summary

LTM is seeking a Senior Data Engineer to design, build and maintain large-scale data pipelines and ETL workflows in a regulated enterprise environment.

The role is based in Sheffield, UK, for a 6-month contract with a rate of 65 GBP per hour inside IR35. You will ingest, transform and manage data assets for analytics platforms, collaborating with DevOps and cybersecurity teams while optimizing cloud-based data ingestion infrastructure.

Qualifications

• Experience with SRE principles and Azure DevOps.
• Strong scripting using Bash, PowerShell, Azure CLI.
• Programming experience with Python and Java, OO development.
• Strong SQL and KQL experience.
• GitHub source control and CI/CD pipelines.
• Linux administration and scripting experience.
• Networking, operating systems, storage, and infrastructure fundamentals.
• Experience in Data Acquisition and Cloud-Based Data Pipelines.
• Experience with Data Transport, Data Cleaning and Data Integration.
• Strong data engineering pipeline automation and productionization skills.
• ETL workflow design and large-scale data pipelines.
• Data warehousing, data modeling and data cataloguing knowledge.
• Real-Time Analytics and Streaming Data Platforms experience.
• Cloud Cost Optimization experience, Azure-focused.

Responsibilities

• Design, build and maintain largescale data pipelines and ETL workflows across enterprise datasets.
• Ingest, provision and manage raw, enriched and curated data assets for analytics platforms.
• Improve reliability, scalability and real-time data ingestion capabilities.
• Support and enhance cloud-based data ingestion infrastructure and platform services.
• Acquire, transform and integrate data from internal and external sources.
• Perform exploratory data analysis and onboard new data sources.
• Develop automated data engineering workflows and optimize data processing pipelines.
• Manage data modelling, data warehousing, data integration and data cataloguing activities.
• Support real-time analytics and streaming data platforms.
• Contribute to cloud infrastructure engineering, platform operations and production support.
• Implement cloud cost optimization and platform efficiency improvements.
• Collaborate with engineering, analytics, infrastructure and cybersecurity teams.
• Support DevOps automation, monitoring and operational excellence initiatives.
• Maintain documentation, governance standards and operational procedures.

Skills

SRE principles

Azure DevOps

Bash & PowerShell

Azure CLI

Python / Java

SQL / KQL

GitHub CI/CD

Linux administration

Networking fundamentals

Azure Data Factory

Azure Databricks

Kubernetes

Terraform

Spark Streaming

SIEM SOAR

Job description

• Design build and maintain largescale data pipelines and ETL workflows across enterprise datasets
• Ingest provision and manage raw enriched and curated data assets for analytics platforms
• Improve reliability scalability and realtime data ingestion capabilities
• Support and enhance cloudbased data ingestion infrastructure and platform services
• Acquire transport clean transform and integrate data from internal and external sources
• Perform exploratory data analysis and onboard new data sources
• Develop automated data engineering workflows and optimize data processing pipelines
• Manage data modelling data warehousing data integration and data cataloguing activities
• Support realtime analytics and streaming data platforms
• Contribute to cloud infrastructure engineering platform operations and production support
• Implement cloud cost optimization and platform efficiency improvements
• Collaborate with engineering analytics infrastructure and cybersecurity teams
• Support DevOps automation monitoring and operational excellence initiatives
• Maintain documentation governance standards and operational procedures

Role Description

Role Senior Data Engineer

Sheffield, UK

6 Months contract

Rate: 65 GBP per hour inside IR35

Responsibilities

• Design build and maintain largescale data pipelines and ETL workflows across enterprise datasets
• Ingest provision and manage raw enriched and curated data assets for analytics platforms
• Improve reliability scalability and realtime data ingestion capabilities
• Support and enhance cloudbased data ingestion infrastructure and platform services
• Acquire transport clean transform and integrate data from internal and external sources
• Perform exploratory data analysis and onboard new data sources
• Develop automated data engineering workflows and optimize data processing pipelines
• Manage data modelling data warehousing data integration and data cataloguing activities
• Support realtime analytics and streaming data platforms
• Contribute to cloud infrastructure engineering platform operations and production support
• Implement cloud cost optimization and platform efficiency improvements
• Collaborate with engineering analytics infrastructure and cybersecurity teams
• Support DevOps automation monitoring and operational excellence initiatives
• Maintain documentation governance standards and operational procedures

To be successful in this role you should have

• Experience with SRE principles and Azure DevOps
• Strong scripting skills using Bash PowerShell and Azure CLI
• Programming experience with Python C Java and objectoriented development
• Strong SQL and Kusto Query Language KQL experience
• Experience with GitHub source control systems and CICD pipelines
• Strong Linux administration and scripting experience
• Knowledge of networking operating systems storage technologies and infrastructure fundamentals
• Experience in Data Acquisition and CloudBased Data Pipelines
• Experience with Data Transport Data Cleaning and Data Integration
• Strong Data Engineering pipeline automation productionization and optimization skills
• Experience designing and maintaining ETL workflows and largescale data pipelines
• Knowledge of Data Warehousing Data Modeling and Data Cataloguing
• Experience with RealTime Analytics and Streaming Data Platforms
• Cloud Cost Optimization experience
• Strong Azure platform knowledge including Azure AD Azure IAM Networking Compute Storage Databases and Containers
• Experience with Azure Data Factory Azure Databricks Unity Catalog Azure Functions Azure Logic Apps Azure Monitor Azure Log Analytics Azure Data Lake Synapse Analytics and Power BI
• Experience with Kubernetes and CloudNative platforms
• Knowledge of Nginx Apache CosmosDB Linux Prometheus Grafana and Elasticsearch
• Experience with Infrastructure as Code tools such as Terraform ARM Chef and Ansible
• Experience with Azure Event Hubs Kafka and Spark Streaming
• Exposure to SIEM and SOAR platforms is advantageous
• Experience working in highly regulated enterprise environments is preferred