Senior Data Platform Engineer

NCC Group; Manchester; On-site; GBP 65,000 - 85,000; Full time

Job summary

A leading data engineering firm in the United Kingdom seeks a Senior Data Platform Engineer responsible for designing, building, and optimising scalable data pipelines. This role involves close collaboration with Data Science teams and utilizing AWS technologies, including EMR and SageMaker. Ideal candidates will have proven experience in data engineering, strong Python skills, and a solid understanding of data governance principles. The position offers opportunities for innovation and career growth.

Qualifications

• Strong experience in data engineering within AWS cloud environments.
• Hands-on experience with AWS big data technologies such as EMR, S3 and SageMaker.
• Proficiency in Python for building scalable data pipelines and processing frameworks.
• Experience with Apache Spark for distributed data processing.
• Experience designing and maintaining scalable batch and real-time data pipelines.
• Solid understanding of ETL/ELT design patterns and data modelling techniques.
• Experience with workflow orchestration tools such as Apache Airflow (ideally deployed on AWS).
• Familiarity with containerisation and orchestration using Docker and Kubernetes (EKS).
• Experience with infrastructure as code (e.g. Terraform) and CI/CD/GitOps practices.
• Proven ability to optimise performance and reduce cloud costs through partitioning, clustering and workload management.
• Understanding of data security principles, including data loss prevention (DLP).

Responsibilities

• Design and build scalable data pipelines for advanced analytics.
• Deliver high-quality, reliable datasets to analytics platforms.
• Collaborate with Data Science team to develop robust data pipelines.
• Leverage AWS technologies to process and orchestrate datasets.
• Embed data loss prevention principles into data pipelines.

Skills

Data engineering in AWS cloud environments

AWS big data technologies (EMR, S3, SageMaker)

Proficiency in Python

Apache Spark

ETL/ELT design patterns

Apache Airflow

Containerization and orchestration (Docker, Kubernetes)

Infrastructure as code (Terraform)

Performance optimization

Data security principles (DLP)

Tools

Amazon SageMaker

Databricks

Kafka

Kinesis

Job description

Senior Data Platform Engineer

Manchester, Cheltenham or London

We are seeking a skilled Data Engineer to join our Engineering team, responsible for designing, building, and optimising scalable data pipelines that power advanced analytics and machine learning solutions.

You will play a key role in enabling data-driven decision-making by delivering high-quality, reliable datasets to tools such as Amazon SageMaker and other analytics platforms.

We are looking for a Data Engineer to work closely with the Data Science team to develop robust data pipelines that feed analytics and machine learning tools such as Amazon SageMaker and third-party platforms like Databricks.

You will leverage AWS technologies such as EMR, S3, EKS and Airflow to process and orchestrate high-volume datasets, ensuring solutions are scalable, resilient and cost-efficient.

You will also play a key role in embedding data loss prevention (DLP) principles and controls into data pipelines to protect sensitive information, while ensuring data is reliable, accessible, well-governed and optimised for downstream consumption.

What we are looking for in you: Essential

• Strong experience in data engineering within AWS cloud environments.
• Hands-on experience with AWS big data technologies such as EMR, S3 and SageMaker.
• Proficiency in Python for building scalable data pipelines and processing frameworks.
• Experience with Apache Spark for distributed data processing.
• Experience designing and maintaining scalable batch and real-time data pipelines.
• Solid understanding of ETL/ELT design patterns and data modelling techniques.
• Experience with workflow orchestration tools such as Apache Airflow (ideally deployed on AWS).
• Familiarity with containerisation and orchestration using Docker and Kubernetes (EKS).
• Experience with infrastructure as code (e.g. Terraform) and CI/CD/GitOps practices.
• Proven ability to optimise performance and reduce cloud costs through partitioning, clustering and workload management.
• Understanding of data security principles, including data loss prevention (DLP).

Desirable

• Experience with Databricks or similar third-party big data platforms.
• Knowledge of real-time streaming technologies (e.g. Kafka, Kinesis).
• Experience implementing data governance and compliance frameworks.
• Familiarity with monitoring and observability tools in AWS environments.
• Exposure to Lakehouse or modern data platform architectures.

Jobs are provided by the Find a Job Service from the Department for Work and Pensions (DWP).