Data Engineer

JAM Recruitment; City Of London; Remote; GBP 60,000 - 80,000; Full time

Job summary

A recruitment agency is seeking an experienced eSC Cleared Data Engineer for a high-profile programme based in the UK, primarily working from home. The successful candidate will work with cutting-edge cloud technologies, building and optimizing data pipelines that support mission-critical systems. Candidates must hold enhanced security clearance and demonstrate strong skills in Python, SQL, and AWS among other technical proficiencies. This position offers competitive daily rates.

Qualifications

• Must hold live and transferrable Enhanced SC Security Clearance.
• Experience in building and optimising data pipelines and platforms.
• Ability to apply best practices in security and compliance.

Responsibilities

• Design and optimise ETL/ELT workflows using Apache Airflow.
• Create reliable ingestion processes from APIs and internal systems.
• Develop and maintain data lakes and warehouses.
• Implement automated validation and monitoring for data integrity.
• Enhance logging and fine-tune performance of workflows.
• Manage schema evolution and partitioning strategies.
• Integrate pipelines into robust CI/CD processes.
• Apply encryption and GDPR-aligned data practices.

Skills

Proficient in Python

Solid experience with Apache Airflow

Strong AWS skills (S3, Redshift)

Big data experience with Apache Spark

Knowledge of data modelling and schema design

Understanding of batch and streaming data architectures

Experience with Docker and/or Kubernetes

Familiarity with Microsoft SQL Server Stack

Job description

eSC Cleared Data Engineer

Bristol or London (Mostly working from home)

Up to £745 per day (Umbrella, Inside IR35)

Must hold live and transferrable Enhanced SC Security Clearance

About the Role

We're seeking an experienced Data Engineer to join a high-profile programme, working with cutting-edge cloud and data technologies. You'll be instrumental in building, optimising, and maintaining scalable data pipelines and platforms that underpin mission-critical systems.

If you thrive on solving complex data challenges, working with modern orchestration tools, and applying best practices in security and compliance, this role offers both technical depth and impact.

Key Responsibilities

• Design & Optimise Pipelines - Build and refine ETL/ELT workflows using Apache Airflow for orchestration.
• Data Ingestion - Create reliable ingestion processes from APIs and internal systems, leveraging tools such as Kafka, Spark, or AWS-native services.
• Cloud Data Platforms - Develop and maintain data lakes and warehouses (e.g., AWS S3, Redshift).
• Data Quality & Governance - Implement automated validation, testing, and monitoring for data integrity.
• Performance & Troubleshooting - Monitor workflows, enhance logging/alerting, and fine-tune performance.
• Data Modelling - Handle schema evolution, partitioning strategies, and efficient data structures.
• CI/CD for Data - Work with DevOps teams to integrate pipelines into robust CI/CD processes, managing version control for DAGs and configurations.
• Security & Compliance - Apply encryption, access control (IAM), and GDPR-aligned data practices.

Technical Skills & Experience

• Proficient in Python and SQL for data processing.
• Solid experience with Apache Airflow - writing and configuring DAGs.
• Strong AWS skills (S3, Redshift, etc.).
• Big data experience with Apache Spark.
• Knowledge of data modelling, schema design, and partitioning.
• Understanding of batch and streaming data architectures (e.g., Kafka).
• Experience with Docker and/or Kubernetes.
• Familiarity with Microsoft SQL Server Stack (SQL Server, SSIS, C#, T-SQL), Elastic/OpenSearch.

Interested?

If you're ready to bring your data engineering expertise to a technically challenging and rewarding programme, apply now with your CV.