SC-Cleared AWS Data Engineer: Cloud Data Pipelines

True North Consulting Group Ltd; City Of London; On-site; GBP 70,000 - 110,000; Full time

Job summary

True North Consulting Group Ltd in the United Kingdom is seeking an experienced AWS Data Engineer with active SC clearance to design, build and maintain scalable data pipelines and infrastructure on AWS.

You will work across data orchestration, automation and containerisation using Python, SQL, Airflow, Docker and modern CI/CD practices, collaborating with cross-functional teams to deliver secure, high-performance data solutions.

Qualifications

• Strong hands-on experience in AWS Data Engineering.
• Excellent Python programming skills and strong SQL proficiency.
• Experience using Apache Airflow for workflow orchestration and DAG management.
• Practical experience with Docker and containerisation.
• Strong knowledge of AWS data services, particularly Redshift, Glue, Lambda, Step Functions, Batch and OpenSearch.
• Experience with CI/CD pipelines, Git and YAML-based configurations.
• Good understanding of AWS security, IAM policies and role-based access control (RBAC).
• Strong problem-solving skills and the ability to troubleshoot complex data engineering issues.
• Experience working in Agile environments and collaborating with cross-functional teams.
• Good communication and stakeholder management skills.

Responsibilities

• Design, develop and maintain scalable data pipelines using Python, SQL and Apache Airflow.
• Build and optimise cloud-based data solutions using AWS services including Redshift, Glue, Lambda, Step Functions, Batch and OpenSearch.
• Develop and manage containerised applications and workflows using Docker.
• Implement and maintain data orchestration processes, ensuring efficient and reliable data processing.
• Collaborate with engineering teams and stakeholders to deliver secure, scalable data infrastructure.
• Monitor, troubleshoot and optimise data pipeline performance and reliability.
• Work with GitLab or similar tools for version control and CI/CD.
• Ensure data quality, governance and security best practices are followed.
• Contribute to technical documentation, automation and continuous improvement initiatives.

Skills

AWS Data Engineering

Python programming

SQL proficiency

Airflow

Docker

CI/CD pipelines

Git

YAML configurations

IAM RBAC

Agile experience

Tools

GitLab

Job description

True North Consulting Group Ltd in the United Kingdom is seeking an experienced AWS Data Engineer with active SC clearance to design, build and maintain scalable data pipelines and infrastructure on AWS.

You will work across data orchestration, automation and containerisation using Python, SQL, Airflow, Docker and modern CI/CD practices, collaborating with cross-functional teams to deliver secure, high-performance data solutions.