Data Engineer (SC cleared)

Stott and May; Guildford; Remote; GBP 50,000 - 70,000; Full time

Job summary

A technology consultancy is seeking a Data Engineer to design and implement ETL/ELT data pipelines with a focus on AWS, Apache Airflow, and data governance. The role is mostly remote but requires access to London or Bristol. Key responsibilities include building ingestion processes and managing CI/CD for data deployments. Familiarity with Python, SQL, and big data processing is essential.

Qualifications

• Experience designing and implementing ETL/ELT data pipelines.
• Proficient in Python and SQL for data processing.
• Familiarity with AWS services and big data frameworks.

Responsibilities

• Design and implement robust ETL/ELT data pipelines using Apache Airflow.
• Develop and maintain data lakes and warehouses on AWS.
• Collaborate with DevOps on CI/CD pipelines for data deployments.

Skills

Python

SQL

Apache Airflow

AWS services (S3, Redshift)

Apache Spark

Docker

Kubernetes

Job description

Data Engineer (SC cleared)

Start: ASAP
Duration: 12 months
Location: Mostly Remote - must have access to London or Bristol
Pay: negotiable, INSIDE IR35

Responsibilities:
- Design, implement robust ETL/ELT data pipelines using Apache Airflow
- Build ingestion processes from internal systems and APIs, using Kafka, Spark, AWS
- Develop and maintain data lakes and warehouses (AWS S3, Redshift)
- Ensuring governance using automated testing tools
- Collaborate with DevOps to manage CI/CD pipelines for data deployments and ensure version control of DAGs
- Apply best practice in security and compliance

Required Tech Skills:
- Python and SQL for processing
- Apache Airflow, writing Airflow DAGs and configuring airflow jobs
- AWS cloud platform and services like S3, Redshift
- Familiarity with big data processing using Apache Spark
- Knowledge of modelling, schema design and partitioning strategies
- Understanding batch Vs streaming data paradigms
- Docker or Kubernetes (containerization)