AWS Data Engineer | Airflow, Redshift & Secure Data Flows

True North Consulting Group Ltd; City Of London; On-site; GBP 70,000 - 100,000; Full time

Job summary

True North Consulting Group Ltd is seeking a data-focused engineer to join our consultancy team in the UK. You will leverage Python and SQL to build data pipelines, orchestrated with Airflow, and containerised with Docker.

You will work with AWS services like Redshift, OpenSearch, Lambda, Glue, Step Functions, and Batch. The role requires hands-on experience with Git and GitLab for collaborative development, along with familiarity with CI/CD pipelines and YAML configurations.

Qualifications

• Proficient in Python and SQL for data tooling.
• Experience with Apache Airflow for DAG orchestration.
• Hands-on Docker containerisation experience.
• Proficient in AWS data services: Redshift, OpenSearch, Lambda, Glue, Step Functions, Batch.
• Familiar with CI/CD pipelines and YAML configurations (GitLab CI/CD).
• Proficient with Git and collaborative development using GitLab.
• Understanding of AWS security best practices, IAM policies and RBAC.

Skills

Python

SQL

Airflow

RBAC security

Tools

Docker

Git

GitLab

GitLab CI/CD

YAML

Redshift

OpenSearch

Lambda

Glue

Step Functions

Batch

Job description

True North Consulting Group Ltd is seeking a data-focused engineer to join our consultancy team in the UK. You will leverage Python and SQL to build data pipelines, orchestrated with Airflow, and containerised with Docker.

You will work with AWS services like Redshift, OpenSearch, Lambda, Glue, Step Functions, and Batch. The role requires hands-on experience with Git and GitLab for collaborative development, along with familiarity with CI/CD pipelines and YAML configurations.