AWS Data Engineer

True North Consulting Group Ltd; City Of London; On-site; GBP 70,000 - 100,000; Full time

Job summary

True North Consulting Group Ltd is seeking a data-focused engineer to join our consultancy team in the UK. You will leverage Python and SQL to build data pipelines, orchestrated with Airflow, and containerised with Docker.

You will work with AWS services like Redshift, OpenSearch, Lambda, Glue, Step Functions, and Batch. The role requires hands-on experience with Git and GitLab for collaborative development, along with familiarity with CI/CD pipelines and YAML configurations.

Qualifications

• Proficient in Python and SQL for data tooling.
• Experience with Apache Airflow for DAG orchestration.
• Hands-on Docker containerisation experience.
• Proficient in AWS data services: Redshift, OpenSearch, Lambda, Glue, Step Functions, Batch.
• Familiar with CI/CD pipelines and YAML configurations (GitLab CI/CD).
• Proficient with Git and collaborative development using GitLab.
• Understanding of AWS security best practices, IAM policies and RBAC.

Skills

Python

SQL

Airflow

RBAC security

Tools

Docker

Git

GitLab

GitLab CI/CD

YAML

Redshift

OpenSearch

Lambda

Glue

Step Functions

Batch

Job description

Working with a mid sized consultancy with a lot of public sector engagements.

• Strong programming skills in Python and proficiency in SQL.
• Experience with Apache Airflow for DAG orchestration and monitoring.
• Hands-on experience with Docker for containerisation.
• Proficient in AWS data services:
• Redshift, OpenSearch, Lambda, Glue, Step Functions, Batch.
• Familiarity with CI/CD pipelines and YAML-based configuration (e.g., GitLab CI/CD).
• Proficient in Git and collaborative development using GitLab (or similar).
• Understanding of AWS security best practices, IAM policies, and RBAC.