SC Cleared Data Engineer

Damia Group; London; Remote; GBP 100,000 - 125,000; Full time

Job summary

An innovative global IT consultancy is seeking an SC Cleared Data Engineer to join their team. This role involves working on data migration and AWS ETL applications, with a focus on PostgreSQL and Python. The ideal candidate will have experience with AWS Glue and be adept at building and maintaining ETL pipelines. You will collaborate with cross-functional teams to ensure effective data handling and logging mechanisms. This position offers the chance to work remotely with occasional travel to London, providing a dynamic work environment where your contributions will directly impact critical projects in the Central Government and Policing Sector.

Qualifications

• Experience in data migration and AWS ETL applications.
• Strong knowledge of PostgreSQL and SQL for data mapping.

Responsibilities

• Build and maintain AWS Glue jobs as part of ETL pipelines.
• Ensure proper audit and logging mechanisms in ETL processes.

Skills

Data migration

AWS

PostgreSQL

SQL

Python

Soft skills

Tools

AWS Glue

AWS RDS

AWS Lambda

AWS S3

AWS Step Functions

Terraform

AWS CloudFormation

Job description

SC Cleared Data Engineer - 12 months+ £600-675pd Inside IR35- Remote with occasional on site travel to London

My client is a Global IT Consultancy looking for an SC Cleared Data Engineer. Candidates ideally have experience working within the Central Gov/Policing Sector.

Essential experience:

• Data migration experience.
• AWS experience, particularly using ETL applications such as AWS Glue and storage applications such as AWS RDS. Other tools include AWS Lambda, AWS S3, AWS Step Functions, and more.
• Excellent knowledge of PostgreSQL databases and SQL, as well as Python for data mapping and manually implementing AWS Glue Jobs.
• Building, maintaining, and refining AWS Glue jobs as part of an ETL pipeline.
• Soft skills, including pro-activeness and working with other teams such as DevOps and Database Analysts.
• Ensuring proper audit and logging mechanisms in ETL pipelines, particularly in AWS.

Ideal to have:

• AWS Database Migration experience (including its Change Data Capture ability).
• Experience handling queries related to the data model (data mapping, extensive SQL knowledge).
• Experience deploying ETL pipelines into AWS environments, such as using Terraform scripts or AWS CloudFormation.
• Handling migration of BLOBs for images and CLOBs (Oracle database or migration experience).
• Data Mapping experience.
• End-to-End Pipeline Execution: Supporting the execution of end-to-end pipelines using tools such as AWS Step Functions.
• Experience using APIs to load data into a target system as part of an ETL pipeline (this is a nice to have but not expected).