Apache Data Engineer (SC Cleared)

Montash Limited; Greater London; In loco; GBP 70.000 - 100.000; Tempo pieno

Descrizione del lavoro

Montash Limited is seeking a data engineer to design and build scalable data pipelines and ETL processes. You will implement change data capture, develop pipeline data models and ensure security, performance and proper metadata labelling within a Lakehouse framework.

You'll work across cloud AWS and on-premises platforms, contribute to sprint delivery, and write comprehensive unit and integration tests while using Git-based CI/CD tooling as part of an integrated team.

Competenze

• Experience building data pipelines within Lakehouse architectures.
• Strong knowledge of ETL, data modelling and data governance.
• Experience with Java or Groovy and object‑oriented design.
• Familiarity with CI/CD tooling and Git based workflows.

Mansioni

• Design and build data pipelines and ETL processes into a Lakehouse architecture across cloud (AWS) and on premises platforms
• Implement change data capture and event-driven patterns to ingest data from source systems
• Develop data models for the pipeline layer
• Handle non-functional requirements including security, performance and metadata labelling
• Write unit and integration tests for pipeline components
• Work within a defined SDLC using Git-based CI/CD tooling
• Contribute to sprint delivery as part of an integrated development team

Conoscenze

Data pipelines

ETL

Lakehouse

Java

Groovy

CI/CD

Unit testing

Strumenti

Apache NiFi

AWS

EMR

S3

SQS

Linux

PostgreSQL/PLSQL

Descrizione del lavoro

• Design and build data pipelines and ETL processes into a Lakehouse architecture across cloud (AWS) and on premises platforms
• Implement change data capture and event-driven patterns to ingest data from source systems
• Develop data models for the pipeline layer
• Handle non-functional requirements including security, performance and metadata labelling
• Write unit and integration tests for pipeline components
• Work within a defined SDLC using Git-based CI/CD tooling
• Contribute to sprint delivery as part of an integrated development team
  Experience building data pipelines within Lakehouse architectures
• Comfortable with Apache tooling, for data engineering work
• Experience with event sourcing, change data capture and event-driven architectures
• Java and/or Groovy, with a solid grasp of object-oriented design principles
• Experience working in an SDLC with an integrated team on CI/CD tooling (Git)

Nice-to-have

• Apache NiFi
• AWS, particularly EMR, S3 and SQS
• Awareness of domain-driven design
• Linux and Scripting
• Postgres or PL/SQL