Data Engineer

Ports North; Birmingham; Hybrid; GBP 76,000 - 99,000; Full time

Job summary

Ports North in Birmingham is seeking a Senior Data Engineer to design, build and optimise large-scale data pipelines in an AWS environment, and to support AI/ML and Generative AI initiatives with scalable data platforms.

You will develop high-throughput Spark pipelines, ETL/ELT workflows for diverse datasets, and Java or Scala components with advanced SQL, contributing to lakehouse architectures and secure data governance.

Qualifications

• 5+ years of commercial Data Engineering experience.
• Strong hands-on Apache Spark development experience.
• Proficient in Java and/or Scala programming.
• Extensive AWS experience with S3, EMR, Glue and IAM.
• Advanced SQL development for large-scale datasets.
• Experience building ETL/ELT pipelines.
• Experience with data lake or lakehouse architectures.

Responsibilities

• Design, build and maintain high-throughput data pipelines using Apache Spark and cloud-native technologies.
• Develop scalable ETL/ELT workflows for structured and unstructured datasets.
• Build and optimise data-processing solutions within AWS environments.
• Develop performant solutions using Java or Scala alongside advanced SQL.
• Support modern data lake and lakehouse architectures.
• Work alongside AI/ML teams to provide data platforms supporting Generative AI, machine learning and advanced analytics use cases.
• Implement appropriate data quality, governance, security and monitoring standards.
• Troubleshoot complex data-processing and distributed-system issues within production environments.

Skills

Apache Spark

Java/Scala

AWS (S3/EMR/Glue/IAM)

SQL

ETL/ELT pipelines

Data governance & security

Distributed data processing

Tools

Spark

Job description

Contract: Initial 12 months
Rate: Up to £475 per day - Inside IR35
Location: Birmingham
Working Pattern: Hybrid - 3 days per week onsite at Birmingham office

We are looking for an experienced Senior Data Engineer to join a major enterprise technology programme within the financial services sector. You will be responsible for designing, building and optimising large-scale data pipelines and distributed processing solutions within an AWS environment. The role will also support modern AI/ML and Generative AI initiatives by developing scalable, secure and reliable data platforms.

Key Responsibilities

• Design, build and maintain high-throughput data pipelines using Apache Spark and cloud-native technologies.
• Develop scalable ETL/ELT workflows for structured and unstructured datasets.
• Build and optimise data-processing solutions within AWS environments.
• Develop performant solutions using Java or Scala alongside advanced SQL.
• Support modern data lake and lakehouse architectures.
• Work alongside AI/ML teams to provide data platforms supporting Generative AI, machine learning and advanced analytics use cases.
• Implement appropriate data quality, governance, security and monitoring standards.
• Troubleshoot complex data-processing and distributed-system issues within production environments.

Essential Experience

• 5+ years of commercial Data Engineering experience.
• Strong hands-on Apache Spark development experience.
• Strong programming experience with Java and/or Scala.
• Strong AWS experience, ideally including technologies such as S3, EMR, Glue and IAM.
• Advanced SQL development skills.
• Strong experience designing and developing ETL/ELT pipelines.
• Experience working with large-scale and distributed data-processing environments.
• Databricks and/or Snowflake.