Senior Data Engineer - AWS Spark ETL for AI/ML

Ports North; Birmingham; On-site; GBP 76,000 - 99,000; Full time

Job summary

Ports North in Birmingham is seeking a Senior Data Engineer to design, build and optimise large-scale data pipelines in an AWS environment, and to support AI/ML and Generative AI initiatives with scalable data platforms.

You will develop high-throughput Spark pipelines, ETL/ELT workflows for diverse datasets, and Java or Scala components with advanced SQL, contributing to lakehouse architectures and secure data governance.

Qualifications

• 5+ years of commercial Data Engineering experience.
• Strong hands-on Apache Spark development experience.
• Proficient in Java and/or Scala programming.
• Extensive AWS experience with S3, EMR, Glue and IAM.
• Advanced SQL development for large-scale datasets.
• Experience building ETL/ELT pipelines.
• Experience with data lake or lakehouse architectures.

Responsibilities

• Design, build and maintain high-throughput data pipelines using Apache Spark and cloud-native technologies.
• Develop scalable ETL/ELT workflows for structured and unstructured datasets.
• Build and optimise data-processing solutions within AWS environments.
• Develop performant solutions using Java or Scala alongside advanced SQL.
• Support modern data lake and lakehouse architectures.
• Work alongside AI/ML teams to provide data platforms supporting Generative AI, machine learning and advanced analytics use cases.
• Implement appropriate data quality, governance, security and monitoring standards.
• Troubleshoot complex data-processing and distributed-system issues within production environments.

Skills

Apache Spark

Java/Scala

AWS (S3/EMR/Glue/IAM)

SQL

ETL/ELT pipelines

Data governance & security

Distributed data processing

Tools

Spark

Job description

Ports North in Birmingham is seeking a Senior Data Engineer to design, build and optimise large-scale data pipelines in an AWS environment, and to support AI/ML and Generative AI initiatives with scalable data platforms.

You will develop high-throughput Spark pipelines, ETL/ELT workflows for diverse datasets, and Java or Scala components with advanced SQL, contributing to lakehouse architectures and secure data governance.