Data Engineer, Commodities Data & AI Pipelines

Millennium; Greater London; On-site; GBP 70,000 - 110,000; Full time

Job summary

Millennium is seeking a data engineer to design and maintain end-to-end ETL pipelines for commodities data, using Python and SQL. You will build scalable data models, ensure data quality, and collaborate with portfolio managers and researchers to refine datasets according to evolving business needs.

The role emphasizes automated testing, Git-based deployments, and orchestration with Airflow, with a focus on accuracy, lineage, and documentation.

Qualifications

• 5+ years of experience in data engineering or analytics engineering building and maintaining ETL pipelines.
• Strong Python and SQL skills with large datasets and complex transformations.
• Hands-on experience with Airflow or comparable workflow orchestration tools.

Responsibilities

• Design, implement, and maintain end-to-end ETL workflows in Python and SQL for commodities data.
• Build and maintain standardized data models, schemas, and metadata.
• Schedule, monitor, and manage data pipelines using Airflow to ensure timely delivery.

Skills

Python

SQL

Airflow

Git

CI/CD

Tools

GitHub Actions

Job description

Millennium is seeking a data engineer to design and maintain end-to-end ETL pipelines for commodities data, using Python and SQL. You will build scalable data models, ensure data quality, and collaborate with portfolio managers and researchers to refine datasets according to evolving business needs.

The role emphasizes automated testing, Git-based deployments, and orchestration with Airflow, with a focus on accuracy, lineage, and documentation.