Data Engineer, Commodities Data & AI Pipelines

Millennium

Greater London

On-site

GBP 70,000 - 110,000

Full time

14 days+

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Job summary

Millennium is seeking a data engineer to design and maintain end-to-end ETL pipelines for commodities data, using Python and SQL. You will build scalable data models, ensure data quality, and collaborate with portfolio managers and researchers to refine datasets according to evolving business needs.

The role emphasizes automated testing, Git-based deployments, and orchestration with Airflow, with a focus on accuracy, lineage, and documentation.

Qualifications

  • 5+ years of experience in data engineering or analytics engineering building and maintaining ETL pipelines.
  • Strong Python and SQL skills with large datasets and complex transformations.
  • Hands-on experience with Airflow or comparable workflow orchestration tools.

Responsibilities

  • Design, implement, and maintain end-to-end ETL workflows in Python and SQL for commodities data.
  • Build and maintain standardized data models, schemas, and metadata.
  • Schedule, monitor, and manage data pipelines using Airflow to ensure timely delivery.

Skills

Python
SQL
Airflow
Git
CI/CD

Tools

GitHub Actions

Job description

Millennium is seeking a data engineer to design and maintain end-to-end ETL pipelines for commodities data, using Python and SQL. You will build scalable data models, ensure data quality, and collaborate with portfolio managers and researchers to refine datasets according to evolving business needs.

The role emphasizes automated testing, Git-based deployments, and orchestration with Airflow, with a focus on accuracy, lineage, and documentation.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Data Engineer - Commodities
Data Engineer - Commodities

Millennium • Greater London

On-site
GBP 70,000 - 110,000
Python Engineer for Rapid Analytics in Commodities
Python Engineer for Rapid Analytics in Commodities

Millennium Management • Greater London

On-site
GBP 90,000 - 140,000
Data Pipelines Engineer - Commodities Analytics
Data Pipelines Engineer - Commodities Analytics

DRW • Greater London

On-site
GBP 65,000 - 95,000
Data Pipeline Software Engineer - Commodities (Python)
Data Pipeline Software Engineer - Commodities (Python)

DRW Holdings, LLC. • Greater London

Hybrid
GBP 70,000 - 110,000
Software Engineer - Rapid Application Development - Commodities
Software Engineer - Rapid Application Development - Commodities

Millennium Management • Greater London

On-site
GBP 90,000 - 140,000
Software Engineer - Market Data
Software Engineer - Market Data

Millennium • Greater London

On-site
GBP 110,000 - 150,000
Senior Data Engineer: Real-Time Pipelines & Dashboards
Senior Data Engineer: Real-Time Pipelines & Dashboards

EDF • Greater London

Hybrid
GBP 90,000 - 130,000
Data Engineer: Build Reliable Data Pipelines & Drive Impact
Data Engineer: Build Reliable Data Pipelines & Drive Impact

Marsbridge Capital Limited • Greater London

Hybrid
GBP 60,000 - 100,000
Data Engineer, Financial Data Platform & Pipelines
Data Engineer, Financial Data Platform & Pipelines

Engineersgate • Greater London

On-site
GBP 65,000 - 90,000
Market Data Software Engineer — C++ & AI Workflows
Market Data Software Engineer — C++ & AI Workflows

Millennium • Greater London

On-site
GBP 110,000 - 150,000