Sustainable HPC Data Engineer – Python & Open Source

European Molecular Biology Laboratory; Cambridge; Hybrid; GBP 52,000 - 72,000; Full time

Benefits offered by this job

Hybrid working patterns

30 days annual leave

Campus shuttle and facilities

On-site nursery support

Job summary

EMBL-EBI is seeking a backend data engineer to join the HPC sub team and help measure, report, and reduce environmental impact of scientific computing. You will develop Python services for HPC telemetry, collaborate with the Green Algorithms team at Cambridge, and contribute to Grafana dashboards and real-time monitoring.

This is a hybrid role based in Cambridge/Biomed campus, with a strong focus on sustainability.

Qualifications

• Production-grade Python with data pipelines and maintainable code.
• Experience with time-series data and PostgreSQL.
• Familiarity with Kafka-based real-time pipelines.
• Proficient in data transformation using Pandas.
• Open-source contribution and CI/CD familiarity.

Responsibilities

• Develop and maintain Python backend services for HPC telemetry.
• Process energy and carbon footprint metrics per job.
• Collaborate with Cambridge GA team to integrate monitoring into GA codebase.
• Work with time series DBs for large-scale telemetry data.
• Enhance carbon and energy reporting tooling and dashboards.
• Produce technical docs and share knowledge across teams.

Skills

Python

Data pipelines

Concurrency

Markdown/Documentation

Pandas

Git

pytest

ruff

Poetry

AI-assisted development

Tools

PostgreSQL

TimescaleDB

Kafka

Grafana

Pandas (lib)

Git tooling

Job description

EMBL-EBI is seeking a backend data engineer to join the HPC sub team and help measure, report, and reduce environmental impact of scientific computing. You will develop Python services for HPC telemetry, collaborate with the Green Algorithms team at Cambridge, and contribute to Grafana dashboards and real-time monitoring.

This is a hybrid role based in Cambridge/Biomed campus, with a strong focus on sustainability.