Python Backend Engineer for Sustainable HPC Metrics

1000 European Molecular Biology Laboratory; Hinxton; Hybrid; GBP 35,000 - 42,000; Full time

Benefits offered by this job

Flexible working

30 days annual leave

Shuttle bus

On-site library

Gym and cafeteria

Casual dress code

Sports & social clubs

Job summary

EMBL-EBI is seeking a backend data engineer to join the HPC sustainability initiative. You will develop Python-based services to process HPC telemetry and energy metrics, collaborate with the Green Algorithms team, and extend reporting tooling.

This role requires strong proficiency in data pipelines, time-series databases, and open-source collaboration. You will work across two teams (EBI and Cambridge) to translate trade-offs into proposals, build consensus, and contribute to open-source code

Qualifications

• Strong experience writing production-grade Python for data pipelines and backend services.
• Solid experience with PostgreSQL and time-series data management.
• Familiarity with TimescaleDB including continuous aggregates is an advantage; comfortable with psycopg2/psycopg3.
• Working knowledge of Apache Kafka as a message broker and real-time pipeline design.
• Proficiency with pandas DataFrames for data transformation and analysis.

Responsibilities

• Develop and maintain Python-based backend services that process HPC job telemetry and energy metrics.
• Contribute to the Green Algorithms open source project with Cambridge GA team.
• Work with time series databases for scalable telemetry storage, aggregation, and retrieval.
• Extend carbon and energy reporting tooling and align with regional methodologies.
• Support Grafana dashboards for environmental impact visibility.
• Contribute to documentation and cross-team collaboration including the E-SCOUT trial.
• Produce clear technical documentation and share knowledge within the team.

Skills

Python

PostgreSQL

Data pipelines

Pandas

CI/CD

Psycopg2 / Psycopg3

Apache Kafka

Timeseries data

Git

pytest

ruff

Poetry

AI-assisted tooling

Open-source collaboration

Tools

PostgreSQL

TimescaleDB

Apache Kafka

Git

pytest

ruff

Poetry

Job description

EMBL-EBI is seeking a backend data engineer to join the HPC sustainability initiative. You will develop Python-based services to process HPC telemetry and energy metrics, collaborate with the Green Algorithms team, and extend reporting tooling.

This role requires strong proficiency in data pipelines, time-series databases, and open-source collaboration. You will work across two teams (EBI and Cambridge) to translate trade-offs into proposals, build consensus, and contribute to open-source code