Backend Engineer - Green HPC / Python Data Pipelines

EMBL; Hinxton; Hybrid; GBP 52,000 - 62,000; Full time

Benefits offered by this job

Flexible working arrangements

30 days annual leave

Job summary

EMBL-EBI is seeking an experienced backend engineer to advance sustainability tooling for high-performance computing. You will process HPC telemetry and energy data, contribute to open-source GA collaborations, and enhance real-time monitoring and Grafana dashboards.

You will work with PostgreSQL/TimescaleDB, Kafka, and pandas, delivering production-grade data pipelines while collaborating across EMBL-EBI and Cambridge teams.

Qualifications

• Proven production-grade Python development with data pipelines.
• Strong PostgreSQL/TimescaleDB experience or exposure.
• Experience with Kafka messaging and real-time data processing.
• Familiarity with pandas DataFrames for data analysis.
• Proficient with Git, pytest, ruff and Poetry.

Responsibilities

• Develop backend services processing HPC telemetry and energy metrics.
• Collaborate with Green Algorithms team to integrate monitoring into GA codebase.
• Work with time-series databases for large-scale telemetry storage/retention.
• Improve carbon and energy reporting tooling and dashboards.
• Contribute to documentation and support two independent teams.

Skills

Python backend

Data pipelines

Pandas

Git

CI/CD

Concurrency

Communication

AI-assisted tools

Tools

PostgreSQL

TimescaleDB

psycopg2/psycopg3

Kafka

Grafana

Python tooling (ruff, pytest, Poetry)

Job description

EMBL-EBI is seeking an experienced backend engineer to advance sustainability tooling for high-performance computing. You will process HPC telemetry and energy data, contribute to open-source GA collaborations, and enhance real-time monitoring and Grafana dashboards.

You will work with PostgreSQL/TimescaleDB, Kafka, and pandas, delivering production-grade data pipelines while collaborating across EMBL-EBI and Cambridge teams.