Data Engineer

Octopus Energy Group; London; On-site; GBP 40,000 - 80,000; Full time

Job summary

An established industry player is seeking a Data Engineer to enhance their Energy Markets team's capabilities. This role involves designing and building key operational pipelines, maintaining critical data flows, and developing dashboards to support business needs. You will work with a variety of technologies, including SQL, Python, and Airflow, while collaborating with international teams to ensure best practices. If you're passionate about data engineering and eager to contribute to the Net Zero transition, this is an exciting opportunity to make a significant impact in a dynamic environment.

Qualifications

• Strong aptitude with SQL and Python for data engineering tasks.
• Experience with Kubernetes and Docker for container management.

Responsibilities

• Design and build operational and reporting pipelines for energy markets.
• Develop automations and alerts to maintain critical data pipelines.
• Create dashboards for operational processes and reporting needs.

Skills

SQL

Python

Airflow

Kubernetes

Docker

Django

Spark

DevOps Monitoring Tools

Tools

Databricks

Jupyter Notebooks

Pyspark

Streamlit

Grafana

Prometheus

Hugo

Markdown

Job description

The Energy Markets team at Octopus Energy is responsible for making sure that we always have the electricity and gas we need to support our customer demand whilst also supporting the grid to enable the Net Zero transition.

To achieve this mission across all Octopus international regions, we have sub-teams focused on forecasting energy demand and generation, hedging and shaping our trade position, tracking and reporting the ongoing risk to Octopus, and driving the proportion of our supply directly from generators via PPA agreements.

The Engineering sub-team owns our global technical platform that supports these different processes and drives forward long-term solutions to enhance Group capabilities.

We are looking for a Data Engineer to help achieve this goal - ideally someone who is comfortable diving into different tasks to support each team using a variety of coding languages across our platform setup, who enjoys developing relationships across the company while explaining technical processes in the most appropriate way, and who keeps an eye on scalable solutions to support data growth.

This is therefore an exciting opportunity to take on a role that combines complex data engineering, visual analytics and business critical need.

What you'll do...

• Supporting different Energy Markets teams to design and build key operational and reporting pipelines across all Octopus Energy regions;
• Taking responsibility for the maintenance of these critical data pipelines supporting core trading, forecasting, risk and PPA processes;
• Developing automations and alerts to quickly debug where these pipelines are failing or showing unprecedented trends;
• Setting up and maintaining processes for capturing, preparing and loading valuable new data into the data lake;
• Designing and building dashboards that cover operational processes and reporting requirements;
• Working with international teams across the Octopus Energy Group to ensure everyone shares the best possible practices and code is standardized where possible;
• Taking ownership of data platform improvements that enhance the capabilities for all Energy Markets teams and drives trust in the stability of the setup;
• Sharing, enhancing and upskilling team members on available tools and best practices.

What you'll need...

• Strong aptitude with SQL, Python and Airflow;
• Experience in Kubernetes, Docker, Django, Spark and related monitoring tools for DevOps a big plus (e.g. Grafana, Prometheus);
• Experience with dbt for pipeline modeling also beneficial;
• Skilled at shaping needs into a solid set of requirements and designing scalable solutions to meet them;
• Able to quickly understand new domain areas and visualize data effectively;
• Team player excited at the idea of ownership across lots of different projects and tools; Passion for driving towards Net Zero;
• Drives knowledge sharing and documentation for a more effective platform;
• Open to traveling to Octopus offices across Europe and the US.

Our Data Stack:

• SQL-based pipelines built with dbt on Databricks
• Analysis via Python Jupyter notebooks
• Pyspark in Databricks workflows for heavy lifting
• Streamlit and Python for dashboarding
• Airflow DAGs with Python for ETL running on Kubernetes and Docker
• Django for custom app/database development
• Kubernetes for container management, with Grafana/Prometheus for monitoring
• Hugo/Markdown for data documentation