Senior Data Engineer

NextGenEnergyJobs; Greater London; Hybrid; GBP 65,000 - 90,000; Full time

Job summary

NextGenEnergyJobs is seeking a full-time data engineer to establish the flexibility team's data capability and set its technical direction within an AWS-based environment. You will upskill data-literate backend engineers, act as the team's data expert, and drive scalable, reusable data pipelines and APIs using Django/FastAPI where appropriate.

You'll maintain and evolve the data warehouse with Databricks, champion DataOps practices, and ensure robust, globally scalable data applications across

Qualifications

• Proficient in SQL, Python and AWS data stack.
• Experience with Databricks data warehouse architecture.
• Familiar with Kubernetes, Docker, Spark and DataOps monitoring tools.

Responsibilities

• Establish the data capability of the flexibility team and set its technical direction.
• Upskill backend engineers and multiply their impact.
• Build greenfield data systems in an AWS environment.
• Provide a robust data processing and warehousing layer for current and future projects.
• Maintain and improve data applications for global scalability and operational efficiency.
• Collaborate with the data warehouse built on Databricks and drive platform stability.

Skills

SQL

Python

AWS

Django

FastAPI

Kubernetes

Docker

Spark

DataDog

Grafana

Prometheus

Airflow

dbt

Tools

Databricks

Job description

We're making energy fair, clean, and simple for all using technology.

Key Responsibilities

• Acting as the flexibility team’s first full-time data engineer, establish the team’s data capability, lead the team’s data engineering technical direction and represent the team to the cross-team data community
• Upskill data-literate backend engineers, act as the team's data expert and as a multiplier on other engineers' impact
• Establish greenfield data engineering systems in a newly established AWS environment
• Provide a foundational layer of high quality data processing and warehousing that will enable future projects for both our team and others
• Maintain and develop critical data applications - scoping and implementing architectural improvements for long-term global scalability, robustness and operational efficiency
• Interface with the established data warehouse, built on Databricks
• Take ownership of data platform improvements that enhance the capabilities for all teams and drive trust in the stability of the system

Requirements

• Strong aptitude with SQL, Python and AWS data stack essential;
• Experience building efficient, scalable databases and APIs (e.g. Django, FastAPI) a huge plus;
• Experience in Kubernetes, Docker, Spark and related monitoring tools (e.g. DataDog, Grafana, Prometheus) for DataOps a huge plus;
• Experience with Airflow a huge plus;
• Experience with dbt for pipeline modelling also beneficial;
• Skilled at shaping needs into a solid set of requirements and designing scalable solutions to meet them;
• Able to quickly understand new domain areas and visualise data effectively;
• Team player excited at the idea of ownership across lots of different projects and tools;
• Drives knowledge sharing and documentation for a more effective platform;