Data Engineer: Pipelines, ML Infra & Decarbonisation Impact

Octopus Group; Greater London; On-site; GBP 60,000 - 80,000; Full time

Job summary

A progressive technology firm based in Greater London is seeking a Data Engineer to build and optimise platforms for data transformation and insights. The ideal candidate will have a passion for decarbonisation and extensive experience in distributed data processing, monitoring, and building robust data pipelines. Key technologies include Airflow, AWS, and Kubernetes. The position offers an inclusive work environment and the opportunity to shape a growing team focused on innovative AI solutions.

Qualifications

• A passion for decarbonisation.
• Experience with distributed data processing.
• Experience with monitoring, testing, and data quality.
• Ability to work with ambiguity and own problems end-to-end.
• Experience building robust pipelines from diverse sources such as SQL & No-SQL databases and API endpoints.
• Familiarity with tools like Airflow, AWS, Kubernetes, Spark, dbt, Terraform.

Responsibilities

• Build and optimise the platform for transforming raw data into business insights.
• Develop and deploy ML infrastructure for AI requirements.
• Collaborate with dev teams for seamless integrations.
• Shape team direction and coach team members.

Skills

Passion for decarbonisation

High quality code writing

Distributed data processing

Monitoring and testing

Problem-solving

Building robust pipelines

Airflow

AWS

Kubernetes

Spark

dbt

Terraform

Databricks

MLOps implementation

Distributed system optimization

Governance

Job description

A progressive technology firm based in Greater London is seeking a Data Engineer to build and optimise platforms for data transformation and insights. The ideal candidate will have a passion for decarbonisation and extensive experience in distributed data processing, monitoring, and building robust data pipelines. Key technologies include Airflow, AWS, and Kubernetes. The position offers an inclusive work environment and the opportunity to shape a growing team focused on innovative AI solutions.