Data Engineer

Octopus Group; Greater London; On-site; GBP 60,000 - 80,000; Full time

Job summary

A progressive technology firm based in Greater London is seeking a Data Engineer to build and optimise platforms for data transformation and insights. The ideal candidate will have a passion for decarbonisation and extensive experience in distributed data processing, monitoring, and building robust data pipelines. Key technologies include Airflow, AWS, and Kubernetes. The position offers an inclusive work environment and the opportunity to shape a growing team focused on innovative AI solutions.

Qualifications

• A passion for decarbonisation.
• Experience with distributed data processing.
• Experience with monitoring, testing, and data quality.
• Ability to work with ambiguity and own problems end-to-end.
• Experience building robust pipelines from diverse sources such as SQL & No-SQL databases and API endpoints.
• Familiarity with tools like Airflow, AWS, Kubernetes, Spark, dbt, Terraform.

Responsibilities

• Build and optimise the platform for transforming raw data into business insights.
• Develop and deploy ML infrastructure for AI requirements.
• Collaborate with dev teams for seamless integrations.
• Shape team direction and coach team members.

Skills

Passion for decarbonisation

High quality code writing

Distributed data processing

Monitoring and testing

Problem-solving

Building robust pipelines

Airflow

AWS

Kubernetes

Spark

dbt

Terraform

Databricks

MLOps implementation

Distributed system optimization

Governance

Job description

Responsibilities

• Build and optimise the platform used by the data team to transform raw data into business insights.
• Build new data sources and pipelines that deliver key data and insights to the business, including integration with third‑party systems and ingestion through APIs.
• Scope and deliver new data engineering projects in collaboration with business stakeholders.
• Develop and deploy ML infrastructure to build out our ever‑growing AI requirements and use cases.
• Collaborate closely with our dev teams to build seamless integrations between our back‑end databases and our data platform.
• Shape the direction of our growing team and coach team members on best practices.

Qualifications

• First and foremost, a passion for decarbonisation.
• A passion for writing high quality code and building lean processes.
• Experience with distributed data processing.
• Experience with monitoring, testing, and data quality.
• Ability to work with ambiguity and own problems end‑to‑end.
• Experience building robust pipelines from diverse sources such as SQL & No‑SQL databases and API endpoints.
• Experience or expertise with:
• Airflow
• AWS
• Kubernetes
• Spark & distributed computing
• dbt
• Terraform
• Databricks (desirable but not required)
• MLOps implementation (desirable but not required)
• Distributed system optimisation techniques.
• Governance.

Equal Opportunity Statement

As an equal opportunity employer, we do not discriminate on the basis of any protected attribute. Our commitment is to provide equal opportunities, an inclusive work environment, and fairness for everyone.