Data Engineering Tech Lead: AI Pipelines & Data Foundations

Eaglecliff; Greater London; On-site; GBP 90,000 - 150,000; Full time

Job summary

Eaglecliff is seeking a Data Engineer Tech Lead to join a world-leading Energy Trading company to lead, design and build data foundations and end-to-end solutions for the Business to maximize value from data.

The candidate will be a subject-matter expert who designs and builds data solutions, as well as AI-coded automated data pipelines, while mentoring junior engineers and translating Vision and Data Strategy into IT solutions.

Qualifications

• Expert data engineering with end-to-end pipeline design.
• Strong hands-on with Databricks, PySpark, Python and SQL.
• Mentor junior engineers and lead data foundation initiatives.
• Translate data strategy into IT solutions and deliver them.

Responsibilities

• Lead data platform design and build data foundations.
• Develop AI-coded automated data pipelines.
• Collaborate with business stakeholders and manage change.
• Drive efficiency and cost reduction across pilots.

Skills

Azure Stack

ELT Experience

Data Modeling

Data Integration

Data Ingestion

Data Manipulation

GitHub

GitHub Actions

Azure DevOps

Databricks

PySpark

Python

SQL

SQL DB

Synapse

Stream Analytics

Glue

Airflow

Kinesis

Redshift

SonarQube

PyTest

AI based Code

Tools

GitHub

GitHub Actions

Azure DevOps

Databricks

PyTest

Airflow

Kinesis

Redshift

Synapse

Data Factory

Glue

Job description

Eaglecliff is seeking a Data Engineer Tech Lead to join a world-leading Energy Trading company to lead, design and build data foundations and end-to-end solutions for the Business to maximize value from data.

The candidate will be a subject-matter expert who designs and builds data solutions, as well as AI-coded automated data pipelines, while mentoring junior engineers and translating Vision and Data Strategy into IT solutions.