Principal AI Data Engineer (AI/GenAI Data Engineer)

Aubay UK; Greater London; On-site; GBP 90,000 - 130,000; Full time

Job summary

Aubay UK is seeking a Principal AI Data Engineer to join a major transformation programme in the energy sector. This 6-month contract focuses on designing, developing and deploying enterprise-scale GenAI and Agentic AI solutions, taking AI products from rapid prototyping through to production adoption.

Based in London, the role combines advanced AI engineering, Databricks expertise, stakeholder engagement and delivery leadership to ensure value delivery across the enterprise.

Qualifications

• 8–12+ years in Data Science, AI or related disciplines.
• Degree in Statistics, Mathematics, Econometrics or similar.
• Deep understanding of LLMs including GPT, Llama, Claude and Mistral.
• Strong Databricks expertise including DLT, Delta Lake and Unity Catalog governance.
• Experience with Spark Structured Streaming and Autoloader.
• Strong Python, SQL and/or Scala skills.
• Hands-on experience with Azure OpenAI, AI Foundry, MLflow, Mosaic AI, Agent Bricks and Genie.
• Experience with GenAI evaluation frameworks and benchmarking methodologies.
• Strong Git and collaborative development practices.

Responsibilities

• Develop and evaluate AI, GenAI and Agentic AI prototypes.
• Build, tune and evaluate RAG solutions.
• Design and deploy AI agents and scalable AI workflows.
• Deliver production-ready GenAI solutions for business users.
• Build and manage large-scale structured and unstructured datasets using Databricks.
• Implement testing, benchmarking and reproducibility frameworks.
• Deploy models using Azure AI services, Databricks and Docker.
• Apply DevOps best practices including CI/CD, testing and GitHub workflows.
• Collaborate with technical and business stakeholders to ensure value delivery.
• Contribute to architecture, planning, sprint ceremonies and backlog development.

Skills

LLMs

Databricks

Python

SQL

Scala

ETL/ELT

Data Modeling

Git

Stakeholder management

CI/CD

Education

Degree in Statistics/Mathematics/Econometrics

Tools

Azure OpenAI

MLflow

Delta Lake

Unity Catalog

Docker

GitHub

Job description

Aubay UK is seeking an experienced Principal AI Data Engineer to join a major transformation programme within the energy sector. This is a 6-month contract engagement (inside IR35) with possible extension, focused on designing, developing and deploying enterprise-scale GenAI and Agentic AI solutions. The role combines advanced AI engineering, Databricks expertise, stakeholder engagement and delivery leadership, taking AI products from rapid prototyping through to production adoption.

Type of Role

Contract Inside IR35 (6 months with possible extension).

Location and Travel

Location: London Waterloo. Working pattern: On-site, 5 days per week. Travel requirements: None. Visa sponsorship available.

Required Skills and Experience

• 8-12+ years of experience in Data Science, AI or related disciplines.
• Degree in Statistics, Mathematics, Econometrics or similar.
• Deep understanding of LLMs including GPT, Llama, Claude and Mistral.
• Strong Databricks expertise including DLT, Delta Lake and Unity Catalog governance.
• Experience with Spark Structured Streaming and Autoloader.
• Strong Python, SQL and/or Scala skills.
• Proven data modelling, ETL/ELT and data architecture experience.
• Hands-on experience with Azure OpenAI, AI Foundry, MLflow, Mosaic AI, Agent Bricks and Genie.
• Experience with GenAI evaluation frameworks and benchmarking methodologies.
• Strong Git and collaborative development practices.
• Excellent communication, stakeholder management and problem-solving skills.

Desired Skills and Experience

• Experience with enterprise AI governance models.
• Knowledge of LangChain, LangGraph, AutoGen and smolagents.
• Experience with Docker and CI/CD pipelines.
• Familiarity with ADF and Qlik Replicate.
• Experience leading backlogs and technical roadmaps.
• Commercial experience supporting trading, supply chain or energy businesses.

Role Responsibilities

• Develop and evaluate AI, GenAI and Agentic AI prototypes.
• Build, tune and evaluate RAG solutions.
• Design and deploy AI agents and scalable AI workflows.
• Deliver production-ready GenAI solutions for business users.
• Build and manage large-scale structured and unstructured datasets using Databricks.
• Implement testing, benchmarking and reproducibility frameworks.
• Deploy models using Azure AI services, Databricks and Docker.
• Apply DevOps best practices including CI/CD, testing and GitHub workflows.
• Collaborate with technical and business stakeholders to ensure value delivery.
• Contribute to architecture, planning, sprint ceremonies and backlog development.

Client Description

Our client is one of the Super Major global energy companies with around 84,000 employees across 70+ countries who are working to power progress through cleaner energy solutions. Specialties include upstream and downstream operations, biofuels, integrated gas, new energies, chemicals, energy and trading. You will have the opportunity to work in a challenging and rewarding environment focused on delivering innovative solutions that support the transition to a lower-carbon energy future.