Senior GenAI Data Engineer – Enterprise AI & Databricks

Aubay UK; Greater London; On-site; GBP 90,000 - 130,000; Full time

Job summary

Aubay UK is seeking a Principal AI Data Engineer to join a major transformation programme in the energy sector. This 6-month contract focuses on designing, developing and deploying enterprise-scale GenAI and Agentic AI solutions, taking AI products from rapid prototyping through to production adoption.

Based in London, the role combines advanced AI engineering, Databricks expertise, stakeholder engagement and delivery leadership to ensure value delivery across the enterprise.

Qualifications

• 8–12+ years in Data Science, AI or related disciplines.
• Degree in Statistics, Mathematics, Econometrics or similar.
• Deep understanding of LLMs including GPT, Llama, Claude and Mistral.
• Strong Databricks expertise including DLT, Delta Lake and Unity Catalog governance.
• Experience with Spark Structured Streaming and Autoloader.
• Strong Python, SQL and/or Scala skills.
• Hands-on experience with Azure OpenAI, AI Foundry, MLflow, Mosaic AI, Agent Bricks and Genie.
• Experience with GenAI evaluation frameworks and benchmarking methodologies.
• Strong Git and collaborative development practices.

Responsibilities

• Develop and evaluate AI, GenAI and Agentic AI prototypes.
• Build, tune and evaluate RAG solutions.
• Design and deploy AI agents and scalable AI workflows.
• Deliver production-ready GenAI solutions for business users.
• Build and manage large-scale structured and unstructured datasets using Databricks.
• Implement testing, benchmarking and reproducibility frameworks.
• Deploy models using Azure AI services, Databricks and Docker.
• Apply DevOps best practices including CI/CD, testing and GitHub workflows.
• Collaborate with technical and business stakeholders to ensure value delivery.
• Contribute to architecture, planning, sprint ceremonies and backlog development.

Skills

LLMs

Databricks

Python

SQL

Scala

ETL/ELT

Data Modeling

Git

Stakeholder management

CI/CD

Education

Degree in Statistics/Mathematics/Econometrics

Tools

Azure OpenAI

MLflow

Delta Lake

Unity Catalog

Docker

GitHub

Job description

Aubay UK is seeking a Principal AI Data Engineer to join a major transformation programme in the energy sector. This 6-month contract focuses on designing, developing and deploying enterprise-scale GenAI and Agentic AI solutions, taking AI products from rapid prototyping through to production adoption.

Based in London, the role combines advanced AI engineering, Databricks expertise, stakeholder engagement and delivery leadership to ensure value delivery across the enterprise.