Data Engineer: AI Pipelines on Databricks (Hybrid Cambridge)

Whitehall Resources Limited; Cambridgeshire and Peterborough; Hybrid; GBP 70,000 - 90,000; Full time

Job summary

Whitehall Resources Limited is seeking an experienced Data Engineer to design and deliver scalable data solutions in Databricks. The role blends onsite collaboration in Cambridge with remote work, on an initial 6‑month contract, and is inside IR35.

You will build modern ETL/ELT pipelines, design Lakehouse architectures, and integrate with enterprise systems and AI platforms. Strong Python, PySpark, SQL and governance experience are essential.

Qualifications

• Minimum 4 years' data engineering experience, with strong experience in Databricks.
• Strong Python, PySpark and SQL skills.
• Experience designing scalable ETL/ELT pipelines and enterprise data models.
• Strong understanding of Data Vault, Delta Lake, Lakehouse and Medallion Architecture.
• Experience integrating enterprise systems through APIs and other data interfaces.
• Experience working with structured and unstructured data.
• Practical experience with Generative AI, RAG, embeddings, vector/semantic search and LLM integration.
• Experience with AI or workflow orchestration, connecting models, data, APIs and tools.
• Experience with automated data quality, testing and monitoring.
• Experience with Git and CI/CD and good understanding of data governance and security.
• Nice to Have: Microsoft Copilot, OpenAI/Azure OpenAI, Microsoft Graph.
• Databricks AI capabilities like Mosaic AI and Vector Search.
• Orchestration frameworks: LangChain, LangGraph, Semantic Kernel.
• Experience developing AI agents or agentic workflows.
• Experience with Jira, ServiceNow or Azure DevOps data.
• Knowledge of Power BI, Tableau or AI/BI.
• Familiarity with Agile delivery and iterative development.
• Strong communication and collaboration skills.

Responsibilities

• Design, build and maintain scalable ETL/ELT pipelines using Databricks, PySpark, SQL and Python.
• Build trusted data products using Lakehouse and Medallion Architecture principles.
• Integrate data from enterprise systems including ServiceNow, Jira, Azure DevOps and AI platforms such as Copilot and OpenAI.
• Build data pipelines for AI usage, adoption, telemetry and performance analytics.
• Design and implement RAG pipelines, including ingestion, embeddings, vector/semantic search and retrieval.
• Integrate LLMs, APIs and enterprise data to support AI applications and agents.
• Build and maintain AI orchestration workflows connecting models, data, tools and business processes.
• Implement automated data quality, testing, monitoring and validation.
• Optimise solutions for performance, scalability, reliability and cost.
• Contribute to data governance, metadata, lineage, security and documentation.
• Use Git and CI/CD to support automated and reliable deployment.

Skills

Databricks

Python

PySpark

SQL

ETL/ELT pipelines

Data Vault

Delta Lake

Lakehouse

Medallion Architecture

APIs

LLM integration

RAG

Vector/semantic search

Generative AI

AI orchestration

Git

CI/CD

Data governance

Security

Communication

Team collaboration

Tools

Jira

ServiceNow

Azure DevOps

Git

CI/CD

Power BI

Tableau

Job description

Whitehall Resources Limited is seeking an experienced Data Engineer to design and deliver scalable data solutions in Databricks. The role blends onsite collaboration in Cambridge with remote work, on an initial 6‑month contract, and is inside IR35.

You will build modern ETL/ELT pipelines, design Lakehouse architectures, and integrate with enterprise systems and AI platforms. Strong Python, PySpark, SQL and governance experience are essential.