Data Engineer - Databricks & AI Data Platform (Hybrid)

La Fosse; Cambridge; Hybrid; GBP 88,000 - 115,000; Full time

Job summary

La Fosse is seeking an experienced Data Engineer to design and build scalable data pipelines in Databricks for an enterprise data platform. The role focuses on AI-ready data products, data governance, and reliable deployments in a hybrid Cambridge setup.

The candidate will integrate with Copilot/OpenAI and other AI platforms, develop RAG workflows, and ensure data quality and security across pipelines and models. This contract runs for six months with a daily rate of £550 within IR35.

Qualifications

• Databricks engineering with Python, PySpark and SQL for scalable pipelines.
• Experience building ETL/ELT pipelines and enterprise data models.
• Data Vault, Lakehouse and Medallion concepts understood.
• APIs and data interfaces integration across systems.

Responsibilities

• Develop, build and maintain scalable ETL/ELT pipelines using Databricks, PySpark, SQL and Python.
• Create scalable data models based on Data Vault principles.
• Develop trusted data products with Lakehouse/Medallion architectures.
• Integrate data across enterprise platforms and AI services.
• Build pipelines supporting analytics for AI usage and telemetry.
• Develop RAG pipelines for ingestion, embeddings and vector search.
• Connect LLMs, APIs and enterprise data for AI applications and agents.
• Implement automated data quality, testing, monitoring and validation.

Skills

Databricks

Python

PySpark

SQL

ETL/ELT

Data Vault

Lakehouse Architecture

CI/CD

API integration

Tools

Git

CI/CD tooling

Azure DevOps

Job description

La Fosse is seeking an experienced Data Engineer to design and build scalable data pipelines in Databricks for an enterprise data platform. The role focuses on AI-ready data products, data governance, and reliable deployments in a hybrid Cambridge setup.

The candidate will integrate with Copilot/OpenAI and other AI platforms, develop RAG workflows, and ensure data quality and security across pipelines and models. This contract runs for six months with a daily rate of £550 within IR35.