Data Engineer

La Fosse; Cambridge; Hybrid; GBP 88,000 - 115,000; Full time

Job summary

La Fosse is seeking an experienced Data Engineer to design and build scalable data pipelines in Databricks for an enterprise data platform. The role focuses on AI-ready data products, data governance, and reliable deployments in a hybrid Cambridge setup.

The candidate will integrate with Copilot/OpenAI and other AI platforms, develop RAG workflows, and ensure data quality and security across pipelines and models. This contract runs for six months with a daily rate of £550 within IR35.

Qualifications

• Databricks engineering with Python, PySpark and SQL for scalable pipelines.
• Experience building ETL/ELT pipelines and enterprise data models.
• Data Vault, Lakehouse and Medallion concepts understood.
• APIs and data interfaces integration across systems.

Responsibilities

• Develop, build and maintain scalable ETL/ELT pipelines using Databricks, PySpark, SQL and Python.
• Create scalable data models based on Data Vault principles.
• Develop trusted data products with Lakehouse/Medallion architectures.
• Integrate data across enterprise platforms and AI services.
• Build pipelines supporting analytics for AI usage and telemetry.
• Develop RAG pipelines for ingestion, embeddings and vector search.
• Connect LLMs, APIs and enterprise data for AI applications and agents.
• Implement automated data quality, testing, monitoring and validation.

Skills

Databricks

Python

PySpark

SQL

ETL/ELT

Data Vault

Lakehouse Architecture

CI/CD

API integration

Tools

Git

CI/CD tooling

Azure DevOps

Job description

I am looking for an experienced Data Engineer to help build and develop scalable, high-quality data solutions within Databricks. This role will be instrumental in the continued evolution of our enterprise data platform, delivering trusted, AI-ready data products that support reporting, analytics, automation and AI across Enterprise IT.

You will be responsible for designing and developing modern data pipelines that connect with a range of enterprise systems and AI platforms, including Microsoft Copilot, OpenAI and other enterprise AI services.

As their AI capabilities continue to grow, you will also engineer solutions that support AI usage and adoption data, Retrieval-Augmented Generation (RAG), LLM integration and AI orchestration.

• Typer: Contract
• Day Rate: £550 per day inside IR35
• Hybrid Working: 2 days onsite in Cambridge office
• 6 month initial contract

Responsibilities:

• Develop, build and maintain scalable ETL/ELT pipelines using Databricks, PySpark, SQL and Python.
• Create scalable data models based on Data Vault principles.
• Develop trusted data products in line with Lakehouse and Medallion Architecture principles.
• Integrate data across enterprise platforms, including ServiceNow, Jira, Azure DevOps and AI platforms such as Copilot and OpenAI.
• Build pipelines to support analytics around AI usage, adoption, telemetry and performance.
• Develop and implement RAG pipelines covering ingestion, embeddings, vector/semantic search and retrieval.
• Connect LLMs, APIs and enterprise data to enable AI applications and agents.
• Develop and maintain AI orchestration workflows that connect models, data, tools and business processes.
• Implement automated approaches to data quality, testing, monitoring and validation.
• Optimise data solutions to improve performance, scalability, reliability and cost.
• Support data governance across metadata, lineage, security and documentation.
• Use Git and CI/CD practices to enable reliable, automated deployments.

Required Skills and Experience:

• At least 4 years of data engineering experience, including strong hands‑on experience with Databricks.
• Strong knowledge of Python, PySpark and SQL.
• Proven experience designing scalable ETL/ELT pipelines and enterprise data models.
• Strong understanding of Data Vault, Delta Lake, Lakehouse and Medallion Architecture.
• Experience integrating enterprise systems using APIs and other data interfaces.
• Experience working across both structured and unstructured data.
• Hands‑on experience with Generative AI, including RAG, embeddings, vector/semantic search and LLM integration.
• Experience with AI or workflow orchestration, bringing together models, data, APIs and tools.
• Experience implementing automated data quality, testing and monitoring.
• Experience using Git and CI/CD, combined with a good understanding of data governance and security.

Nice to Have Skills and Experience:

• Experience with Microsoft Copilot, OpenAI/Azure OpenAI or Microsoft Graph.
• Experience using Databricks AI capabilities, including Mosaic AI and Vector Search.
• Experience with orchestration frameworks such as LangChain, LangGraph or Semantic Kernel.
• Experience developing AI agents or agentic workflows.
• Experience working with data from Jira, ServiceNow or Azure DevOps.
• Knowledge of Power BI, Tableau or AI/BI.
• Familiarity with Agile delivery approaches and iterative development.
• Strong communication and collaboration skills.