Databricks-Driven Data Platform & RAG Pipelines

Whitehall Resources Limited; Cambridge; Hybrid; GBP 65,000 - 90,000; Full time

Job summary

Whitehall Resources Limited is seeking a Data Engineer for a hybrid role with two days per week onsite in Cambridge and the remainder remote on a six-month contract. You will design and deliver scalable data solutions in Databricks, build AI-ready data products, and integrate enterprise systems.

The role requires strong Python, PySpark and SQL skills with expertise in data vault and lakehouse architectures.

Qualifications

• Minimum 4 years' data engineering experience with Databricks.
• Strong Python, PySpark and SQL skills.
• Experience designing scalable ETL/ELT pipelines and enterprise data models.
• Strong understanding of Data Vault, Delta Lake, Lakehouse and Medallion Architecture.
• Experience integrating enterprise systems via APIs and data interfaces.
• Experience with structured and unstructured data.
• Practical experience with Generative AI, RAG, embeddings, vector/semantic search and LLM integration.
• Experience with AI workflows, connecting models, data, APIs and tools.
• Experience with automated data quality, testing and monitoring.
• Experience with Git and CI/CD; understanding of data governance and security.

Responsibilities

• Design, build and maintain scalable ETL/ELT pipelines using Databricks, PySpark, SQL and Python.
• Build trusted data products using Lakehouse and Medallion Architecture.
• Integrate data from enterprise systems and AI platforms for reporting and analytics.
• Develop data pipelines for AI usage, telemetry and performance analytics.
• Implement RAG pipelines including embeddings and vector search.
• Integrate LLMs and enterprise data to support AI applications and agents.
• Create AI orchestration workflows connecting models, data, tools and business processes.
• Ensure automated data quality, testing and monitoring; optimise for performance and cost.
• Contribute to data governance, metadata, lineage, security and documentation.
• Use Git and CI/CD for automated deployment.

Skills

Databricks

Python

PySpark

SQL

ETL/ELT pipelines

Data modeling

AI/RAG/LLM integration

APIs/integration

Git & CI/CD

Data governance & security

Tools

Databricks platform

Spark

Job description

Whitehall Resources Limited is seeking a Data Engineer for a hybrid role with two days per week onsite in Cambridge and the remainder remote on a six-month contract. You will design and deliver scalable data solutions in Databricks, build AI-ready data products, and integrate enterprise systems.

The role requires strong Python, PySpark and SQL skills with expertise in data vault and lakehouse architectures.