Senior Data Engineer: AI-Driven Databricks on Azure

Acuity Analytics; West of England; On-site; GBP 90,000 - 130,000; Full time

Job summary

Acuity Analytics is seeking a Senior Data Engineer who wants to do more than build pipelines. You’ll help turn complex data into commercial, customer-facing products, combining strong Data Engineering skills with emerging AI and LLM technologies.

Working across Databricks, Azure, Python and PySpark, you’ll build scalable data solutions, explore new AI capabilities and collaborate with product and commercial teams to bring ideas into production.

Qualifications

• Hands-on Databricks experience.
• Advanced Python, PySpark and SQL.
• Experience building scalable data pipelines and data models.
• Experience with Azure or another cloud platform.
• Experience building APIs and exposing data.
• Experience with data products or customer-facing data solutions.
• Exposure to Generative AI / LLMs and real-world AI use cases.
• Familiarity with RAG, embeddings, vector databases or semantic search.

Responsibilities

• Build scalable data products and customer-facing data solutions.
• Develop production-ready pipelines using Databricks, Python, PySpark and SQL.
• Transform raw data into structured, enriched datasets.
• Build and integrate APIs to expose data and power products.
• Apply Generative AI/LLMs to classification, enrichment, search and insights.
• Work with structured and unstructured data, embeddings, semantic search.
• Help evolve our Databricks and Azure data platform.
• Improve data quality, testing, observability and CI/CD.
• Partner with product/commercial teams to turn opportunities into solutions.
• Mentor engineers and shape our Data & AI capability.

Skills

Databricks

Python

PySpark

SQL

Data pipelines

Data models

APIs

Generative AI

LLMs

RAG

Embeddings

Vector databases

Semantic search

Communication

Product mindset

Tools

Azure

Job description

Acuity Analytics is seeking a Senior Data Engineer who wants to do more than build pipelines. You’ll help turn complex data into commercial, customer-facing products, combining strong Data Engineering skills with emerging AI and LLM technologies.

Working across Databricks, Azure, Python and PySpark, you’ll build scalable data solutions, explore new AI capabilities and collaborate with product and commercial teams to bring ideas into production.