Senior Data Engineer: Databricks & Azure AI Data Solutions

Acuity Analytics; Greater London; Hybrid; GBP 90,000 - 130,000; Full time

Benefits offered by this job

Airbyte

Terraform

A/B testing

AI governance

Life sciences experience

Job summary

Acuity Analytics in the United Kingdom seeks a Senior Data Engineer to turn complex data into commercial, customer-facing products and services. This hands-on role blends engineering with mentoring and explorations into AI capabilities.

You will work across Databricks, Azure, Python and PySpark to build scalable data solutions, expose data via APIs, and apply generative AI to enhance classification, enrichment and insight generation.

Qualifications

• Hands-on experience with Databricks.
• Advanced Python, PySpark and SQL.
• Experience building scalable data pipelines and data models.
• Experience with Azure or another cloud platform.
• Experience building APIs to expose data.
• Experience with data products or customer-facing data solutions.
• Exposure to Generative AI / LLMs.
• Familiarity with RAG, embeddings, vector databases or semantic search.
• Strong communication skills with technical, product and commercial stakeholders.
• A product mindset – you think about how data creates value.

Responsibilities

• Build scalable data products and customer-facing data solutions.
• Develop production-ready pipelines using Databricks, Python, PySpark and SQL.
• Transform raw data into structured, enriched and commercially valuable datasets.
• Build and integrate APIs to expose data and power products.
• Apply LLMs, RAG and Generative AI to classification, enrichment, search and insight generation.
• Work with structured and unstructured data, embeddings and semantic search.
• Help evolve our Databricks and Azure data platform.
• Improve data quality, testing, observability and CI/CD.
• Partner with product and commercial teams to turn business opportunities into technical solutions.
• Mentor engineers and help shape our growing Data & AI capability.

Skills

Databricks

Python

PySpark

SQL

Data pipelines

Data models

APIs

Generative AI

LLMs

Azure

Embeddings

Vector databases

Semantic search

Communication

Product mindset

Job description

Acuity Analytics in the United Kingdom seeks a Senior Data Engineer to turn complex data into commercial, customer-facing products and services. This hands-on role blends engineering with mentoring and explorations into AI capabilities.

You will work across Databricks, Azure, Python and PySpark to build scalable data solutions, expose data via APIs, and apply generative AI to enhance classification, enrichment and insight generation.