Senior Data Engineer – Data Products & AI

Beacon

Greater London

Hybrid

GBP 80,000 - 90,000

Full time

8 days ago

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Benefits offered by this job

Hybrid working

Flexible working arrangements

Bonus

Job summary

Beacon Intelligence in London seeks a Senior Data/AI Engineer to build and scale data products, collaborating with product, commercial and technical teams to transform datasets into customer-facing solutions.

You will shape the data platform using Databricks, Azure, Airbyte and CI/CD practices, applying AI/LLMs to deliver practical, product-ready capabilities while mentoring the team and driving commercial value. London hybrid pattern with growth opportunities.

Qualifications

Deep, hands-on experience designing and operating production data platforms on Azure Databricks.
Strong Spark, PySpark, Spark SQL and Delta Lake experience, incl. batch/streaming workloads.
Experience with Unity Catalog, data governance and schema design for production assets.
Advanced Python and SQL for production-scale data products and APIs.
Experience applying AI and LLMs to real-world data solutions.
CI/CD experience with Azure DevOps or GitHub Actions.

Responsibilities

Design and build scalable data products underpinning Beacon’s offerings.
Transform raw data into structured, enriched datasets for product use.
Partner with product and commercial teams to package and monetize data.
Enable delivery via APIs, internal tools and customer platforms.
Apply AI/LLM capabilities to enrich data and build features.
Own data platform architecture and cost optimisation.
Mentor engineers and promote a data/AI engineering mindset.

Skills

Python

SQL

Spark

Data modeling

APIs

LLMs

Product thinking

Stakeholder collaboration

Tools

Azure Databricks

Azure Cloud

Airbyte

Terraform

GitHub Actions

LangChain

LlamaIndex

Azure DevOps

Job description

Beacon Intelligence delivers data and insights that help R&D scientists develop better pharmaceuticals, faster than ever before.

Our focus is simple: we curate high-accuracy data enriched and transformed with AI. This is delivered through a range of platforms, increasingly via AI-native products and features that enable customers to extract insights faster than ever before.

This is a rare opportunity to make a meaningful impact on patients’ lives worldwide.

The Role

As a Senior Data/AI Engineer, you will play a central role in building and scaling our data products.

This is not a traditional back-end data engineering role. You will work closely with product, commercial and technical colleagues, transforming datasets into commercial, customer-facing solutions.

You will combine deep technical expertise with a strong understanding of how data can be structured, enriched and operationalised through AI systems.

What You’ll Be Doing

Data Product Development (Core Focus)

Design and build scalable data products underpinning Beacon’s commercial offerings
Transform raw and third-party data into structured, enriched, product-ready datasets
Partner with product and commercial teams to define how data is packaged, accessed and monetised
Enable delivery via APIs, internal tools and customer-facing platforms
Apply AI and LLM capabilities to enrich and enhance data (e.g. classification, tagging, summarisation, insight generation)
Design and build LLM-powered product features
Integrate RAG into pipelines to improve data quality and unlock new features
Support development of AI-driven products such as recommendation engines, search and insight tools
Ensure systems are optimised, well-governed and resilient
Support deployment and lifecycle management of data and AI systems
Own and evolve the data platform architecture (Databricks, Azure, Airbyte) to support scalability
Own performance and cost optimisation across Azure Databricks and supporting Azure services, including compute selection, autoscaling, workload monitoring and resource-efficiency improvements.
Build and maintain robust data pipelines (batch and streaming) delivering reliable, production-ready datasets
Ensure high standards in data quality, testing and observability
Improve efficiency through automation, CI/CD and engineering best practice

Commercial & Stakeholder Impact

Partner with senior stakeholders to identify high-value data product opportunities
Translate business needs into practical, scalable data and AI solutions
Act as a bridge between technical teams and commercial/product stakeholders
Contribute to prioritisation based on business impact

Leadership & Growth

Shape the evolution of our data product and AI strategy
Mentor junior engineers and promote a strong engineering and product mindset
Help build a high-performing, commercially aware data and AI engineering team

What We’re Looking For

Core Engineering Experience

Deep, hands‑on experience designing, building and operating production data platforms using Azure Databricks and Azure Cloud environments
Strong experience with Apache Spark, PySpark, Spark SQL and Delta Lake, including incremental processing, schema evolution, performance optimisation and batch and streaming workloads.
Strong experience implementing Unity Catalog, including catalog and schema design, access control, data lineage, discovery and governance of production data assets.
Proven experience designing lakehouse and Medallion architectures and converting raw and third‑party data into governed, reusable data products.
Advanced Python and SQL, focused on production‑quality, scalable solutions
Proven experience designing and building data pipelines and models
Experience working with APIs and data delivery mechanisms (critical for productisation)
Experience deploying Databricks workloads through automated CI/CD, preferably using Azure DevOps or GitHub Actions, automated testing and Databricks Declarative Automation Bundles.
Experience integrating Azure Databricks with ADLS Gen2, Azure identity, Key Vault, APIs and external data sources.
Experience building or supporting data products or customer‑facing data solutions
Strong understanding of how to structure, expose and scale data for end users or clients
Ability to think beyond pipelines and focus on commercial value
Experience working closely with product or commercial teams
Familiarity with generative AI / LLMs (Azure OpenAI, LangChain etc.) and experience applying them to real‑world use cases
Experience evaluating, monitoring and optimising AI systems
Experience working with unstructured data (text, documents, web data)
Familiarity with embeddings, vector databases or semantic search
Understanding of AI governance and data ethics
Experience with Airbyte or modern data ingestion tools
Experience with Terraform or infrastructure‑as‑code
Exposure to LLM frameworks (e.g. LangChain, LlamaIndex)
Experience with experimentation or A/B testing frameworks
Background in building data‑driven / AI‑enabled products
Domain experience in life sciences

How You Work

Focused on shipping products, not just experimentation
Think in terms of products, not just pipelines
Commercially aware and motivated by impact
Combine technical excellence with pragmatism and pace
Collaborate effectively across disciplines and influence stakeholders
Curious about AI and emerging technologies, with a focus on reliable, scalable delivery

Why Join Beacon Intelligence

Build products, not just infrastructure – directly helping scientists treat and cure diseases
Shape our AI future – define how we embed AI into products and workflows
High‑impact role – influence product strategy and commercial performance
Modern, evolving stack – work with cutting‑edge tools across data engineering and AI
Growth and ownership – join early and grow as we scale our capabilities
Collaborative environment – stakeholders who actively value data and insight

Location & Working Pattern