Senior Data Engineer - Databricks & Agentic AI

Primus Connect; Greater London; Hybrid; GBP 80,000 - 90,000; Full time

Job summary

Primus Connect is seeking a Senior Data Engineer to advance its Databricks platform with a strong focus on agentic AI. This hands-on role involves building scalable data pipelines, driving AI-driven workflows, and shaping enterprise data engineering at scale.

You'll work in a hybrid London setup, collaborating across EMEA and APAC, owning architecture decisions, and delivering production-grade solutions with a proactive, analytical mindset.

Qualifications

• Strong Databricks expertise including Delta Lake, Declarative Pipelines, DABs and Spark optimisation.
• Advanced SQL and strong Python/PySpark skills.
• Experience building scalable data solutions and troubleshooting production issues.
• Proactive, analytical mindset with interest in AI-driven engineering.

Responsibilities

• Building agentic AI workflows to automate data engineering, code generation, testing and CI/CD.
• Developing scalable data pipelines, data models and ingestion frameworks.
• Working extensively with Databricks, Delta Lake, SQL, Python and PySpark.
• Improving data quality, platform performance and production reliability.
• Delivering solutions across master data management, client data and geospatial processing.

Skills

Databricks

Delta Lake

SQL

Python

PySpark

Spark optimisation

Unity Catalog

Geospatial data

Agentic AI interest

Job description

Senior Data Engineer - Databricks & Agentic AI

London (Hybrid)

Salary - £80k - £90k DOE plus bonus and benefits

Permanent

I'm working with a global business looking for a Senior Data Engineer to help evolve its established Databricks platform. This role will have a strong focus on agentic AI, automation and the future of data engineering.

This is a hands on role where you'll work on complex transformation projects, influence technical decisions and help introduce AI-driven workflows at enterprise scale.

What you'll be doing:

• Building agentic AI workflows to automate data engineering, code generation, testing and CI/CD.
• Developing scalable data pipelines, data models and ingestion frameworks.
• Working extensively with Databricks, Delta Lake, SQL, Python and PySpark.
• Improving data quality, platform performance and production reliability.
• Delivering solutions across master data management, client data and geospatial processing.

What they're looking for:

• Strong Databricks expertise, including Delta Lake, Declarative Pipelines, DABs and Spark optimisation.
• Advanced SQL and strong Python/PySpark skills.
• Experience building scalable data solutions and troubleshooting production issues.
• A proactive, analytical mindset and an interest in AI-driven engineering.
• Experience with coding agents, agentic AI, Unity Catalog or geospatial data would be advantageous.

Why is this a great opportunity?

• Work at the cutting edge of AI: Help shape how agentic AI is applied to real-world data engineering, developing skills that are still relatively rare in the market.
• Genuine technical ownership: Influence architecture, make meaningful engineering decisions and help shape the platform's future direction.
• Build, don't rebuild: Join a mature Databricks environment with established foundations, allowing you to focus on innovation rather than starting from scratch.
• Invest in your development: Work alongside a technically ambitious team with dedicated learning time and the freedom to explore emerging technologies.
• Make an impact at scale: Contribute to transformation initiatives across EMEA and APAC, tackling complex challenges across multiple business domains.

If you're a Senior Data Engineer with strong Databricks experience and want to be part of the next evolution of data engineering,