Senior Databricks Engineer - PySpark, SQL, Airflow, Azure

Cornwallis Elt Ltd; England; Hybrid; GBP 60,000 - 75,000; Full time

Job summary

A well-established financial services organisation is seeking a Senior Databricks Engineer to design and maintain data pipelines while mentoring a small team of Data Engineers. This hybrid role requires strong expertise in Databricks, Azure cloud services, and a collaborative approach to problem-solving. Ideal candidates will bring experience in Agile development, Python, and SQL alongside a passion for data engineering.

Qualifications

• Experience with Azure cloud services.
• Strong Databricks experience including Unity Catalog.
• Solid coding skills in Python, Spark, and SQL.

Responsibilities

• Mentor and support a small team of Data Engineers.
• Develop and maintain data models, schemas, and documentation.
• Design, build, and maintain scalable data pipelines.

Skills

Databricks experience

Azure cloud services

Python

Spark

SQL

Mentorship

Problem-solving

Agile practices

Tools

Terraform

Airflow

Azure Data Factory

Job description

Senior Databricks Engineer

Location: Banbury or Manchester (Hybrid - 2/3 days in office)

Salary: £60,000 - £75,000 per annum

Contract: Permanent, Full-Time

We're working with a well-established financial services organisation that's growing its data capability - and they're looking for a Senior Databricks Engineer to join the team.

What you'll be doing

As a Senior Databricks Engineer, you'll play a key role in designing, building, and maintaining data pipelines that help the business make better decisions. You'll work closely with the Data Architect and Data Product Manager to turn business requirements into reliable, high-quality data solutions.

You'll spend most of your time developing hands‑on - but you'll also lead and support a small team of Data Engineers, helping them deliver great work and follow best practices.

Day to day, you'll:

• Mentor and support a small team of Data Engineers.
• Work with business teams to understand data needs and translate them into practical, well-built solutions.
• Develop and maintain data models, schemas, and documentation.
• Lead by example - setting standards for coding, design, and delivery in Databricks.
• Design, build, and maintain scalable data pipelines and ETL processes across cloud platforms, databases, and APIs.
• Optimise data systems for performance, reliability, and scalability.
• Collaborate with the Data Architect to shape and deliver the data architecture roadmap.
• Maintain strong governance and ensure data quality through validation and monitoring.
• Stay current with emerging tools and technologies to recommend improvements.
• Work within an Agile environment using Azure DevOps for CI/CD, version control, and sprint delivery.

What we're looking for

You’ll be an experienced Databricks Engineer who loves solving data problems, writing clean and efficient code, and helping others grow. You'll bring a mix of technical know‑how, curiosity, and collaboration to the team.

• Experience with Azure cloud services.
• Strong Databricks experience (including Unity Catalog).
• Familiarity with Terraform and infrastructure as code.
• A good understanding of Agile delivery and CI/CD practices.
• Experience building data pipelines with Airflow or Azure Data Factory.
• Solid coding skills in Python, Spark, and SQL.
• Great communication and problem‑solving skills.
• The ability to mentor, guide, and support other engineers.