Databricks Platform Engineer

Fltalent; Sefton; Hybrid; GBP 70,000 - 90,000; Full time

Job summary

Fltalent is recruiting a Cloud Data Platform Engineer to own and operate the data platform supporting data engineering, analytics and ML teams. You will make the platform dependable, secure and efficient, combining hands-on engineering with operational ownership to reduce infra and access barriers.

You will work with engineering and analytics colleagues to enhance platform capabilities, enforce controls, and improve deployment, monitoring and maintenance processes.

Qualifications

• Experience managing Databricks in production environments.
• Strong understanding of Spark, Delta Lake and Unity Catalog.
• Experience with one or more major cloud providers (storage/networking/identity).
• Proficiency with Terraform and Git-based deployment pipelines.
• Good Python and SQL skills; PySpark for automation.

Responsibilities

• Platform engineering and automation of data workloads and deployments.
• Manage compute resources for data processing, analytics and modelling.
• Establish automated release processes with test and deployment pipelines.
• Maintain configurations and engineering standards for workloads.
• Support scheduling and integrations with the data ecosystem.
• Provide platform foundations for data science experimentation and production modelling.

Skills

Databricks management

Apache Spark

Unity Catalog

Python

SQL

PySpark

Automation scripting

Tools

Terraform

Git

Job description

The opportunity

We are recruiting a Cloud Data Platform Engineer to take ownership of the technology environment supporting our data engineering, analytics and machine learning teams.

Your focus will be on making the platform dependable, secure and efficient. You will combine hands‑on engineering with operational ownership, helping technical teams deliver their work without unnecessary infrastructure or access barriers.

Working alongside engineering and analytics colleagues, you will develop the platform’s capabilities, strengthen its controls and improve the way it is deployed, monitored and maintained.

What you will be responsible for

Platform engineering and automation

• Manage Databricks environments and the compute resources used for data processing, SQL analysis and modelling.
• Develop repeatable infrastructure deployments using Terraform and version‑controlled configuration.
• Establish automated release processes, ensuring platform changes are tested, reviewed and deployed consistently.
• Maintain reusable configurations and engineering standards that simplify the setup of new workloads.
• Support workflow scheduling and integrations with the wider data ecosystem.

Security and data governance

• Administer Unity Catalog and maintain appropriate permissions across data assets and environments.
• Configure identities, service accounts, authentication integrations and secrets management.
• Work with relevant colleagues to implement network controls, encryption and secure access.
• Maintain the controls and records needed to support auditing, data traceability and compliance requirements.

Performance and operational management

• Investigate and improve the performance of Spark workloads, Delta tables and SQL compute.
• Review resource consumption and identify ways to reduce unnecessary platform and cloud expenditure.
• Set up monitoring and alerting to identify failures, performance issues and capacity constraints.
• Resolve platform incidents, document their causes and introduce improvements that prevent recurrence.
• Maintain operational documentation and review changes for quality and reliability.

Supporting analytics and machine learning

• Provide the platform foundations needed for data science experimentation and production modelling.
• Support capabilities such as MLflow, feature management and model deployment.
• Help users diagnose platform‑related problems and adopt consistent engineering practices.
• Contribute to technical reviews, knowledge sharing and mentoring across the team.

What we are looking for

• Practical experience managing Databricks in a production environment.
• A strong understanding of Apache Spark, Delta Lake and Unity Catalog.
• Experience working with one or more major cloud providers, including storage, networking and identity management.
• Confidence using Terraform and Git‑based deployment pipelines.
• Good Python and SQL skills, with experience using PySpark and scripting for automation.
• Experience diagnosing performance issues and optimising distributed data workloads.
• Familiarity with single sign‑on, automated identity provisioning and secure credential management.
• An understanding of data protection requirements and security controls within a commercial environment.

Working arrangements

This is a permanent, full-time position with hybrid working and two days per week in a London office. Occasional UK travel may be required. Applicants must have the right to work in the UK.