Databricks Platform Engineer — Hybrid London (FinOps & Security)

Stichting Codam; Greater London; Hybrid; GBP 70,000 - 110,000; Full time

Job summary

Sagacity is seeking a Data Platform Engineer to design, build and operate the Databricks Lakehouse platform powering client analytics, predictive modelling, and AI work. You will collaborate with Data Scientists and Data Engineers to ensure reliability, governance, performance and cost control across financial services, retail, energy, telecoms and not-for-profit sectors.

You will contribute to secure, scalable data workflows, participate in FinOps, and help drive value through resilient

Qualifications

• 3+ years experience as a Data Platform Engineer, Cloud Engineer or Data Engineer with hands-on Databricks production experience.
• Strong knowledge of the Databricks Lakehouse, Delta Lake, Apache Spark and Unity Catalog.
• Production experience administering Databricks on AWS, Azure or GCP, including networking, IAM and storage.
• Proficiency with Terraform (Databricks provider) and Git-based CI/CD workflows.
• Solid programming skills in Python and SQL; comfortable with PySpark and shell scripting.
• Experience tuning Spark jobs and Delta tables (partitioning,).
• Working knowledge of SSO/SCIM, identity providers and secrets management.
• Understanding of data security, encryption, network isolation and frameworks such as SOC 2, ISO 27001 or GDPR.
• Highly numerate with strong attention to detail.
• Solutions orientated with natural problem-solving skills.
• Self-motivated and pro-active, with an ability to multi-task, prioritise, be organised and plan your work.
• Excellent communication skills, both written and verbal, with a willingness to engage and influence others.
• A team player with a positive attitude; personable, adaptable and willingness to learn.
• Are flexible and self-motivated; Are good under pressure; Able to travel throughout the UK; Have the right to work in the UK.

Responsibilities

• Provision, configure and operate Databricks workspaces, clusters, SQL warehouses and pools across our cloud environment.
• Implement and maintain Unity Catalog: catalogs, schemas, external locations, row/column-level security, lineage and audit.
• Build reusable Terraform modules for Databricks and automate platform changes through CI/CD pipelines.
• Tune Spark, Delta and Photon workloads to improve performance and reduce DBU and cloud spend.
• Manage identity, SSO/SCIM, service principals, secret scopes and network isolation to meet security and compliance needs.
• Operate Databricks Workflows, Delta Live Tables and orchestration integrations used by Data Scientists and Data & Analytics.
• Support MLflow, model serving and feature store usage for predictive modelling and GenAI work.
• Monitor jobs, clusters and warehouses; lead incident response and post-incident reviews.
• QA own work and that of others.

Job description

Sagacity is seeking a Data Platform Engineer to design, build and operate the Databricks Lakehouse platform powering client analytics, predictive modelling, and AI work. You will collaborate with Data Scientists and Data Engineers to ensure reliability, governance, performance and cost control across financial services, retail, energy, telecoms and not-for-profit sectors.

You will contribute to secure, scalable data workflows, participate in FinOps, and help drive value through resilient