Databricks Platform Engineer

Sagacity; Greater London; On-site; GBP 60,000 - 80,000; Full time

Job summary

Sagacity is seeking a skilled data platform engineer in Greater London to design and implement scalable Databricks Lakehouse platforms on AWS and Azure. The ideal candidate will have over 3 years of experience, strong hands-on knowledge of Databricks, and a passion for creating best-practice solutions.

The role involves working directly with clients, conducting technical workshops, and providing detailed documentation. Excellent communication skills and a proactive approach are essential for success in this consultancy environment.

Qualifications

• 3+ years experience in data platform engineering or cloud engineering.
• Strong hands-on experience with Databricks, Apache Spark, and Delta Lake.
• Experience designing and deploying data platforms on AWS and/or Azure.

Responsibilities

• Design and implement scalable Databricks platforms on AWS/Azure.
• Work with clients to translate requirements into platform designs.
• Produce high-quality technical documentation including architecture diagrams.

Skills

Data platform engineering

Cloud engineering

Databricks

Apache Spark

Infrastructure-as-code (Terraform)

CI/CD pipelines

Excellent communication skills

Tools

AWS

Azure

GitHub Actions

Job description

Platform Architecture & Engineering responsibilities

• Design and implement scalable Databricks Lakehouse platforms on AWS and/or Azure aligned to client requirements
• Architect end-to-end data platforms including ingestion, storage (Delta Lake), processing, and consumption layers
• Build and configure cloud infrastructure using infrastructure-as-code (e.g. Terraform & Declarative Automation Bundles (DAB's))
• Establish secure, compliant environments including networking (VNet/VPC, Private Link), identity (IAM/Entra ID), data governance (Unity Catalog), and access controls
• Define environment strategies (dev/test/prod), CI/CD pipelines, and release processes for Databricks deployments
• Implement monitoring, logging, cost optimisation, and performance tuning across the platform
• Design and implement data pipelines using Delta Live Tables, Auto Loader, and Databricks Workflows for both batch and streaming workloads

Client Delivery & Enablement responsibilities

• Work directly with clients to translate business and technical requirements into scalable platform designs
• Lead technical workshops, architecture sessions, and whiteboarding engagements with client stakeholders
• Support rapid prototyping and proof‑of‑concept builds within Databricks to demonstrate platform capabilities and accelerate client adoption
• Provide best practice guidance on Lakehouse architecture, data modelling, workload optimisation, and cost management
• Produce high‑quality technical documentation including architecture diagrams, architecture decision records (ADRs), runbooks, and deployment guides
• Enable client teams through structured knowledge transfer, training, and platform handover
• Collaborate with data engineers, data scientists, and product teams to ensure successful delivery outcomes

Governance & Security

• Implement Unity Catalog for centralised data governance, including access control (RBAC/ABAC), data lineage, audit logging, and compliance enforcement
• Apply security best practices across platform design: network isolation, secret management, encryption at rest and in transit, and identity federation
• Ensure platform designs meet client regulatory and compliance requirements (e.g. GDPR, ISO 27001, sector‑specific standards)

What success looks like in the role

• Delivery of robust, secure, and scalable Databricks platforms that meet client performance and cost expectations
• Clear, well‑architected solutions that balance flexibility, governance, and operational efficiency
• Strong client relationships built on trust, technical credibility, and effective communication
• Accelerated client adoption of the Lakehouse platform through well‑designed enablement and documentation
• Reduced deployment time through reusable infrastructure patterns and automation
• Proactive identification of risks, trade‑offs, and optimisation opportunities across platform design and delivery
• Contribution to the organisation's growing body of reusable platform accelerators, reference architectures, and internal knowledge

Competencies and Behaviours

• 3+ years experience in data platform engineering, cloud engineering, or similar roles
• Strong hands‑on experience with Databricks, including Apache Spark, Delta Lake, Workflows
• Proven experience designing and deploying data platforms on AWS and/or Azure (e.g. ADLS, S3, VNet/VPC, IAM)
• Experience with infrastructure-as-code tools (e.g. Terraform preferred) and CI/CD pipelines (e.g. Azure DevOps, GitHub Actions)
• Solid understanding of data architecture concepts including Lakehouse medallion architecture and dimensional modelling
• Familiarity with security and governance frameworks (e.g. RBAC, ABAC, data masking, audit, compliance standards)
• Excellent communication skills with the ability to explain complex technical concepts to non‑technical stakeholders
• Comfortable working in a client‑facing consultancy environment with multiple concurrent engagements
• Proactive, self‑driven, and able to take ownership of end‑to‑end platform delivery
• Willingness to travel within the UK as required
• Right to work in the UK