Senior Data Engineer - Databricks Platform Architect

Jobtailor; Greater London; On-site; GBP 90,000 - 140,000; Full time

Job summary

Jobtailor in London is seeking a senior Data Engineer to design, build and operate scalable data pipelines across multiple sources, with a focus on Databricks, governance and performance.

You will co-own architecture decisions, apply IaC with Terraform, mentor engineers, and help establish Bronze-Silver-Gold medallion architecture while ensuring PII handling.

Qualifications

• Built and scaled complex data products in a growing environment.
• Hands-on Databricks expertise including workspace, governance, orchestration and engineering best practices.
• Experience designing schemas and data models, medallion architecture and Kimball/Star Schema concepts.
• Fluency with Git workflows, branching, troubleshooting, and code reviews.
• Write clean Python, PySpark, and SQL code that performs at scale.
• Experience with SQL/NoSQL databases such as Redshift, Postgres, MongoDB.
• Knowledge of infrastructure as code with Terraform.
• Mentoring and developing less experienced engineers.
• Ability to design systems, reason through trade-offs, and defend architectural decisions.
• Understanding of PII handling and data governance patterns.
• Nice to have fintech experience, re-architecting a data platform, AI/ML model building, streaming data processing, Databricks Certified Data Engineer Professional, Docker, Kubernetes, AWS/GCP.

Responsibilities

• Design, build, and run scalable, secure data pipelines across multiple sources.
• Co-own architecture decisions across ingestion, modelling, and the presentation layer.
• Design and evolve Bronze, Silver, and Gold medallion architecture.
• Define engineering standards including naming conventions, partitioning, testing, and contract templates.
• Apply Databricks best practices and Infrastructure as Code across the platform.
• Own governance patterns in Unity Catalog, including permissions, access models, and sensitive-data handling.
• Set technical governance standards for the wider team.
• Enable downstream consumers with self-service data patterns.
• Provide technical leadership, mentor engineers, review code, and raise engineering standards.

Skills

Databricks Expertise

Data Pipeline Design

Python Programming

PySpark

SQL

Terraform

Git Workflows

Medallion Architecture

Data Governance

Education

Databricks Certified Data Engineer Professional

Tools

Databricks

Redshift

PostgreSQL

MongoDB

Docker

Kubernetes

AWS

GCP

Job description

• Design, build, and run scalable, secure data pipelines across multiple sources
• Consider compute sizing and query performance
• Co-own architecture decisions across ingestion, modelling, and the presentation layer
• Design and evolve the Bronze, Silver, and Gold medallion architecture
• Define engineering standards including naming conventions, partitioning, testing strategy, and contract templates
• Apply Databricks best practices and Infrastructure as Code across the platform
• Own governance patterns in Unity Catalog, including permissions, access models, and sensitive-data handling
• Set technical governance standards for the wider team
• Enable downstream consumers with self-service data patterns
• Provide technical leadership, mentor engineers, review code, and raise engineering standards

Requirements

• Built and scaled complex data products in a growing environment
• Deep, hands-on Databricks expertise, including workspace and compute configuration, governance, orchestration, user management, and engineering best practices
• Experience designing schemas and data models, including medallion architecture and Kimball/Star Schema concepts
• Fluency with Git workflows, branching, troubleshooting, and code reviews
• Ability to write clean, efficient Python, PySpark, and SQL code that performs at scale
• Experience with SQL/NoSQL databases such as Redshift, Postgres, and MongoDB
• Knowledge of database fundamentals including indexing, performance, and design trade-offs
• Experience managing infrastructure as code with Terraform
• Previous experience mentoring and developing less experienced engineers
• Ability to design systems, reason through trade-offs, and defend architectural decisions
• Understanding of PII handling and data governance patterns
• Nice to have: fintech experience, re-architecting a data platform, AI/ML model building, Databricks Certified Data Engineer Professional or equivalent, streaming data processing, Docker, Kubernetes, and AWS/GCP
• Empathy, self-awareness, sound judgement, resilience, and a collaborative mindset
• Availability for Monday–Thursday 9:00 AM–6:00 PM and Friday 8:00 AM–4:00 PM

Core Competencies

Demonstrates expertise in designing and building scalable data pipelines, applying Databricks best practices, and implementing governance standards. Proficient in Python, PySpark, SQL, and infrastructure as code with Terraform, while mentoring engineers and enhancing engineering standards.

Highest-signal resume keywords

• Databricks Expertise
• Data Pipeline Design
• Python Programming
• Infrastructure as Code
• Data Governance

ATS Optimization Keywords

Hard Skills

• Data Modeling
• SQL
• PySpark
• Terraform
• Git Workflows
• Medallion Architecture
• Database Fundamentals
• Performance Optimization
• Streaming Data Processing
• AI/ML Model Building

Soft Skills

• Empathy
• Collaboration
• Resilience
• Self-Awareness
• Sound Judgement

Certifications & Qualifications

• Databricks Certified Data Engineer Professional

Industry Keywords

• Fintech
• Data Governance
• Sensitive Data Handling
• Architectural Decisions
• Self-Service Data Patterns

Tools & Technologies

• Databricks
• Redshift
• Postgres
• MongoDB
• Docker
• Kubernetes
• AWS
• GCP