Senior Databricks Data Platform Engineer

Intetics; United Kingdom; On-site; GBP 90,000 - 130,000; Full time

Job summary

Intetics seeks a data engineering specialist to own production Databricks pipelines across Azure and AWS. You will monitor data flows, optimize performance, and drive cost efficiencies while ensuring security and governance in a multi-tenant environment.

You will migrate legacy ETL/ELT stacks to Databricks, collaborate with data science, product, and infra teams, and support onboarding of new customers with isolated, reliable data pipelines. Strong PySpark/SQL/Python skills required.

Qualifications

• 4+ years of data engineering experience.
• At least 2 years of experience with Databricks or the Apache Spark ecosystem across Azure and/or AWS.
• Proficiency in PySpark, SQL, and Python with a strong track record of building and operating production-grade pipelines under SLA constraints.
• Hands‑on experience with Delta Lake, including schema evolution, ACID transactions, optimize/vacuum lifecycle, and both incremental and streaming processing patterns.
• Hands‑on experience with pipeline performance tuning and compute optimization in production Databricks environments.
• Solid working knowledge of PostgreSQL, including query optimization, schema design, and use as a source or sink in production data pipelines.
• Experience supporting and maintaining legacy ETL tooling (SSIS, Informatica, custom Python/SQL pipelines, or similar) in production.
• Experience supporting large‑scale multi‑tenant architectures with a focus on tenant isolation, per‑tenant performance, and data privacy.
• Proven ability to work collaboratively across data science, product, and infrastructure teams, owning end‑to‑end delivery in a cross‑functional environment.
• Strong understanding of data governance, security, and compliance principles, including access control, data privacy, and protection of sensitive enterprise data across multi‑tenant environments.

Responsibilities

• Own Databricks production support for the company's data platform, including monitoring, alerting, and incident response across all production data flows.
• Maintain and report on SLA performance metrics for data pipeline delivery, ensuring visibility into platform health and accountability across internal and external stakeholders.
• Identify and implement pipeline optimizations that reduce Databricks compute costs, improve throughput, and reduce processing windows while tracking impacts through measurable KPIs.
• Migrate legacy ETL/ELT pipelines to Databricks, building automation tooling to reduce manual intervention and ensure uninterrupted data delivery during transitions.
• Support new customer onboarding by provisioning, validating, and hardening tenant data pipelines that deliver reliable, isolated data from day one.
• Design and build high-performance Databricks pipelines that ingest, transform, and serve ERP and CRM data at scale across both Azure and AWS environments.
• Own the Delta Lake architecture, including schema design, partitioning strategies, data quality enforcement, and incremental processing patterns.
• Enforce data security best practices across Databricks environments, including role-based access control, secrets management, and compliance requirements for enterprise business data.
• Implement data quality monitoring and observability across pipeline health and ML model inputs, ensuring data integrity that directly supports predictive analytics.
• Apply and enforce multi-tenant data isolation patterns, ensuring reliable and secure data delivery across enterprise customers.
• Partner with the Enterprise Architecture team to ensure data pipelines integrate seamlessly with the broader AI and analytics ecosystem.
• Support a globally distributed operation through on-call rotation and after-hours incident response, meeting SLAs across multiple time zones.
• Maintain technical documentation, runbooks, and architectural decision records, contributing to team knowledge sharing and operational readiness across on-call and incident response scenarios.
• Apply CI/CD best practices to data pipeline development, including version control, automated testing, and deployment tooling to ensure reliable and repeatable pipeline delivery.

Skills

Data engineering

SLA management

On-call rotation

Data governance

Security/compliance

Cross-functional delivery

Tools

Databricks

Apache Spark

Delta Lake

PySpark

SQL

Python

PostgreSQL

SSIS

Informatica

Infrastructure as Code

CI/CD

Databricks Feature Store

Job description

Intetics seeks a data engineering specialist to own production Databricks pipelines across Azure and AWS. You will monitor data flows, optimize performance, and drive cost efficiencies while ensuring security and governance in a multi-tenant environment.

You will migrate legacy ETL/ELT stacks to Databricks, collaborate with data science, product, and infra teams, and support onboarding of new customers with isolated, reliable data pipelines. Strong PySpark/SQL/Python skills required.