Remote Databricks Engineer - PySpark, Delta Lake, ADO

X4 Technology; Greater London, Birmingham; Hybrid; GBP 65,000 - 95,000; Full time

Job summary

X4 Technology is seeking a skilled Databricks/Data Platform Engineer to design, build and optimise data pipelines in Databricks, using PySpark, SQL and Delta Lake. You will manage production deployments via Azure DevOps and implement governance with Unity Catalog.

The role requires hands-on experience with production data platforms, strong problem-solving, and an ability to collaborate with business stakeholders and the Platform team to translate requirements into scalable technical solutions.

Qualifications

• Hands-on experience building Databricks data pipelines.
• Strong SQL and PySpark development skills.
• Experience with Delta Lake and Unity Catalog for governance.
• Proven ability to manage production data platforms.
• Familiarity with CI/CD using Azure DevOps.

Responsibilities

• Build, maintain and optimise Databricks data pipelines using PySpark, SQL and Delta Lake.
• Manage production deployments through Azure DevOps (ADO), including code reviews, testing, release management and work tracking.
• Monitor and troubleshoot data pipelines, resolving issues and optimising performance, reliability and cost.
• Implement data quality checks and governance controls using Unity Catalog to ensure data integrity, security and compliance.
• Collaborate with business stakeholders and the Platform team to translate requirements into technical solutions, maintaining clear documentation and support runbooks.
• Proven commercial experience as a Databricks Engineer, Data Engineer or Data Platform Engineer, with hands-on experience supporting production environments.
• Strong technical expertise in Databricks, PySpark, SQL and Delta Lake, including data transformation and pipeline optimisation.
• Experience with Azure DevOps (ADO), including CI/CD pipelines, code reviews, automated testing and production releases.
• Knowledge of Unity Catalog, data governance, data quality validation and production monitoring.
• Strong problem-solving and communication skills, with the ability to work collaboratively with business and technical teams

Skills

Collaborative

Analytical thinking

Problem-solving

Communication

Tools

Azure DevOps

Databricks

PySpark

SQL

Delta Lake

Unity Catalog

Job description

X4 Technology is seeking a skilled Databricks/Data Platform Engineer to design, build and optimise data pipelines in Databricks, using PySpark, SQL and Delta Lake. You will manage production deployments via Azure DevOps and implement governance with Unity Catalog.

The role requires hands-on experience with production data platforms, strong problem-solving, and an ability to collaborate with business stakeholders and the Platform team to translate requirements into scalable technical solutions.