Senior Data Engineer: Azure Databricks & Spark Lead

Mirai Tech Recruitment; Greater London; Hybrid; GBP 62,000 - 102,000; Full time

Job summary

We are recruiting for a large organisation for a senior hands-on contract role based in London. This Senior Data Engineer will design and improve engineering patterns across Azure Databricks and Azure Data Factory, drive performance, and act as a technical quality gate.

The role is London hybrid (one day onsite per week), 3-6 months, inside IR35, ASAP start. If you have strong SQL, Delta Lake, Unity Catalog and CI/CD experience, apply now.

Qualifications

• Strong hands-on experience across Azure Databricks.
• Strong hands-on experience across Azure Data Factory.
• Strong hands-on experience across Spark / PySpark.
• Advanced SQL and performance optimisation experience.
• Experience with CI/CD and Azure DevOps.
• Experience with Delta Lake and medallion architecture.
• Experience with Unity Catalog.
• Experience with data governance and orchestration.
• Experience with API-based data ingestion.
• Experience working with high-volume operational, IoT or telemetry data is particularly valuable.
• Experience at Senior or Lead Data Engineer level.
• Ability to work autonomously and hit the ground running.

Responsibilities

• Design and improve engineering patterns across Azure Databricks and Azure Data Factory.
• Drive performance and optimisation across caching, indexing, partitioning and cost.
• Review code and act as a technical quality gate.
• Improve DevOps practices across CI/CD, environment promotion, branching and release management.
• Identify opportunities to automate repetitive engineering processes.
• Work with advanced SQL across existing engineering patterns.
• Support data quality, governance and orchestration.
• Work closely with data engineering, analytics, data science and platform teams.
• Help establish consistent engineering standards across the wider service.
• Quickly understand an established data estate and identify opportunities for improvement.
• Provide immediate technical leadership across the data engineering function.

Job description

We are recruiting for a large organisation for a senior hands-on contract role based in London. This Senior Data Engineer will design and improve engineering patterns across Azure Databricks and Azure Data Factory, drive performance, and act as a technical quality gate.

The role is London hybrid (one day onsite per week), 3-6 months, inside IR35, ASAP start. If you have strong SQL, Delta Lake, Unity Catalog and CI/CD experience, apply now.