Senior Data Engineer

Mirai Talent; Greater London; On-site; GBP 90,000 - 120,000; Full time

Job summary

Mirai Talent is partnering with a large global organisation to appoint a Senior Data Engineer who will lead technical direction within its evolving data engineering function. You’ll own engineering standards, design patterns, performance and quality across a sizeable Azure data estate.

You’ll work across high-volume operational data, help shape how the engineering service operates and act as a key technical authority across the function.

Qualifications

• 6+ years' experience in data engineering and data management.
• Hands-on with Azure data services and big data patterns.
• Strong SQL performance tuning and data governance.
• Experience with CI/CD, DevOps and code review.
• Ability to mentor engineers and set quality standards.
• Knowledge of IoT/telemetry data is highly desirable.

Responsibilities

• Design and improve engineering patterns across Azure Databricks and Azure Data Factory.
• Set the optimisation strategy across caching, indexing, partitioning and cost.
• Define clear data engineering and integration engineering patterns.
• Act as the engineering quality gate, reviewing internal and third-party code before release.
• Own DevOps practices end-to-end, including CI/CD, branching standards, environment promotion and release governance.
• Drive automation across the engineering lifecycle, removing repetitive manual processes.
• Work with advanced SQL and existing data engineering patterns.
• Establish governance across data and algorithms used for analytics and automated decision-making.
• Collaborate with analytics engineering and data science to maintain data quality, security and governance.
• Mentor engineers and help build the technical capability of the wider team.
• Work with external engineering partners and hold them to a high technical standard.

Skills

Spark

Advanced SQL

CI/CD and DevOps

Unity Catalog

Medallion architecture

APIs (OAuth2, pagination, rate limits)

Data governance

IoT data/telemetry

Azure Databricks

Azure Data Factory

Tools

Azure Databricks

Azure Data Factory

Azure Synapse Analytics

Power BI

Job description

We’re partnering with a large, complex global organisation looking for a Senior Data Engineer to take a leading technical role within its evolving data engineering function.

This is much more than a pipeline delivery role.

You’ll own engineering standards, design patterns, performance and quality across a sizeable Azure data estate. You’ll work across high-volume operational data, help shape how the engineering service operates and act as a key technical authority across the function.

What you’ll be doing

• Designing and improving engineering patterns across Azure Databricks and Azure Data Factory
• Setting the optimisation strategy across caching, indexing, partitioning and cost
• Defining clear data engineering and integration engineering patterns
• Acting as the engineering quality gate, reviewing internal and third-party code before release
• Owning DevOps practices end-to-end, including CI/CD, branching standards, environment promotion and release governance
• Driving automation across the engineering lifecycle, removing repetitive manual processes
• Working with advanced SQL and existing data engineering patterns
• Establishing governance across data and algorithms used for analytics and automated decision-making
• Working closely with analytics engineering and data science to maintain data quality, security and governance
• Mentoring engineers and helping build the technical capability of the wider team
• Working with external engineering partners and holding them to a high technical standard

You’ll need strong hands-on experience across:

• Spark
• Advanced SQL, including query optimisation, indexing and performance diagnosis
• CI/CD and DevOps
• Unity Catalog
• Medallion architecture
• APIs, including OAuth2, pagination, rate limits and schema drift
• Data governance and orchestration
• IoT and telemetry data, particularly high-volume, high-frequency datasets

Exposure to Synapse, Power BI, complex data types or spatial data would be useful too.

What we’re looking for

You’ll ideally have 6+ years’ experience across data engineering and wider data management disciplines, including integration, modelling, optimisation and data quality.

This environment will suit someone who has worked with complex, large-scale operational data and understands what good engineering looks like beyond simply getting something into production.

You’ll be a standard setter. Someone whose approach to code, review, documentation and engineering discipline becomes the benchmark for others.

You’ll also need to be comfortable communicating across technical, business and senior stakeholder groups, mentoring other engineers and constructively challenging both internal teams and external suppliers when quality isn’t where it needs to be.

There’s a lot to get your teeth into here. You’ll be joining at a point where the organisation is evolving its data engineering operating model, with genuine scope to influence how the function develops and a longer‑term pathway into broader Platform Ownership.

At Mirai, we believe diverse teams build better technology. We’re committed to creating an inclusive recruitment process and encourage applications from people of all backgrounds, experiences and identities.

If this role sounds exciting but you don’t meet every single requirement, please don’t let that stop you from applying. We know that great people don’t always tick every box, and we’d still love to hear from you.