Senior Data Engineer: Lead Scalable Pipelines & Lakehouse

Optima Partners

Greater London

On-site

GBP 70,000 - 110,000

Full time

4 days ago
Be an early applicant

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Benefits offered by this job

Competitive base salary
Annual discretionary bonus up to 15%
Up to 37 days holiday incl. personal &
Private medical insurance
Group life & income protection
Pension salary sacrifice
Perks and discounts portal
Development time and funding

Job summary

Optima Partners Limited is seeking a Senior Data Engineer to lead the design, build, and maintenance of scalable data pipelines and platforms for diverse clients across sectors. You will work with engineers, data scientists, and product specialists to translate business needs into robust technical solutions.

You will mentor colleagues, uphold engineering standards, and drive production readiness while delivering value through practical data insights.

Qualifications

  • Strong hands-on Python and SQL skills.
  • Experience building distributed data processing workloads with Spark/Databricks.
  • Experience with Delta Lake or lakehouse technologies.
  • Experience delivering data solutions on major cloud platforms (AWS/Azure/GCP).
  • Strong relational databases, SQL optimisation and performance troubleshooting.
  • Experience using orchestration tools (Airflow, NiFi, Talend).
  • Data warehousing, modelling and schema design for analytics.

Responsibilities

  • Lead design, build, test and operation of scalable data pipelines and data platforms.
  • Design and develop data warehouses, data lakes and lakehouse solutions.
  • Build and optimise ETL/ELT workflows with data quality checks and monitoring.
  • Translate business requirements into technical designs and communicate trade-offs.
  • Deliver well-modelled datasets for reporting with Power BI/Looker/QlikView.
  • Provide hands-on technical leadership through code reviews and mentoring.
  • Troubleshoot production issues and improve performance and cost efficiency.

Skills

Python
SQL
Data pipelines
Data modelling

Tools

Databricks
Spark
Delta Lake
Airflow
Git

Job description

Optima Partners Limited is seeking a Senior Data Engineer to lead the design, build, and maintenance of scalable data pipelines and platforms for diverse clients across sectors. You will work with engineers, data scientists, and product specialists to translate business needs into robust technical solutions.

You will mentor colleagues, uphold engineering standards, and drive production readiness while delivering value through practical data insights.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Senior Data Engineer — Lakehouse Data Pipelines Lead
Senior Data Engineer — Lakehouse Data Pipelines Lead

Harnham • Greater London

Hybrid
GBP 60,000 - 80,000
Senior Data Engineer
Senior Data Engineer

Optima Partners • City of Edinburgh

On-site
GBP 80,000 - 110,000
Senior Data Engineer - Flexible Hybrid, Scalable Pipelines
Senior Data Engineer - Flexible Hybrid, Scalable Pipelines

Copello Global Ltd • Birmingham

Hybrid
GBP 50,000 - 70,000
Pension scheme with up to 12% employee
Private GP
3x life assurance
+2
Senior Data Engineer: AWS Pipelines, Lead & Hybrid Role
Senior Data Engineer: AWS Pipelines, Lead & Hybrid Role

Center Parcs • Newark and Sherwood

Hybrid
GBP 60,000 - 65,000
Bonus scheme
Pension
Recognition & rewards
+6
Data Engineer: Build Scalable Lakehouse Pipelines
Data Engineer: Build Scalable Lakehouse Pipelines

DVF Recruitment • England

On-site
GBP 60,000 - 90,000
Senior Data Engineer
Senior Data Engineer

ISx4 Group • Belfast City District

On-site
GBP 70,000 - 100,000
Cloud Data Engineer: Architect Scalable Data Pipelines
Cloud Data Engineer: Architect Scalable Data Pipelines

VM2R Services Ltd • West of England

On-site
GBP 27,000 - 33,000
Senior Data Platform Engineering Leader (Hybrid, London)
Senior Data Platform Engineering Leader (Hybrid, London)

Preply • Greater London

Hybrid
GBP 140,000 - 190,000
Lessons allowance
Learning & Development budget
Health insurance
+2
Senior Data Engineer: Build Scalable Data Pipelines
Senior Data Engineer: Build Scalable Data Pipelines

Moneybox • Greater London

Hybrid
GBP 90,000 - 120,000
Company bonus scheme
Company pension scheme
Hybrid working
+3
Data Engineer: AI Pipelines & Insight Dashboards
Data Engineer: AI Pipelines & Insight Dashboards

Optima Partners • City of Edinburgh

On-site
GBP 42,000 - 54,000
Competitive salary
Annual bonus 15%
37 days holiday
+5