Senior Data Engineer

Optima Partners

Greater London

On-site

GBP 70,000 - 110,000

Full time

2 days ago
Be an early applicant

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Benefits offered by this job

Competitive base salary
Annual discretionary bonus up to 15%
Up to 37 days holiday incl. personal &
Private medical insurance
Group life & income protection
Pension salary sacrifice
Perks and discounts portal
Development time and funding

Job summary

Optima Partners Limited is seeking a Senior Data Engineer to lead the design, build, and maintenance of scalable data pipelines and platforms for diverse clients across sectors. You will work with engineers, data scientists, and product specialists to translate business needs into robust technical solutions.

You will mentor colleagues, uphold engineering standards, and drive production readiness while delivering value through practical data insights.

Qualifications

  • Strong hands-on Python and SQL skills.
  • Experience building distributed data processing workloads with Spark/Databricks.
  • Experience with Delta Lake or lakehouse technologies.
  • Experience delivering data solutions on major cloud platforms (AWS/Azure/GCP).
  • Strong relational databases, SQL optimisation and performance troubleshooting.
  • Experience using orchestration tools (Airflow, NiFi, Talend).
  • Data warehousing, modelling and schema design for analytics.

Responsibilities

  • Lead design, build, test and operation of scalable data pipelines and data platforms.
  • Design and develop data warehouses, data lakes and lakehouse solutions.
  • Build and optimise ETL/ELT workflows with data quality checks and monitoring.
  • Translate business requirements into technical designs and communicate trade-offs.
  • Deliver well-modelled datasets for reporting with Power BI/Looker/QlikView.
  • Provide hands-on technical leadership through code reviews and mentoring.
  • Troubleshoot production issues and improve performance and cost efficiency.

Skills

Python
SQL
Data pipelines
Data modelling

Tools

Databricks
Spark
Delta Lake
Airflow
Git

Job description

Who we are

Optima Partners is a VC-backed, scaling AI and data consultancy headquartered in Edinburgh. We bring together expertise in strategy, design, data science and engineering to help leading brands solve real business challenges.


We turn data into better customer experiences, smarter decisions and lasting business value.


Our clients include Lloyds Banking Group, NatWest Group, Bank of Ireland, Nationwide, Aviva, Eon Next, OVO and GSK. We work alongside their teams to deliver practical solutions and share the skills to build on them.


Within our Data practice, you’ll work in mixed-capability pods alongside data engineers, analysts, data scientists and ML/AI engineers. You’ll bring technical depth, guide colleagues and turn data into practical solutions, clear insights and better customer experiences.


The opportunity

As a Senior Data Engineer within our Data practice, reporting to a Technical Lead, you will lead the design, build, testing and maintenance of reliable, scalable data pipelines and platforms for clients across diverse sectors. Working in multi-disciplinary teams, you’ll take ownership of technical delivery and help shape solutions from requirements through to production.


This role suits an experienced data engineer who combines strong hands‑on delivery with sound technical judgement, client‑facing confidence and a commitment to developing others.


You will work alongside engineers, data scientists, architects and product specialists to make pragmatic design decisions, resolve complex challenges and improve engineering standards. You’ll mentor colleagues, share knowledge and continue to develop your technical and consulting expertise.


What you will be doing


  • Lead the design, build, testing and operation of scalable data pipelines using platforms such as Databricks, taking ownership of delivery quality and production readiness.


  • Design and develop data warehouses, data lakes and lakehouse solutions, making pragmatic architecture and data modelling decisions with architects and technical leads.


  • Build and optimise robust ETL/ELT workflows, embedding data quality checks, monitoring and controls to deliver reliable, accessible data.


  • Work directly with clients and multi-disciplinary colleagues to translate business requirements into technical designs, estimate work and communicate trade-offs, risks and progress.


  • Deliver well‑modelled, documented datasets that support analytics, dashboards and visualisations in tools such as Power BI, Looker and QlikView, collaborating with analysts to meet reporting needs.


  • Provide hands‑on technical leadership through design and code reviews, mentoring and knowledge sharing, helping colleagues adopt consistent engineering practices.


  • Troubleshoot complex production issues and improve performance, reliability and cost efficiency through automation, testing and CI/CD. Maintain clear documentation and support effective client handover.



What skills we would like you to have


  • Strong hands‑on Python and SQL skills, with experience writing maintainable, tested code for production data workloads.


  • Practical experience building and optimising distributed data processing workloads using Spark and Databricks, or comparable platforms.


  • Experience with Delta Lake or comparable lakehouse technologies, including notebooks, workflows and reliable production deployment.


  • Experience delivering and operating data solutions on at least one major cloud platform, such as AWS, Azure or GCP, including appropriate access and security controls.


  • Strong understanding of relational databases, advanced SQL and query optimisation, with the ability to diagnose and resolve performance bottlenecks.


  • Experience implementing orchestration, scheduling, dependencies and failure handling using tools such as Airflow, Apache NiFi, Talend or platform-native workflows.


  • Strong data warehousing, modelling and schema design skills, with experience creating curated datasets for analytics and an understanding of how tools such as Power BI, QlikView and Looker consume data.


  • Experience using Git, automated testing and CI/CD to support collaborative development, controlled releases and maintainable data solutions.


  • Proven experience delivering production data engineering solutions end to end, including technical design, implementation, deployment and ongoing support.


  • Experience working with clients or business stakeholders to clarify requirements, explain technical choices and manage delivery expectations, alongside mentoring engineers and reviewing their work.


  • Experience delivering iteratively within Agile or Scrum teams, breaking down work, estimating effort and identifying dependencies and technical risks.



What you will bring


  • Strong analytical and problem‑solving skills, with sound judgement when balancing technical quality, delivery timelines and client priorities.


  • An inquisitive mindset and a commitment to continuous improvement, with the ability to evaluate new approaches and share practical learning across the Data practice.


  • Clear, confident communication, with the ability to explain complex technical ideas, challenge assumptions constructively and build trusted client relationships.


  • A collaborative, accountable approach, with the initiative to lead technical work, support colleagues’ development and give and receive constructive feedback.



As technology partners with leading companies such as AWS and Databricks, we’ll invest in your training and development, alongside hands‑on project experience with some of the UK’s leading organisations.


What we offer


  • Competitive base salary.


  • Inclusion in our annual discretionary bonus plan with an on‑target performance bonus of up to 15%.


  • Up to 37 days holiday inclusive of personal and public allocations. Seven are fixed days for Christmas and New Year, and 30 are floating days taken at your discretion, subject to client scheduling and line manager approval.


  • Private medical insurance, single cover.


  • Group life and income protection insurance.


  • Salary sacrifice pension scheme after three months’ employment.


  • Access to over 1,000 perks and discounts via an employee discount portal.


  • Dedicated development time plus tools and funding to support personal and professional development.


Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Data Engineer (early career)
Data Engineer (early career)

Optima Partners • City of Edinburgh

On-site
GBP 42,000 - 54,000
Competitive salary
Annual bonus 15%
37 days holiday
+5
Senior Data Engineer
Senior Data Engineer

Optima Partners • City of Edinburgh

On-site
GBP 80,000 - 110,000
Data & AI Consultant
Data & AI Consultant

Methods • West of England

On-site
GBP 90,000 - 110,000
LinkedIn Learning access
Management development programme
Wellness 24/7 Confidential employeeass
+8
Data Engineer
Data Engineer

Methods Business And Digital Technology • Greater London, Sheffield, West of England

Hybrid
GBP 70,000 - 100,000
LinkedIn Learning
Flexible Working
Pension
+3
Data Engineer
Data Engineer

Methods Business And Digital Technology • Bristol

Hybrid
GBP 70,000 - 110,000
Remote working flexibility
Pension
Private medical
+2
Data Architect
Data Architect

Affinity Water • Hatfield

On-site
GBP 70,000 - 110,000
Annual leave 26-30 days (increasing)
Bonus scheme
Wellbeing centre
+4
Principal Data Consultant - AWS / Databricks
Principal Data Consultant - AWS / Databricks

The Scale Factory Ltd • Greater London

Hybrid
GBP 110,000 - 140,000
25 Days of Annual Leave
Pension plan
Health benefits
+4
Senior Data Engineer
Senior Data Engineer

LV= • Bournemouth

On-site
GBP 65,000 - 95,000
Flexibility to buy or sell holiday
Annual bonus scheme
Flexible benefits package
+3
Senior Data Engineer
Senior Data Engineer

Scott Logic • Newcastle upon Tyne

Hybrid
GBP 50,000 - 80,000
25 days’ annual leave, rising to 30 days
Generous family leave policies
Employer pension scheme
+2
Senior Data Engineer
Senior Data Engineer

Menzies LLP • Woking

On-site
GBP 80,000 - 110,000
Career development
Private medical
Pension matching
+2