Data Engineer

Redcat Digital; Greater London; Hybrid; GBP 100,000 - 140,000; Full time

Job summary

Redcat Digital in London is seeking an experienced Senior Data Engineer to join a large-scale data environment. This hands-on role requires coding 30–50% of the time and technical ownership across delivery, with collaboration across engineers and stakeholders.

You will work with Python, Databricks and Spark to design, develop and optimise data pipelines and lakehouse architectures. The role combines hands-on development with technical leadership in an Agile setting.

Qualifications

• 10+ years' experience across data engineering and/or software engineering.
• Proven experience as a Senior Data Engineer or Technical Lead.
• Hands-on Python development experience is essential.
• Extensive experience with Databricks (notebooks, workflows, Delta Lake).
• Strong Spark / PySpark experience.
• Experience building and optimizing large-scale ETL/ELT pipelines.

Responsibilities

• Contribute to design, development and optimisation of data pipelines using Python, PySpark and Databricks.
• Build and enhance ETL/ELT processes and reusable data engineering frameworks.
• Troubleshoot complex data and performance issues; provide hands-on production support.
• Code reviews, debugging, testing and performance optimisation.
• Own delivery of data engineering initiatives; coordinate with stakeholders.

Skills

Python

Databricks

Spark / PySpark

SQL

Cloud platforms

Data modeling

CI/CD / DevOps

Tools

Databricks

Docker

Kubernetes

Kafka

Job description

London | Hybrid, 3 days per week in the office

6-month initial contract

We are looking for an experienced Senior Data Engineer with strong hands-on expertise across Python, Databricks and Spark to join a large-scale data engineering environment in London.

This is a genuinely hands-on role, with around 30–50% of your time expected to be spent coding, solving technical problems and contributing directly to the development and optimisation of data platforms and pipelines.

Alongside the hands-on engineering work, you will take technical ownership across delivery, provide direction to other engineers and work closely with stakeholders to ensure data engineering initiatives are delivered effectively.

Key responsibilities

• Contribute directly to the design, development and optimisation of data pipelines using Python, PySpark and Databricks
• Build and enhance ETL/ELT processes and reusable data engineering frameworks
• Troubleshoot complex data and performance issues and provide hands‑on support with production issues
• Contribute to code reviews, debugging, testing and performance optimisation
• Own delivery of data engineering initiatives, ensuring work is delivered to a high standard and aligned with business requirements
• Support sprint planning, backlog prioritisation and delivery tracking
• Identify and resolve technical risks, dependencies and blockers
• Provide technical direction and architectural oversight across data engineering initiatives
• Design and support modern Lakehouse architectures using Databricks and Delta Lake
• Lead and mentor other data engineers, providing technical coaching and hands‑on guidance
• Work closely with product owners, architects and business stakeholders to translate requirements into effective technical solutions
• Promote best practice across engineering, CI/CD, DevOps, Agile delivery, data quality and governance

What we're looking for

• 10+ years' experience across data engineering and/or software engineering
• Proven experience as a Senior Data Engineer, Technical Lead or similar
• Strong, recent and hands‑on Python development experience is essential
• Extensive experience with Databricks, including notebooks, workflows and Delta Lake
• Strong Apache Spark / PySpark experience
• Proven experience building and optimising large-scale ETL/ELT data pipelines
• Strong understanding of data architecture and modern Lakehouse concepts
• Strong SQL skills and experience working with distributed data systems
• Experience with at least one major cloud platform, such as AWS, Azure or GCP
• Experience working in Agile delivery environments
• Ability to balance hands‑on engineering with technical leadership and delivery responsibility
• Experience with Kafka or Spark Structured Streaming
• CI/CD and DevOps practices
• Delta Lake optimisation
• Docker and/or Kubernetes
• Exposure to machine learning pipelines

The key requirement is that you remain hands-on. This is not a pure delivery or management position. You will be expected to write code, solve complex technical problems and contribute directly to the engineering work, while also providing technical leadership and supporting delivery.

Contract: 6 months initially

Location: London, hybrid with 3 days per week in the office