Senior Data Engineer

GCS Recruitment Specialists; Greater London; Hybrid; GBP 90,000 - 130,000; Full time

Job summary

GCS Recruitment Specialists is seeking a Senior Data Engineer for a hybrid, banking-sector contract role in London. You will build scalable data pipelines using Python, Databricks and Spark, and own critical data platforms end-to-end.

The role demands strong SQL, Databricks expertise and experience delivering enterprise-grade data solutions in cloud environments. You will collaborate with engineers and stakeholders to drive performance and reliability.

Qualifications

• Proven data engineering experience in Python, Databricks and Spark environments.
• Strong SQL development and data integration design.
• Experience building scalable ETL/ELT pipelines at enterprise scale.
• Hands-on experience with Databricks Lakehouse architectures.
• Experience with cloud platforms (Azure, AWS or GCP).
• Familiar with Agile development and CI/CD practices.
• Excellent problem-solving and debugging skills.

Responsibilities

• Design, develop and maintain scalable data pipelines.
• Develop Databricks-based data solutions and lakehouse architectures.
• Optimise data processing workloads for performance and reliability.
• Collaborate with stakeholders to deliver effective data solutions.
• Promote testing, code reviews, and CI/CD adoption.
• Mentor engineers and contribute to best practices.

Skills

Data engineer experience

Python

Databricks

Spark / PySpark

SQL

ETL design

Distributed computing

Cloud platforms

Software best practices

Agile

Troubleshooting

Tools

Databricks

Airflow

Databricks Workflows

Kafka

Docker

Kubernetes

CI/CD

Job description

Senior Data Engineer Role (Python / Spark / Databricks) - Hybrid - Contract - Banking

Role - Senior Data Engineer

Rate - £850 p/d (Inside IR35)

Duration - 6 months with very likely extension

Location - Hybrid / Liverpool Street (London) - 3 days per week in a Liverpool Street office

About the Role

Global financial institution seeking a highly skilled Senior Data Engineer to join a growing data engineering function responsible for delivering critical data platforms and analytics capabilities across the business.

This is a genuinely hands-on engineering role, requiring strong expertise in Python, Databricks, Spark/PySpark and SQL. The successful candidate will be actively involved in the design, development, optimisation and support of enterprise-scale data platforms, working alongside engineers, architects and business stakeholders to deliver robust and scalable solutions.

The role offers the opportunity to work on complex data challenges within a modern cloud-based environment, leveraging the latest data engineering technologies and best practices.

Key Responsibilities

Data Engineering & Development

• Design, develop and maintain scalable data pipelines using Python, Databricks and PySpark
• Build and optimise ETL/ELT solutions processing large-scale datasets
• Develop high-quality, reusable code and engineering frameworks
• Deliver robust, performant and maintainable data solutions
• Troubleshoot and resolve complex production and performance issues
• Support the ongoing enhancement and modernisation of enterprise data platforms

Databricks & Data Platform Engineering

• Develop and support Databricks-based data solutions and Lakehouse architectures
• Utilise Delta Lake and Spark technologies to deliver scalable data processing capabilities
• Optimise distributed data processing workloads for performance and reliability
• Implement data quality controls and monitoring solutions
• Contribute to platform stability, scalability and operational excellence

Solution Delivery

• Work closely with business and technology stakeholders to understand requirements and deliver effective solutions
• Participate in Agile delivery processes including sprint planning, refinement and estimation
• Support the delivery of strategic data initiatives across multiple business areas
• Collaborate with architects, engineers and product teams to deliver high-quality outcomes
• Identify and resolve technical risks and implementation challenges

Engineering Best Practice

• Promote software engineering best practices including testing, code reviews and version control
• Support CI/CD adoption and deployment automation
• Contribute to coding standards and engineering governance
• Drive continuous improvement across development and support processes
• Mentor junior engineers and contribute to knowledge sharing within the team

Required Skills & Experience

• Strong commercial experience as a Data Engineer or Senior Data Engineer
• Expert-level Python development skills
• Extensive hands-on experience with Databricks in enterprise environments
• Strong Apache Spark and PySpark expertise
• Advanced SQL development and optimisation skills
• Experience designing and building large-scale ETL and data integration solutions
• Strong understanding of distributed computing concepts and data processing frameworks
• Experience working with modern cloud platforms including Azure, AWS or GCP
• Knowledge of software engineering principles, testing and code quality practices
• Experience working within Agile development environments
• Excellent problem-solving and troubleshooting capabilities

Desirable Experience

• Experience within investment banking, financial services or other highly regulated environments
• Exposure to risk, trading, treasury or regulatory data domains
• Experience with Kafka, streaming technologies or real-time data processing
• Databricks certifications
• Experience with Airflow, Databricks Workflows or similar orchestration tools
• CI/CD pipeline development and DevOps practices
• Docker and Kubernetes
• Experience supporting data science or machine learning workloads

This role will suit a Senior Data Engineer who enjoys remaining close to the technology and has strong recent experience building production-grade solutions using Python, Databricks, Spark/PySpark and SQL in enterprise environments.