Senior Data Engineer

GCS; Greater London; Hybrid; GBP 37,638,000 - 39,852,000; Full time

Job summary

GCS is seeking a Senior Data Engineer to design, develop and optimize scalable data pipelines in a hybrid London setting. The role focuses on Python, Databricks, Spark/PySpark and SQL within a modern cloud environment.

You will work closely with engineers and stakeholders, participate in Agile delivery, and contribute to platform stability and data quality through best practices, testing and CI/CD.

Qualifications

• Extensive hands-on Databricks experience in enterprise environments.
• Strong Python development skills and SQL optimization.
• Experience with Azure, AWS or GCP and Agile teams.

Responsibilities

• Design, develop and maintain scalable data pipelines using Python, Databricks and PySpark.
• Build and optimise ETL/ELT solutions processing large-scale datasets.
• Develop high-quality, reusable code and engineering frameworks.
• Deliver robust, performant and maintainable data solutions.
• Troubleshoot and resolve complex production and performance issues.

Skills

Python

Databricks

Spark/PySpark

SQL

Cloud platforms

Agile

Data Engineering

Problem solving

Tools

Airflow

Docker

Kubernetes

Job description

Senior Data Engineer Role (Python / Spark / Databricks) - Hybrid - Contract - Banking

Role - Senior Data Engineer

Rate - £850 p/d (Inside IR35)

Duration - 6 months with very likely extension

Location - Hybrid / Liverpool Street (London) - 3 days per week in a Liverpool Street office

About the Role

Global financial institution seeking a highly skilled Senior Data Engineer to join a growing data engineering function responsible for delivering critical data platforms and analytics capabilities across the business.

This is a genuinely hands‑on engineering role, requiring strong expertise in Python, Databricks, Spark/PySpark and SQL. The successful candidate will be actively involved in the design, development, optimisation and support of enterprise‑scale data platforms, working alongside engineers, architects and business stakeholders to deliver robust and scalable solutions.

The role offers the opportunity to work on complex data challenges within a modern cloud‑based environment, leveraging the latest data engineering technologies and best practices.

Key Responsibilities

Data Engineering & Development

• Design, develop and maintain scalable data pipelines using Python, Databricks and PySpark
• Build and optimise ETL/ELT solutions processing large‑scale datasets
• Develop high‑quality, reusable code and engineering frameworks
• Deliver robust, performant and maintainable data solutions
• Troubleshoot and resolve complex production and performance issues

Databricks & Data Platform Engineering

• Develop and support Databricks‑based data solutions and Lakehouse architectures
• Utilise Delta Lake and Spark technologies to deliver scalable data processing capabilities
• Optimise distributed data processing workloads for performance and reliability
• Implement data quality controls and monitoring solutions
• Contribute to platform stability, scalability and operational excellence

Solution Delivery

• Work closely with business and technology stakeholders to understand requirements and deliver effective solutions
• Participate in Agile delivery processes including sprint planning, refinement and estimation
• Support the delivery of strategic data initiatives across multiple business areas
• Collaborate with architects, engineers and product teams to deliver high‑quality outcomes
• Identify and resolve technical risks and implementation challenges

Engineering Best Practice

• Promote software engineering best practices including testing, code reviews and version control
• Support CI/CD adoption and deployment automation
• Contribute to coding standards and engineering governance
• Drive continuous improvement across development and support processes
• Mentor junior engineers and contribute to knowledge sharing within the team

Required Skills & Experience

• Strong commercial experience as a Data Engineer or Senior Data Engineer
• Expert‑level Python development skills
• Extensive hands‑on experience with Databricks in enterprise environments
• Strong Apache Spark and PySpark expertise
• Advanced SQL development and optimisation skills
• Experience designing and building large‑scale ETL and data integration solutions
• Strong understanding of distributed computing concepts and data processing frameworks
• Experience working with modern cloud platforms including Azure, AWS or GCP
• Knowledge of software engineering principles, testing and code quality practices
• Experience working within Agile development environments
• Excellent problem‑solving and troubleshooting capabilities

Desirable Experience

• Experience within investment banking, financial services or other highly regulated environments
• Exposure to risk, trading, treasury or regulatory data domains
• Experience with Kafka, streaming technologies or real‑time data processing
• Databricks certifications
• Experience with Airflow, Databricks Workflows or similar orchestration tools
• CI/CD pipeline development and DevOps practices
• Docker and Kubernetes
• Experience supporting data science or machine learning workloads

This role will suit a Senior Data Engineer who enjoys remaining close to the technology and has strong recent experience building production‑grade solutions using Python, Databricks, Spark/PySpark and SQL in enterprise environments.

GCS is acting as an Employment Business in relation to this vacancy