Senior Data Engineer

GCS Recruitment; Greater London; Hybrid; GBP 136,000 - 178,000; Full time

Job summary

GCS Recruitment seeks a Senior Data Engineer (Python / Spark / Databricks) for a hybrid London-based contract. The role focuses on building scalable data pipelines, Databricks lakehouse architectures, and robust data platforms in a modern cloud environment.

Strong expertise in Python, Databricks, Spark/PySpark and SQL is required, with hands-on delivery across ETL/ELT, data quality and CI/CD practices. Banking domain experience is a plus; three days in the Liverpool Street office each week.

Qualifications

• Strong commercial experience as a Data Engineer or Senior Data Engineer.
• Expert-level Python development skills.
• Extensive hands-on experience with Databricks in enterprise environments.
• Strong Apache Spark and PySpark expertise.
• Advanced SQL development and optimisation skills.
• Experience designing and building large-scale ETL and data integration solutions.
• Experience with modern cloud platforms (Azure, AWS or GCP).
• Experience working in Agile environments.

Responsibilities

• Design, develop and maintain scalable data pipelines using Python, Databricks and PySpark.
• Build and optimise ETL/ELT solutions processing large-scale datasets.
• Develop high-quality, reusable code and engineering frameworks.
• Deliver robust, performant and maintainable data solutions.
• Troubleshoot and resolve complex production and performance issues.
• Support ongoing enhancement and modernisation of enterprise data platforms.
• Develop and support Databricks-based data solutions and Lakehouse architectures.
• Utilise Delta Lake and Spark technologies for scalable data processing.

Skills

Python

PySpark

Spark

SQL

Data Engineering

Agile development

Tools

Databricks

Delta Lake

Airflow

Docker

Kubernetes

CI/CD

Job description

Senior Data Engineer Role (Python / Spark / Databricks) - Hybrid - Contract - Banking

Role - Senior Data Engineer

Rate - £850 p/d (Inside IR35)

Duration - 6 months with very likely extension

Location - Hybrid / Liverpool Street (London) - 3 days per week in a Liverpool Street office

About the Role

Global financial institution seeking a highly skilled Senior Data Engineer to join a growing data engineering function responsible for delivering critical data platforms and analytics capabilities across the business.

This is a genuinely hands-on engineering role, requiring strong expertise in Python, Databricks, Spark/PySpark and SQL. The successful candidate will be actively involved in the design, development, optimisation and support of enterprise-scale data platforms, working alongside engineers, architects and business stakeholders to deliver robust and scalable solutions.

The role offers the opportunity to work on complex data challenges within a modern cloud-based environment, leveraging the latest data engineering technologies and best practices.

Key Responsibilities

Data Engineering & Development

• Design, develop and maintain scalable data pipelines using Python, Databricks and PySpark
• Build and optimise ETL/ELT solutions processing large-scale datasets
• Develop high-quality, reusable code and engineering frameworks
• Deliver robust, performant and maintainable data solutions
• Troubleshoot and resolve complex production and performance issues
• Support the ongoing enhancement and modernisation of enterprise data platforms

Databricks & Data Platform Engineering

• Develop and support Databricks-based data solutions and Lakehouse architectures
• Utilise Delta Lake and Spark technologies to deliver scalable data processing capabilities
• Optimise distributed data processing workloads for performance and reliability
• Implement data quality controls and monitoring solutions
• Contribute to platform stability, scalability and operational excellence

Solution Delivery

• Work closely with business and technology stakeholders to understand requirements and deliver effective solutions
• Participate in Agile delivery processes including sprint planning, refinement and estimation
• Support the delivery of strategic data initiatives across multiple business areas
• Collaborate with architects, engineers and product teams to deliver high-quality outcomes
• Identify and resolve technical risks and implementation challenges

Engineering Best Practice

• Promote software engineering best practices including testing, code reviews and version control
• Support CI/CD adoption and deployment automation
• Contribute to coding standards and engineering governance
• Drive continuous improvement across development and support processes
• Mentor junior engineers and contribute to knowledge sharing within the team

Required Skills & Experience

• Strong commercial experience as a Data Engineer or Senior Data Engineer
• Expert-level Python development skills
• Extensive hands-on experience with Databricks in enterprise environments
• Strong Apache Spark and PySpark expertise
• Advanced SQL development and optimisation skills
• Experience designing and building large-scale ETL and data integration solutions
• Strong understanding of distributed computing concepts and data processing frameworks
• Experience working with modern cloud platforms including Azure, AWS or GCP
• Knowledge of software engineering principles, testing and code quality practices
• Experience working within Agile development environments
• Excellent problem-solving and troubleshooting capabilities

Desirable Experience

• Experience within investment banking, financial services or other highly regulated environments
• Exposure to risk, trading, treasury or regulatory data domains
• Experience with Kafka, streaming technologies or real-time data processing
• Databricks certifications
• Experience with Airflow, Databricks Workflows or similar orchestration tools
• CI/CD pipeline development and DevOps practices
• Docker and Kubernetes
• Experience supporting data science or machine learning workloads

This role will suit a Senior Data Engineer who enjoys remaining close to the technology and has strong recent experience building production-grade solutions using Python, Databricks, Spark/PySpark and SQL in enterprise environments.

GCS is acting as an Employment Business in relation to this vacancy.