Data Engineer ( Hadoop / Spark / Scala / Python)

Capgemini; Greater London; Hybrid; GBP 60,000 - 80,000; Full time

Job summary

A leading technology consultancy in Greater London is seeking a highly experienced Data Engineer with expertise in Hadoop, Spark, Scala, and Python. In this role, you will design and develop applications and data pipelines while managing Hadoop clusters. You should have a strong background in ETL processes and cloud solutions. The position offers a flexible work environment, blending company offices, client sites, and home work.

Qualifications

• Deep expertise in Spark, Scala, and Python.
• Strong background in designing large-scale ETL/ELT workflows.
• Experience with Hadoop, including cluster management.

Responsibilities

• Design and develop Hadoop based applications and data pipelines.
• Build, operate, monitor, and troubleshoot Hadoop clusters.
• Write scalable ETL processes using tools like Hive, Pig, and Spark.
• Collaborate with data scientists and analysts to support data needs.
• Evaluate and integrate cloud based big data solutions.

Skills

Spark

Scala

Python

Hadoop

ETL processes

Data ingestion

Tools

Hive

Pig

Sqoop

Flume

Kafka

AWS

GCP

Azure

Job description

We are seeking a highly experienced Data Engineer ( Hadoop / Spark / Scala / Python) to join our Financial Services Data Practice. You will have deep hands‑on expertise in Hadoop, Spark, Scala, Python.

The places that you work from day to day will vary according to your role, your needs, and those of the business; it will be a blend of Company offices, client sites, and your home.

Your Role

• Design and develop Hadoop based applications and data pipelines
• Build operate monitor and troubleshoot Hadoop clusters
• Write scalable ETL processes using tools like Hive Pig and Spark
• Develop and maintain data ingestion processes using Sqoop Flume or Kafka
• Optimize MapReduce jobs and manage HDFS storage
• Collaborate with data scientists and analysts to support data needs
• Ensure data security and compliance with organizational policies
• Create and maintain technical documentation and playbooks
• Evaluate and integrate cloud based big data solutions AWS GCP Azure

Your Skills

• Data/software engineering experience with deep expertise in Spark, Scala, and Python.
• Strong background in designing large‑scale ETL/ELT workflows and distributed compute optimization.

We are a Disability Confident Employer

Capgemini is proud to be a Disability Confident Employer (Level 2) under the UK Government’s Disability Confident scheme. As part of our commitment to inclusive recruitment, we will offer an interview to all candidates who:

• Declare they have a disability, and
• Meet the minimum essential criteria for the role.