Senior Data Engineer - Hadoop, Spark, Scala & Python

Capgemini; Greater London; Hybrid; GBP 60,000 - 80,000; Full time

Job summary

A leading technology consultancy in Greater London is seeking a highly experienced Data Engineer with expertise in Hadoop, Spark, Scala, and Python. In this role, you will design and develop applications and data pipelines while managing Hadoop clusters. You should have a strong background in ETL processes and cloud solutions. The position offers a flexible work environment, blending company offices, client sites, and home work.

Qualifications

• Deep expertise in Spark, Scala, and Python.
• Strong background in designing large-scale ETL/ELT workflows.
• Experience with Hadoop, including cluster management.

Responsibilities

• Design and develop Hadoop based applications and data pipelines.
• Build, operate, monitor, and troubleshoot Hadoop clusters.
• Write scalable ETL processes using tools like Hive, Pig, and Spark.
• Collaborate with data scientists and analysts to support data needs.
• Evaluate and integrate cloud based big data solutions.

Skills

Spark

Scala

Python

Hadoop

ETL processes

Data ingestion

Tools

Hive

Pig

Sqoop

Flume

Kafka

AWS

GCP

Azure

Job description

A leading technology consultancy in Greater London is seeking a highly experienced Data Engineer with expertise in Hadoop, Spark, Scala, and Python. In this role, you will design and develop applications and data pipelines while managing Hadoop clusters. You should have a strong background in ETL processes and cloud solutions. The position offers a flexible work environment, blending company offices, client sites, and home work.