Data Engineer: Scalable Pipelines & Azure Spark

Capgemini; Greater London; On-site; GBP 50,000 - 75,000; Full time

Job summary

A global technology consulting firm is seeking a Data Engineer to build and maintain scalable data pipelines and infrastructure. This role requires strong expertise in Azure Databricks and Apache Spark, along with proficiency in Python, Scala, and SQL. Candidates should have 4-6 years of experience in data engineering, a solid understanding of ETL processes, and the ability to work collaboratively across teams. This is a great opportunity to shape your career in a collaborative environment focused on technology and innovation.

Qualifications

• 4-6 years of experience as a Data Engineer or Software Developer in data engineering.
• Strong expertise in Azure Databricks, Apache Spark, and big data technologies.
• Proficient in programming languages such as Python, Scala, and SQL.
• Hands-on experience with ETL processes and cloud platforms (Azure preferred).
• Familiarity with CI/CD practices for data pipelines and DevOps principles.

Responsibilities

• Build and maintain scalable data pipelines and infrastructure for data storage and processing.
• Design, develop, and optimize data workflows using Azure Databricks and Apache Spark.
• Ensure data reliability, scalability, and quality across all systems.
• Integrate multiple data sources into a unified platform for analytics and reporting.
• Monitor, troubleshoot, and enhance data pipeline performance.

Skills

Azure Databricks

Apache Spark

Python

Scala

SQL

ETL processes

CI/CD practices

Data reliability

Job description

A global technology consulting firm is seeking a Data Engineer to build and maintain scalable data pipelines and infrastructure. This role requires strong expertise in Azure Databricks and Apache Spark, along with proficiency in Python, Scala, and SQL. Candidates should have 4-6 years of experience in data engineering, a solid understanding of ETL processes, and the ability to work collaboratively across teams. This is a great opportunity to shape your career in a collaborative environment focused on technology and innovation.