Data Engineer

Capgemini Hong Kong Limited; Hong Kong; On-site; HKD 480,000 - 760,000; Full time

Job summary

Capgemini Hong Kong Limited is seeking a data engineer to build and maintain batch and streaming data pipelines end-to-end. You will ingest, transform, validate and publish data, and develop reusable transformation patterns for analytics and operations.

The role requires hands-on Python/Java, SQL, Spark, Airflow, and cloud experience, with an emphasis on data quality, performance optimization, and collaboration with upstream/downstream teams.

Qualifications

• 5+ years of data engineering delivery experience.
• Strong Python/Java with advanced SQL and data modeling.
• Experience with Spark and orchestration like Airflow.
• Experience with cloud data platforms (Azure/AWS/GCP).
• Familiar with streaming platforms and data governance basics.

Responsibilities

• Build end-to-end batch and streaming pipelines.
• Develop reusable data transformation patterns.
• Design, test, deploy, and support from requirements to production.
• Implement data quality checks and monitoring.
• Collaborate with upstream/downstream teams.
• Maintain documentation and runbooks.
• Troubleshoot production issues and perform root-cause analysis.

Skills

Python

Java

SQL

Data modelling

Spark

Airflow

Kafka

Git

CI/CD

Observability

Azure/AWS/GCP

Tools

Spark

Airflow

Kafka

Git

CI/CD

Azure

AWS

GCP

Job description

Build and maintain batch and/or streaming data pipelines end-to-end (ingest, transform, validate, publish)

Develop reusable data transformation patterns and curated datasets for analytics and operational use cases

Work from requirements through to production delivery: design, build, test, deploy, and support

Implement data quality checks, reconciliations, and monitoring/alerting to keep data trustworthy

Optimize performance and cost (query tuning, partitioning, efficient compute usage)

Maintain clear documentation, data definitions, and runbooks for supported pipelines

Troubleshoot production issues, perform root cause analysis, and implement permanent fixes

Collaborate with upstream/downstream teams to resolve data issues and improve interfaces/contracts

Follow security and governance expectations for sensitive data (access controls, auditability)

Required skills and experience

• Minimum 5 years of strong hands-on experience delivering data engineering solutions
• Strong programming skills in Python and/or Java experience, with advanced SQL and solid data modelling skills
• Experience with distributed processing (e.g. Spark) and orchestration (e.g. Airflow or equivalent)
• Experience with streaming/event platforms (e.g. Kafka/PubSub) is beneficial; ability to learn quickly if not
• Strong engineering discipline: Git, code reviews, automated testing, CI/CD, and observability
• Experience working with cloud data platforms (Azure/AWS/GCP) and lake/lakehouse/warehouse patterns
• Proven ability to manage multiple data requests, priorities effectively, and deliver at pace

Capgemini is a global business and technology transformation partner, helping organizations to accelerate their dual transition to a digital and sustainable world. It is a responsible and diverse group of 340,000 team members in more than 50 countries. With its strong over 55-year heritage, Capgemini is trusted by its clients to unlock the value of technology to address the entire breadth of their business needs. It delivers end-to-end services and solutions leveraging strengths from strategy and design to engineering, all fueled by its market leading capabilities in AI, cloud and data, combined with its deep industry expertise and partner ecosystem.

Capgemini is a global business and technology transformation partner, helping organizations to accelerate their dual transition to a digital and sustainable world. It is a responsible and diverse group of 340,000 team members in more than 50 countries. With its strong over 55-year heritage, Capgemini is trusted by its clients to unlock the value of technology to address the entire breadth of their business needs. It delivers end-to-end services and solutions leveraging strengths from strategy and design to engineering, all fueled by its market leading capabilities in AI, cloud and data, combined with its deep industry expertise and partner ecosystem.