Senior Data Engineer: Cloud Data Platforms & Pipelines

Capgemini; New York (NY); On-site; USD 85,000 - 95,000; Full time

Benefits offered by this job

Paid time off

Company holidays

Personal days

Health insurance

401(k) retirement plan

Life and disability insurance

Employee assistance program

Job summary

Capgemini is seeking a Senior Data Engineer in New York to build and optimize data pipelines using Python, PySpark, and AWS services. You will design batch data ingestion and work with Glue, Lambda, S3, Redshift and Databricks, ensuring scalable, secure data platforms and governance.

You will collaborate across teams, contribute to DevOps CI/CD, and leverage Kafka/Spark Streaming and DBT for real-time and transformation needs.

Qualifications

• Hands-on programming in Python and PySpark.
• Advanced SQL with query optimization techniques.
• Strong understanding of AWS and distributed computing paradigms.
• Design and develop data ingestion programs to process large data sets in batch mode using Glue, Lambda, S3, Redshift and Databricks.
• Basic understanding of IAM roles, security and AWS infrastructure.
• Experience with large-scale data platforms on AWS.
• Familiar with DevOps pipelines using Jenkins and GitLab and strong data modeling and data architecture knowledge.
• Sound knowledge of data governance principles and tools (Alation, Glue, Data Quality Mesh).
• Ability to propose solution architecture for diverse applications.
• Nice to have real-time ingestion using Kafka, Spark Streaming and related technologies; knowledge of DBT.

Responsibilities

• Develop and maintain data ingestion pipelines for large datasets.
• Collaborate on batch and near-real-time data processing solutions.
• Contribute to data governance, architecture, and modeling.
• Support DevOps practices and CI/CD for data products.
• Advise on suitable technologies and infrastructure choices.

Skills

Python

PySpark

SQL

AWS

Data ingestion

IAM security

DevOps

Data governance

Solution architecture

Kafka

Tools

Glue

Lambda

S3

Redshift

Databricks

Kafka

Spark Streaming

DBT

Job description

Capgemini is seeking a Senior Data Engineer in New York to build and optimize data pipelines using Python, PySpark, and AWS services. You will design batch data ingestion and work with Glue, Lambda, S3, Redshift and Databricks, ensuring scalable, secure data platforms and governance.

You will collaborate across teams, contribute to DevOps CI/CD, and leverage Kafka/Spark Streaming and DBT for real-time and transformation needs.