Senior AWS Data Architect & ETL/Testing Lead

Capgemini; New York (NY); On-site; USD 150,000 - 200,000; Full time

Job summary

Capgemini is seeking a data testing and engineering professional to design scalable AWS data architectures and manage end-to-end testing of data pipelines. You will build ETL/ELT processes, real-time ingestion, and data quality checks, leveraging Spark, Python, and Scala.

Collaboration with product and data teams is essential for quality delivery. The role emphasizes leadership of a testing team, CI/CD quality gates, and ongoing process improvements across ingestion, transformation, and

Qualifications

• Experience designing scalable data architectures on AWS.
• Building and maintaining ETL/ELT pipelines for structured and unstructured data.
• Developing real-time and batch ingestion pipelines with AWS services.
• Proficiency in data quality checks, validation and automated error handling.

Responsibilities

• Design and implement scalable data architectures using AWS services (S3, Redshift, Glue, Athena, EMR, DynamoDB).
• Build, optimize, and maintain ETL/ELT pipelines for structured and unstructured data.
• Develop real time and batch ingestion pipelines using AWS Glue, Lambda, Kinesis, or Kafka
• Utilize Spark (AWS EMR/Glue), Python, or Scala to build transformation workflows.
• Implement data quality checks, validation rules, and automated error handling mechanisms.
• Ensure data is optimized for analytics, BI, and ML use cases.
• Build and maintain data lakes on Amazon S3 and data warehouses on Redshift/Snowflake.
• Create and optimize database schemas, partitioning, compression, and performance tuning.
• Manage metadata, cataloging, and data lineage via AWS Glue Data Catalog or similar tools.
• Lead end to end testing strategy, covering functional, integration, regression, performance, and UAT cycles.
• Build and manage a testing team, including test engineers, automation engineers, and QA analysts.
• Develop test plans, test cases, and automation frameworks for data pipelines, ETL processes, APIs, and cloud platforms.
• Ensure data validation, reconciliation, and quality assurance across ingestion, transformation, and consumption layers.
• Implement automated testing solutions using tools such as PyTest, Selenium, JMeter, Postman, Robot Framework, or AWS-native testing utilities.
• Establish quality gates within CI/CD pipelines to ensure reliable deployments.
• Collaborate closely with Product Owners, Data Engineering teams, and Business SMEs to ensure high-quality delivery.
• Drive continuous improvement in test processes, automation coverage, and defect reduction.
• Provide regular QA metrics, defect summaries, and quality dashboards to leadership and stakeholders.
• Design and implement scalable data architectures using AWS services (S3, Redshift, Glue, Athena, EMR, DynamoDB).
• Build, optimize, and maintain ETL/ELT pipelines for structured and unstructured data.
• Develop real time and batch ingestion pipelines using AWS Glue, Lambda, Kinesis, or Kafka
• Utilize Spark (AWS EMR/Glue), Python, or Scala to build transformation workflows.
• Implement data quality checks, validation rules, and automated error handling mechanisms.
• Ensure data is optimized for analytics, BI, and ML use cases.
• Build and maintain data lakes on Amazon S3 and data warehouses on Redshift/Snowflake.
• Create and optimize database schemas, partitioning, compression, and performance tuning.
• Manage metadata, cataloging, and data lineage via AWS Glue Data Catalog or similar tools.

Skills

AWS

Testing

S3

Redshift

Glue

Athena

EMR

DynamoDB

Tools

Python

Scala

Spark

Job description

Capgemini is seeking a data testing and engineering professional to design scalable AWS data architectures and manage end-to-end testing of data pipelines. You will build ETL/ELT processes, real-time ingestion, and data quality checks, leveraging Spark, Python, and Scala.

Collaboration with product and data teams is essential for quality delivery. The role emphasizes leadership of a testing team, CI/CD quality gates, and ongoing process improvements across ingestion, transformation, and