AWS Data Engineer

Trantor; Chandigarh; Sur place; INR 1 800 000 - 3 000 000; Plein temps

Résumé du poste

Trantor in India seeks an experienced AWS Data Engineer to design and optimize scalable data pipelines on AWS using S3, Glue, Lambda, and Step Functions. You will handle ingestion, transformation, and orchestration across sources, ensuring reliability and cost efficiency.

The role requires hands-on expertise with Python, SQL, PySpark, and data lake architectures, with a focus on performance tuning and production troubleshooting. Opportunity to shape scalable data solutions in a dynamic team.

Qualifications

• 6+ years of experience in AWS data engineering.
• Hands-on with end-to-end AWS data pipelines.
• Strong knowledge of S3, Glue, Lambda, and Step Functions.

Responsabilités

• Design and develop end-to-end AWS data pipelines using S3, Glue, Lambda, and Step Functions.
• Implement data ingestion from files, databases, APIs, and event-driven sources.
• Build data validation, cleansing, transformation, and enrichment workflows using Python, SQL, and PySpark.
• Optimize Athena queries through partitioning, Parquet/ORC formats, compression, and cost-effective data scanning.
• Apply Spark optimization techniques, including broadcast joins, data skew handling, and salting.
• Implement S3 storage lifecycle policies and optimize storage costs.
• Configure workflow retries, exception handling, idempotency, and failure recovery.
• Monitor production pipelines, troubleshoot failures, and ensure data quality, reliability, and performance.
• Maintain technical documentation and support production deployments.

Connaissances

AWS data engineering

Python

SQL

PySpark

ETL/ELT pipelines

Data lake architectures

Spark optimization

S3

Glue

Step Functions

Athena

Data quality monitoring

Outils

S3

Glue

Lambda

Step Functions

Athena

Description du poste

We are looking for an 6+years AWS Data Engineer with strong hands-on expertise in building, developing, and optimizing scalable data pipelines on AWS. The candidate should have practical experience with data ingestion, transformation, orchestration, performance tuning, and production troubleshooting.

Key Responsibilities

• Design and develop end-to-end AWS data pipelines using S3, Glue, Lambda, and Step Functions.


• Implement data ingestion from files, databases, APIs, and event-driven sources.


• Build data validation, cleansing, transformation, and enrichment workflows using Python, SQL, and PySpark.


• Optimize Athena queries through partitioning, Parquet/ORC formats, compression, and cost-effective data scanning.


• Apply Spark optimization techniques, including broadcast joins, data skew handling, and salting.


• Implement S3 storage lifecycle policies and optimize storage costs.


• Configure workflow retries, exception handling, idempotency, and failure recovery.


• Monitor production pipelines, troubleshoot failures, and ensure data quality, reliability, and performance.


• Maintain technical documentation and support production deployments.

Required Skills

• Strong hands-on experience with AWS services: S3, Glue, Lambda, Step Functions, and Athena.


• Proficiency in Python, SQL, and PySpark.


• Experience designing end-to-end ETL/ELT pipelines and data lake architectures.


• Strong understanding of S3 raw, curated, and consumption layers, partitioning, and storage formats.


• Experience with Spark performance optimization, broadcast joins, and data skew/salting.


• Experience of AWS Step Functions Standard and Express workflows, retries, error handling, and orchestration.


• Experience with production monitoring, troubleshooting, data validation, and pipeline recovery.


• Strong understanding of data processing trade-offs, scalability, and cost optimization.