AWS Data Engineer

Cognizant; Chennai District; On-site; INR 1,200,000 - 2,500,000; Full time

Job summary

Cognizant in Chennai, India is seeking a data engineer to design, develop, and maintain scalable data pipelines using Python and PySpark, and to build ETL/ELT workflows.

The role focuses on Spark optimization, cloud-native data solutions on AWS, and managing Data Lakes on AWS, while ensuring data quality, security, and governance through collaboration with cross-functional teams. You will also participate in code reviews, testing, and deployment documentation.

Qualifications

• Proficient in Python programming.
• Hands-on PySpark experience.
• Strong SQL skills with query optimization.

Responsibilities

• Design, develop, and maintain scalable data pipelines using Python and PySpark.
• Build and manage ETL/ELT workflows for processing structured and unstructured data.
• Develop and optimize Spark applications to handle large-scale datasets efficiently.
• Implement cloud-native data solutions using AWS services.
• Design and maintain Data Lakes on AWS.
• Perform data ingestion from multiple sources including APIs, databases, files, and streaming platforms.
• Monitor, troubleshoot, and optimize data workflows for performance and reliability.
• Ensure data quality, integrity, security, and governance standards are followed.
• Implement automation and orchestration of data workflows using scheduling tools.
• Collaborate with cross-functional teams for requirement gathering and solution design.
• Conduct code reviews, unit testing, and performance tuning.
• Prepare technical documentation and deployment procedures.
• Support production incidents and perform root cause analysis.

Skills

Python

PySpark

SQL

Job description

Key Responsibilities

• Design, develop, and maintain scalable data pipelines using Python and PySpark.
• Build and manage ETL/ELT workflows for processing structured and unstructured data.
• Develop and optimize Spark applications to handle large-scale datasets efficiently.
• Implement cloud-native data solutions using AWS services.
• Design and maintain Data Lakes on AWS.
• Perform data ingestion from multiple sources including APIs, databases, files, and streaming platforms.
• Monitor, troubleshoot, and optimize data workflows for performance and reliability.
• Ensure data quality, integrity, security, and governance standards are followed.
• Implement automation and orchestration of data workflows using scheduling tools.
• Collaborate with cross-functional teams for requirement gathering and solution design.
• Conduct code reviews, unit testing, and performance tuning.
• Prepare technical documentation and deployment procedures.
• Support production incidents and perform root cause analysis.

Mandatory Skills

• Strong expertise in Python
• Hands-on experience with PySpark
• Proficiency in SQL and query optimization