Aws Data Engineer

Happiest Minds Technologies; Dadri; Hybrid; INR 900,000 - 1,300,000; Full time

Job summary

Happiest Minds Technologies is seeking a data engineer to design, build, and optimize large-scale data pipelines on AWS. You will work with S3, Glue, Lambda, and Step Functions, while crafting SQL queries for Redshift and Athena. You will diagnose issues in EMR clusters and Spark jobs and set up robust monitoring with CloudWatch.

The role focuses on delivering scalable, reliable data processing solutions and collaborating with cross-functional teams to meet business needs.

Qualifications

• Design, develop, test, deploy, and maintain large-scale data pipelines using AWS services.
• Develop complex SQL queries to extract insights from data in Redshift and Athena.
• Troubleshoot issues related to EMR clusters, Spark jobs, and data pipelines.

Responsibilities

• Design, develop, test, deploy, and maintain large-scale data pipelines on AWS (S3, Glue, Lambda, Step Functions).
• Collaborate with cross-functional teams to gather requirements and design solutions.
• Develop complex SQL queries to extract insights from data in Redshift and Athena.
• Troubleshoot issues related to EMR clusters, Spark jobs, and data pipelines.
• Ensure high availability and scalability by implementing monitoring with CloudWatch.

Skills

Data pipeline design

SQL querying

Troubleshooting

Cloud monitoring

AWS data services

Tools

Amazon EMR

Apache Spark

Amazon Redshift

Amazon Athena

CloudWatch

Job description

Roles and Responsibilities

• Design, develop, test, deploy, and maintain large-scale data pipelines using AWS services such as S3, Glue, Lambda, Step Functions.
  
• Collaborate with cross-functional teams to gather requirements and design solutions that meet business needs.
  
• Develop complex SQL queries to extract insights from large datasets stored in Amazon Redshift and Athena.
  
• Troubleshoot issues related to EMR clusters, Spark jobs, and data pipeline failures.
  
• Ensure high availability and scalability of the system by implementing monitoring tools like CloudWatch.