AWS Data Engineer

Careernet; Kolkata District, Bengaluru; On-site; INR 900,000 - 1,500,000; Full time

Job summary

Careernet is seeking an experienced Data Engineer to design and optimize scalable data pipelines on AWS. You will build ETL/ELT workflows with Glue, Lambda, EMR, and Step Functions and enable data ingestion via Kinesis, Kafka, and SQS.

You will contribute to data lakes and warehouses using S3, Glue Data Catalog, Redshift, and Snowflake, with a focus on data quality, monitoring, and CI/CD practices.

Qualifications

• 6+ years of hands-on AWS data engineering experience.
• Strong Python, PySpark and SQL for data transformations.
• Experience with data lakes/warehouses on AWS (S3, Redshift, Glue Data Catalog).
• Familiarity with CI/CD, data quality, monitoring and logging.

Responsibilities

• Design, build, and maintain scalable data pipelines on AWS.
• Develop ETL/ELT workflows using Glue, Lambda, EMR, and Step Functions.
• Develop data ingestion frameworks using Kinesis, Kafka, and SQS.
• Build and optimize data lakes using S3, Glue Data Catalog, and Lake Formation.
• Develop data warehousing solutions using Redshift and Snowflake.
• Implement data quality checks, monitoring, logging, and CI/CD deployments.
• Collaborate with analytics and business teams to deliver high-quality datasets.
• Design, develop, and deploy solutions using appropriate tools and coding standards.
• Coordinate with process owners to understand as-is processes and automate workflows.
• Troubleshoot day-to-day data/automation issues and maintain documentation.

Education

Bachelor's or Master's degree in Computer Science, Information Technology, Engineering, or related field

Tools

AWS Glue

S3

Lambda

IAM

Step Functions

PySpark

SQL

Glue Data Catalog

Redshift

Spark/EMR

Kinesis/Kafka/SQS

Job description

Key Skills: IAM, AWS, Python, AWS Lambda, Data Engineer, AWS S3, Spark

Roles and Responsibilities:

• Design, build, and maintain scalable data pipelines on AWS.
• Develop ETL/ELT workflows using AWS Glue, Lambda, EMR, and Step Functions.
• Develop data ingestion frameworks using Kinesis, Kafka, and SQS.
• Build and optimize data lakes using S3, Glue Data Catalog, and Lake Formation.
• Develop and maintain data warehousing solutions using Redshift and Snowflake.
• Implement data quality checks, monitoring, logging, and CI/CD deployments.
• Collaborate with analytics and business teams to deliver high-quality datasets.
• Design, develop, and deploy solutions using appropriate tools, design principles, and coding standards.
• Configure robotics processes and objects using core workflow principles, ensuring solutions are maintainable and easy to understand.
• Understand existing processes and manage changing requirements through structured change control processes.
• Troubleshoot day-to-day issues in data and automation processes and provide timely resolutions.
• Maintain proper documentation for solutions, test procedures, and scenarios throughout UAT and Production phases.
• Coordinate with process owners and business stakeholders to understand as-is processes and design appropriate automation workflows.

Skills Required:

• 6+ years of strong hands-on experience with AWS Glue, S3, Lambda, IAM, and Step Functions.
• Strong proficiency in Python, PySpark, SQL, and data transformation logic.
• Experience with Redshift, Athena, and Glue Data Catalog.
• Strong knowledge of Apache Spark and distributed data processing using EMR/Glue.
• Experience with data ingestion technologies such as Kinesis, Kafka, or SQS.
• Good understanding of data modeling, data lake architecture, and data engineering best practices.
• Experience implementing data quality, monitoring, logging, and CI/CD processes.
• Familiarity with Terraform or CloudFormation for Infrastructure as Code (IaC) is an added advantage.
• Strong problem-solving and analytical skills.
• Good communication and stakeholder management skills.

Education:

Bachelor's or Master's degree in Computer Science, Information Technology, Engineering, or a related field.