Aws Data Engineer

Qentelli LLC; Bengaluru; On-site; INR 1,400,000 - 2,200,000; Full time

Job summary

Qentelli LLC is seeking a Senior Data Engineer with 6+ years of experience to design, build, and support scalable data pipelines. The role requires hands-on expertise in Python, PySpark, AWS data services (EMR, S3, Lambda, Step Functions), Delta Lake, and streaming with Kinesis.

You will optimize ETL/ELT processes, handle large datasets, and collaborate with cross-functional teams to deliver reliable data solutions.

Qualifications

• 6+ years of experience in Data Engineering.
• Hands-on Python experience.
• Strong PySpark and Apache Spark experience.
• Hands-on AWS EMR experience.
• Hands-on AWS Lambda experience.
• Hands-on AWS Step Functions experience.
• Experience with Delta Lake.
• Experience with Kinesis Data Streams and Kinesis Data Firehose.
• Strong ETL/ELT and data pipelines understanding.
• Experience working with large datasets and distributed processing.
• Strong troubleshooting and production support skills.
• Experience with AWS S3 and other AWS services.

Responsibilities

• Design, develop, and maintain scalable data pipelines using Python and PySpark.
• Build and manage ETL/ELT processes for large datasets.
• Develop data processing solutions using Apache Spark and AWS EMR.
• Implement serverless data workflows using AWS Lambda and Step Functions.
• Work with AWS S3, Glue, Athena, Redshift, and other AWS data services.
• Build and maintain real-time and streaming data pipelines using Kinesis.
• Work with Delta Lake for reliable data storage and processing.
• Perform data modelling, data transformation, and performance optimization.
• Monitor data pipelines and troubleshoot failures and data quality issues.
• Handle data hydration-related issues quickly and effectively.
• Implement best practices for code quality, version control, and deployment.
• Work with Git and Bitbucket for source control and CI/CD.

Skills

Python

PySpark

Apache Spark

AWS EMR

AWS Lambda

AWS Step Functions

Delta Lake

Kinesis Data Streams

Kinesis Data Firehose

ETL/ELT

Data pipelines

Troubleshooting

S3

Tools

Git

Bitbucket

CI/CD

Job description

Job Summary:

We are looking for a Senior Data Engineer with 6+ years of experience in designing, developing, and supporting scalable data pipelines. The candidate should have strong hands-on experience in Python, PySpark, AWS data services, Spark, EMR, Delta Lake, Kinesis, and workflow orchestration.

The role involves building and maintaining ETL/ELT pipelines, working with large datasets, optimizing data processing, and quickly resolving data-related issues.

Key Responsibilities:

• Design, develop, and maintain scalable and reliable data pipelines using Python and PySpark.
• Build and manage ETL/ELT processes for large and complex datasets.
• Develop data processing solutions using Apache Spark and AWS EMR.
• Implement serverless data workflows using AWS Lambda and Step Functions.
• Work with AWS S3, Glue, Athena, Redshift, and other AWS data services.
• Build and maintain real-time and streaming data pipelines using Amazon Kinesis Data Streams and Kinesis Data Firehose.
• Work with Delta Lake for reliable data storage and processing.
• Perform data modelling, data transformation, and performance optimization.
• Monitor data pipelines and troubleshoot failures and data quality issues.
• Handle and resolve data hydration-related issues quickly and effectively.
• Implement best practices for code quality, version control, and deployment.
• Work with Git and Bitbucket for source code management and CI/CD processes.
• Collaborate with cross-functional teams to understand requirements and deliver data solutions on time.
• Support production data pipelines and resolve issues within agreed timelines.

Mandatory Skills:

• 6+ years of experience in Data Engineering.
• Strong hands-on experience with Python.
• Strong experience with PySpark and Apache Spark.
• Hands-on experience with AWS EMR.
• Hands-on experience with AWS Lambda.
• Hands-on experience with AWS Step Functions.
• Experience with Delta Lake.
• Experience with Amazon Kinesis Data Streams and Kinesis Data Firehose.
• Strong understanding of ETL/ELT processes and data pipelines.
• Experience working with large datasets and distributed data processing.
• Strong troubleshooting and production support skills.
• Experience with AWS S3 and other AWS data services.

Preferred Skills:

• Experience with AWS Glue, Athena, and Redshift.
• Experience with Snowflake and cloud-based data warehousing.
• Knowledge of data modelling and performance optimization.
• Experience with CI/CD processes.
• Hands-on experience with Git and Bitbucket.
• Experience in monitoring and troubleshooting production data pipelines.
• Good understanding of data quality and data validation processes.
• Strong communication and collaboration skills.
• Ability to work independently and deliver solutions within tight timelines.