Data Engineer

Quest Global; Hyderabad; On-site; INR 1,200,000 - 1,600,000; Full time

Job summary

Quest Global is seeking a skilled Data Engineer in Hyderabad to design, develop, and maintain scalable data pipelines and ETL processes, leveraging modern big data tools.

You will work with PySpark, Python, and SQL on AWS services (Redshift, EMR, Airflow) to deliver robust data solutions, ensure data quality, and collaborate with Data Analysts, Data Scientists, and business stakeholders.

Qualifications

• Strong proficiency in Python.
• Hands-on experience with PySpark.
• Advanced knowledge of SQL.
• Experience with AWS Services: Redshift, EMR, Airflow.
• Experience in data warehousing and ETL development.
• Strong problem-solving and analytical skills.
• Excellent communication and collaboration abilities.

Responsibilities

• Design, develop, and optimize scalable data pipelines and ETL processes.
• Build and maintain data processing workflows using PySpark and Python.
• Develop and manage data solutions on AWS cloud services.
• Work with large datasets to ensure data quality, reliability, and performance.
• Create and optimize SQL queries for data extraction, transformation, and analysis.
• Implement and manage workflow orchestration using Apache Airflow.
• Utilize AWS Redshift for data warehousing and analytics.
• Manage and optimize big data processing using AWS EMR.
• Collaborate with cross-functional teams including Data Analysts, Data Scientists, and Business Stakeholders.
• Ensure adherence to data governance, security, and best practices.

Skills

Python

PySpark

SQL

Tools

AWS

Redshift

EMR

Airflow

Job description

Work Mode: 5 Days Work From Office (WFO)

Notice Period: Immediate Joiners / Serving Notice Period Preferred

About the Role

We are looking for a skilled and passionate Data Engineer to join our growing team. The ideal candidate will have strong experience in designing, developing, and maintaining scalable data pipelines, working with big data technologies, and leveraging cloud platforms to build robust data solutions.

Key Responsibilities

• Design, develop, and optimize scalable data pipelines and ETL processes.
• Build and maintain data processing workflows using PySpark and Python.
• Develop and manage data solutions on AWS cloud services.
• Work with large datasets to ensure data quality, reliability, and performance.
• Create and optimize SQL queries for data extraction, transformation, and analysis.
• Implement and manage workflow orchestration using Apache Airflow.
• Utilize AWS Redshift for data warehousing and analytics.
• Manage and optimize big data processing using AWS EMR.
• Collaborate with cross-functional teams including Data Analysts, Data Scientists, and Business Stakeholders.
• Ensure adherence to data governance, security, and best practices.

Required Skills

Strong proficiency in Python

Hands-on experience with PySpark

Advanced knowledge of SQL

Experience with AWS Services:

• Redshift
• EMR
• Airflow

Experience in data warehousing and ETL development

Strong problem-solving and analytical skills

Excellent communication and collaboration abilities

Preferred Qualifications

• Experience working with large-scale distributed data processing systems.
• Knowledge of data modeling and performance tuning.
• Familiarity with CI/CD pipelines and version control tools such as Git.