Senior Data Engineer

Quest Global; Hyderabad; On-site; INR 1,500,000 - 2,800,000; Full time

Job summary

Quest Global is seeking a skilled Data Engineer with 5+ years of experience to design, build, and optimize large-scale data pipelines. You will work across data migration projects, data warehousing, and cloud-based solutions on AWS, collaborating with data scientists and business teams to deliver reliable, secure data products.

The role emphasizes data quality, scalability, and cost-efficient cloud resource management, with exposure to Hadoop, Spark, and AWS services such as Redshift, S3, Glue,

Qualifications

• Master's degree or equivalent in CS/Engineering/Analytics/Math/Statistics/IT.
• 5+ years of data engineering and data migration experience.
• Proficient in SQL and Python for data processing and analysis.
• Strong experience in data modeling, warehousing, and building data pipelines.
• Hands-on with Hadoop, Spark, and related big data tech.
• AWS services including Redshift, S3, Glue, EMR, Kinesis, Firehose, Lambda, IAM.
• ETL development best practices and principles.

Responsibilities

• Design, build, and maintain scalable data pipelines and ETL processes.
• Develop and optimize data models and data warehouse architectures.
• Implement and manage big data technologies and cloud-based data solutions.
• Perform data migration, transformation, and integration from multiple sources.
• Collaborate with data scientists, analysts, and business teams to understand data needs and deliver solutions.
• Ensure data quality, consistency, and security across pipelines and storage.
• Optimize performance and manage cost-efficient AWS resources.

Skills

SQL

Python

Data modeling

Data warehousing

ETL pipelines

Hadoop

Spark

AWS

Redshift

S3

Glue

EMR

Lambda

Education

Master's degree in CS/Engineering/Analytics

Tools

Airflow

QuickSight

Tableau

Power BI

Job description

We are looking for a skilled and experienced Data Engineer with over 5 years of experience in data engineering and data migration projects. The ideal candidate should possess strong expertise in SQL, Python, data modeling, data warehousing, and ETL pipeline development. Experience with big data tools like Hadoop and Spark, along with AWS services such as Redshift, S3, Glue, EMR, and Lambda, is essential. This role provides an excellent opportunity to work on large-scale data solutions, enabling data-driven decision-making and operational excellence.

Key Responsibilities:

• Design, build, and maintain scalable data pipelines and ETL processes.
• Develop and optimize data models and data warehouse architectures.
• Implement and manage big data technologies and cloud-based data solutions.
• Perform data migration, data transformation, and integration from multiple sources.
• Collaborate with data scientists, analysts, and business teams to understand data needs and deliver solutions.
• Ensure data quality, consistency, and security across all data pipelines and storage systems.
• Optimize performance and manage cost-efficient AWS cloud resources.

Basic Qualifications:

• Master's degree in computer science, Engineering, Analytics, Mathematics, Statistics, IT, or equivalent.
• 5+ years of experience in Data Engineering and data migration projects.
• Proficient in SQL and Python for data processing and analysis.
• Strong experience in data modeling, data warehousing, and building data pipelines.
• Hands-on experience with big data technologies like Hadoop, Spark, etc.
• Expertise in AWS services including Redshift, S3, AWS Glue, EMR, Kinesis, Firehose, Lambda, and IAM.
• Understanding of ETL development best practices and principles.

Preferred Qualifications:

• Knowledge of data security and data privacy best practices.
• Experience with DevOps and CI/CD practices related to data workflows.
• Familiarity with data lake architectures and real-time data streaming.
• Strong problem-solving abilities and attention to detail.
• Excellent verbal and written communication skills.
• Ability to work independently and in a team-oriented environment.

Good to Have:

• Experience with orchestration tools like Airflow or Step Functions.
• Exposure to BI/Visualization tools like QuickSight, Tableau, or Power BI.
• Understanding of data governance and compliance standards.