Data Engineer

Quest Global; Maharashtra; On-site; INR 1,200,000 - 1,600,000; Full time

Job summary

Quest Global in Pune, India, is seeking a data engineer to join our analytics team. You will develop scalable data pipelines using Spark, Scala, and PySpark, processing large-scale datasets and collaborating with customer teams.

The role emphasizes performance optimization, data quality, and rapid delivery in a fast-paced environment. You will work with cloud storage (S3/HDFS), streaming (Spark Streaming), and Airflow, contributing to end-to-end data solutions.

Qualifications

• Strong hands-on experience with Apache Spark.
• Proficiency in Scala and/or Python (PySpark).
• Good understanding of Spark internals (RDD, DataFrame, Dataset APIs).
• Experience with data formats like Parquet, Avro, JSON.
• Familiarity with distributed systems and big data concepts.
• Strong SQL skills.
• Experience with cloud platforms (AWS preferred – S3, EMR, Glue, Kinesis, Firehose, Hive).
• Knowledge of performance tuning and optimization techniques.
• Experience with CI/CD pipelines.
• Exposure to streaming frameworks (Spark Streaming).
• Familiarity with workflow orchestration tools like Apache Airflow.

Responsibilities

• The candidate will cooperate with both Quest Global and Quest Global customer teams within a collaborative development framework.
• Write efficient and reusable code in Scala and/or Python (PySpark)
• Process large-scale structured and unstructured datasets
• Perform data transformations, aggregations, and joins across multiple sources
• Optimize Spark jobs for performance and resource utilization
• Work with distributed storage systems like S3 / HDFS
• Debug and troubleshoot production data issues
• Collaborate with data engineers, analysts, and stakeholders
• Ensure data quality, consistency, and reliability
• Collaborating with cross-functional teams to analyze, design, and implement new applications.
• Ensuring optimal performance, quality, and responsiveness of the application/services.

Skills

Apache Spark

Scala

Python (PySpark)

SQL

Cloud platforms

CI/CD

Spark Streaming

Airflow

Data wrangling

Big data concepts

Tools

Parquet

Avro

JSON

S3

HDFS

Hive

EMR

Glue

Kinesis

Airflow

Job description

Job Requirements

At Quest Global, it’s not just what we do but how and why we do it that makes us different. With over 25 years as an engineering services provider, we believe in the power of doing things differently to make the impossible possible. Our people are driven by the desire to make the world a better place—to make a positive difference that contributes to a brighter future. We bring together technologies and industries, alongside the contributions of diverse individuals who are empowered by an intentional workplace culture, to solve problems better and faster.

At Quest Global, it’s not just what we do but how and why we do it that makes us different. With over 25 years as an engineering services provider, we believe in the power of doing things differently to make the impossible possible. Our people are driven by the desire to make the world a better place—to make a positive difference that contributes to a brighter future. We bring together technologies and industries, alongside the contributions of diverse individuals who are empowered by an intentional workplace culture, to solve problems better and faster.

Key Responsibilities

• The candidate will cooperate with both Quest Global and Quest Global customer teams within a collaborative development framework.
• Write efficient and reusable code in Scala and/or Python (PySpark)
• Process large-scale structured and unstructured datasets
• Perform data transformations, aggregations, and joins across multiple sources
• Optimize Spark jobs for performance and resource utilization
• Work with distributed storage systems like S3 / HDFS
• Debug and troubleshoot production data issues
• Collaborate with data engineers, analysts, and stakeholders
• Ensure data quality, consistency, and reliability
• Collaborating with cross-functional teams to analyze, design, and implement new applications.
• Ensuring optimal performance, quality, and responsiveness of the application/services.

We are known for our extraordinary people who make the impossible possible every day. Questians are driven by hunger, humility, and aspiration. We believe that our company culture is the key to our ability to make a true difference in every industry we reach. Our teams regularly invest time and dedicated effort into internal culture work, ensuring that all voices are heard.

We wholeheartedly believe in the diversity of thought that comes with fostering a culture rooted in respect, where everyone belongs, is valued, and feels inspired to share their ideas. We know embracing our unique differences makes us better, and that solving the worlds hardest engineering problems requires diverse ideas, perspectives, and backgrounds. We shine the brightest when we tap into the many dimensions that thrive across over 21,000 difference-makers in our workplace.

Work Experience

Required Skills

• Strong hands-on experience with Apache Spark
• Proficiency in Scala and/or Python (PySpark)
• Good understanding of Spark internals (RDD, DataFrame, Dataset APIs)
• Experience with data formats like Parquet, Avro, JSON
• Familiarity with distributed systems and big data concepts
• Strong SQL skills
• Experience with cloud platforms (AWS preferred – S3, EMR, Glue, Kinesis, Firehose, Hive)
• Knowledge of performance tuning and optimization techniques
• Experience with CI/CD pipelines
• Exposure to streaming frameworks (Spark Streaming)
• Familiarity with workflow orchestration tools like Apache Airflow

Preferred Skills

• Familiarity with Agile development practices.
• Strong analytical and problem-solving skills.
• Good communication and stakeholder collaboration abilities.
• Ability to work independently in a fast-paced environment.
• Ownership mindset with attention to data quality and reliability.

Additional Information

• Team player.
• Effective communication skills.
• Readiness to broaden skills (or enhance existing skills) to GoLang or any other technology related to Java/J2EE.
• Mandatory requirement to work from the Customer Office in Pune location.