Data Engineer-Quest Global-Pune

Quest Global; Pune District; On-site; INR 900,000 - 1,400,000; Full time

Job summary

Quest Global in Pune is seeking a data engineering specialist with strong hands-on Apache Spark experience to design and optimize large-scale data pipelines. You will work with Scala and/or Python (PySpark), SQL, and data formats like Parquet/Avro/JSON to deliver reliable analytics platforms.

The role requires familiarity with cloud platforms (AWS: S3/EMR/Glue), Spark Streaming, Airflow, and CI/CD pipelines, with an emphasis on data quality and performance tuning in a fast-paced environment.

Qualifications

• Strong hands-on experience with Apache Spark.
• Proficiency in Scala and/or Python (PySpark).
• Good understanding of Spark internals (RDD, DataFrame, Dataset APIs).
• Experience with data formats like Parquet, Avro, JSON.
• Familiarity with distributed systems and big data concepts.
• Strong SQL skills.
• Experience with cloud platforms (AWS preferred S3, EMR, Glue, Kinesis, Firehose, Hive).
• Knowledge of performance tuning and optimization techniques.
• Experience with CI/CD pipelines.
• Exposure to streaming frameworks (Spark Streaming).
• Familiarity with workflow orchestration tools like Apache Airflow.

Responsibilities

• Team player with effective communication skills.
• Readiness to broaden skills to GoLang or related Java/J2EE technologies.
• Mandatory requirement to work from the Customer Office in Pune location.

Skills

Strong SQL skills

Apache Spark

Scala and/or Python (PySpark)

Spark internals (RDD, DataFrame, DSE)

Distributed systems & big data

Data formats (Parquet, Avro, JSON)

CI/CD pipelines

Spark Streaming

Airflow

Performance tuning & optimization

Tools

Apache Spark

PySpark

Scala

Python

AWS

S3

EMR

Glue

Kinesis

Firehose

Hive

Airflow

Spark Streaming

Job description

Required Skills

• Strong hands-on experience with Apache Spark
• Proficiency in Scala and/or Python (PySpark)
• Good understanding of Spark internals (RDD, DataFrame, Dataset APIs)
• Experience with data formats like Parquet, Avro, JSON
• Familiarity with distributed systems and big data concepts
• Strong SQL skills
• Experience with cloud platforms (AWS preferred S3, EMR, Glue, Kinesis, Firehose, Hive)
• Knowledge of performance tuning and optimization techniques
• Experience with CI/CD pipelines
• Exposure to streaming frameworks (Spark Streaming)
• Familiarity with workflow orchestration tools like Apache Airflow

Preferred Skills

• Familiarity with Agile development practices.
• Strong analytical and problem-solving skills.
• Good communication and stakeholder collaboration abilities.
• Ability to work independently in a fast-paced environment.
• Ownership mindset with attention to data quality and reliability.

Additional Information

• Team player.
• Effective communication skills.
• Readiness to broaden skills (or enhance existing skills) to GoLang or any other technology related to Java/J2EE.
• Mandatory requirement to work from the Customer Office in Pune location.