Senior Data Engineer

Quest Global; Pune District; On-site; INR 1,500,000 - 3,000,000; Full time

Job summary

Quest Global in Pune, India, seeks a data engineer to design, implement, and maintain enterprise ETL data pipelines. You will leverage Spark, Airflow, and SQL to deliver scalable solutions across global clients, collaborating with senior engineers to ensure data quality, governance, and efficient processing.

The role emphasizes hands-on PySpark work, Cloudera CDP expertise, and CI/CD practices to drive reliable data platforms. On-site collaboration is required with cross-functional teams.

Qualifications

• Strong hands-on expertise in Spark, Airflow, and SQL.
• Deep understanding and experience with modern Data Platforms – Cloudera Data Platform (CDP).
• Strong hands-on experience with PySpark.
• Experience with Cloudera Data Platform (CDP components) and Apache Ranger.
• Deep understanding of distributed data systems and Hive Metastore.
• Experience with cataloging, lineage, and governance.
• Experience / understanding Open Data Contract Standard (ODCS) and its implementation.
• Experience working with SQL, file formats (Iceberg/Parquet), and partitioning/bucketing strategies.

Responsibilities

• Support the design, implementation, and maintenance of enterprise ETL processes for data platforms, for a global client base.
• Develop scalable and efficient code to process data, ensuring availability and accessibility in a timely manner.
• Leverage big data processing frameworks such as Apache Spark and Hadoop to build and optimize data pipelines.
• Collaborate with senior engineers to address data challenges, contributing to solutions that maintain high data quality.
• Assist in the data delivery process, working alongside Data Engineers and Analysts to support accurate, high-value data solutions across various clients and industries.
• Build strong working relationships with team members and clients, contributing to both local and global projects.
• Learn and apply industry best practices, including version control, code reviews, and data validation, to ensure quality in data processes.
• Use SQL and other database technologies to help optimize data processing and reduce the time required to handle large data sets.
• Design, implement, and maintain data pipelines using ETL frameworks, orchestration tools, and distributed data processing engines.
• Participate in efforts to automate routine data tasks and streamline processes.

Skills

Spark

Airflow

SQL

PySpark

Tools

Cloudera Data Platform

Apache Ranger

Iceberg

Parquet

Job description

• Support the design, implementation, and maintenance of enterprise ETL processes for data platforms, for a global client base.
• Develop scalable and efficient code to process data, ensuring availability and accessibility in a timely manner.
• Leverage big data processing frameworks such as Apache Spark and Hadoop to build and optimize data pipelines.
• Collaborate with senior engineers to address data challenges, contributing to solutions that maintain high data quality.
• Assist in the data delivery process, working alongside Data Engineers and Analysts to support accurate, high-value data solutions across various clients and industries.
• Build strong working relationships with team members and clients, contributing to both local and global projects.
• Learn and apply industry best practices, including version control, code reviews, and data validation, to ensure quality in data processes.
• Use SQL and other database technologies to help optimize data processing and reduce the time required to handle large data sets.
• Design, implement, and maintain data pipelines using ETL frameworks, orchestration tools, and distributed data processing engines.
• Participate in efforts to automate routine data tasks and streamline processes.

Job Requirements

• Support the design, implementation, and maintenance of enterprise ETL processes for data platforms, for a global client base.
• Develop scalable and efficient code to process data, ensuring availability and accessibility in a timely manner.
• Leverage big data processing frameworks such as Apache Spark and Hadoop to build and optimize data pipelines.
• Collaborate with senior engineers to address data challenges, contributing to solutions that maintain high data quality.
• Assist in the data delivery process, working alongside Data Engineers and Analysts to support accurate, high-value data solutions across various clients and industries.
• Build strong working relationships with team members and clients, contributing to both local and global projects.
• Learn and apply industry best practices, including version control, code reviews, and data validation, to ensure quality in data processes.
• Use SQL and other database technologies to help optimize data processing and reduce the time required to handle large data sets.
• Design, implement, and maintain data pipelines using ETL frameworks, orchestration tools, and distributed data processing engines.
• Participate in efforts to automate routine data tasks and streamline processes.

Work Experience

Years of Experience: 4-6 Years

Strong hands-on expertise in Spark, Airflow, and SQL.

Deep understanding and experience with modern Data Platforms – Cloudera Data Platform (CDP)

• Strong hands-on experience with PySpark.
• Experience with Cloudera Data Platform (CDE, CDW, Ozone, Airflow, SDX), Apache Ranger
• Deep understanding of distributed data systems and Hive Metastore
• Experience and understanding of cataloging, lineage, and governance
• Experience / understanding Open Data Contract Standard (ODCS) and its implementation
• Experience working with SQL, file formats (Iceberg/Parquet), and partitioning/bucketing strategies.

Nice to have

• Experience modernizing enterprise finance systems or regulated environments
• Knowledge of CI/CD, data engineering best practices
• Understanding of financial data structures, accounting processes, or reconciliation workflows.