Senior Data Engineer

Questhiring; Gurugram District; Hybrid; INR 1,200,000 - 1,800,000; Full time

Job summary

Questhiring seeks an experienced Data Engineer II in Gurgaon to own end-to-end data pipelines on GCP, building scalable batch and streaming solutions. You will craft transformations with BigQuery, SQL, and dbt/Dataform while orchestrating workflows via Airflow.

Collaboration with cross-functional teams to ensure data quality and cost efficiency is essential. The role requires 6+ years of hands-on data engineering, strong GCP expertise, and a solid background in data modelling and testing.

Qualifications

• 6+ years of hands-on Data Engineering experience.
• Strong hands-on experience with GCP and BigQuery.
• Advanced SQL – complex joins, CTEs, window functions, and query optimization.
• Experience with production ETL/ELT pipelines.
• Hands-on experience with Cloud Composer / Apache Airflow.
• Experience with Dataform and/or dbt.
• Experience with Dataflow / Apache Beam or Dataproc / Spark / PySpark.
• Strong understanding of data modelling and large-scale datasets.
• Experience with incremental and idempotent processing.
• Understanding of data contracts, schema evolution, data quality, and reconciliation.
• Experience with Git, automated testing, and CI/CD.
• Strong production troubleshooting and problem-solving skills.

Responsibilities

• Design, develop, and maintain production‑grade data pipelines on GCP.
• Build scalable batch and incremental processing solutions.
• Develop complex transformations using BigQuery, SQL, and Dataform/dbt.
• Build and maintain workflows using Cloud Composer / Apache Airflow.
• Develop reusable transformation components, frameworks, and orchestration patterns.
• Build distributed data‑processing solutions using Dataflow / Apache Beam and/or Dataproc / Spark.
• Design data models for analytical and downstream data products.
• Implement data‑quality checks, validation, reconciliation, and data lineage.
• Optimize pipelines and BigQuery workloads for performance and cloud cost.
• Implement automated testing and integrate data workloads with CI/CD.
• Monitor production pipelines and troubleshoot issues through root‑cause analysis.
• Collaborate with Data Engineering, Platform Engineering, DevOps, Architecture, and business teams.

Skills

6+ years data engineering

GCP

BigQuery

SQL

ETL/ELT

Cloud Composer / Airflow

Dataform / dbt

Dataflow / Apache Beam / Dataproc / Py

Data modelling

Data quality checks

Git / CI/CD

Python

Tools

BigQuery

Dataproc

Spark / PySpark

Dataflow

Apache Beam

Cloud Composer

Airflow

Dataform

dbt

Git

CI/CD tooling

Job description

Location: Gurgaon

Work Mode: Hybrid

Experience: 6+ Years

About the Role

We are looking for a hands‑on Data Engineer II with strong GCP experience to build scalable data pipelines, transformations, and reusable data‑platform capabilities.

The role involves taking solutions end‑to‑end—from design and development to testing, deployment, and production support—while building reliable and reusable data engineering solutions.

Key Responsibilities

• Design, develop, and maintain production‑grade data pipelines on GCP.
• Build scalable batch and incremental processing solutions.
• Develop complex transformations using BigQuery, SQL, and Dataform/dbt.
• Build and maintain workflows using Cloud Composer / Apache Airflow.
• Develop reusable transformation components, frameworks, and orchestration patterns.
• Build distributed data‑processing solutions using Dataflow / Apache Beam and/or Dataproc / Spark.
• Design data models for analytical and downstream data products.
• Implement data‑quality checks, validation, reconciliation, and data lineage.
• Optimize pipelines and BigQuery workloads for performance and cloud cost.
• Implement automated testing and integrate data workloads with CI/CD.
• Monitor production pipelines and troubleshoot issues through root‑cause analysis.
• Collaborate with Data Engineering, Platform Engineering, DevOps, Architecture, and business teams.

Must‑Have Skills

• 6+ years of hands‑on Data Engineering experience.
• Strong hands‑on experience with GCP and BigQuery.
• Advanced SQL – complex joins, CTEs, window functions, and query optimization.
• Experience with production ETL/ELT pipelines.
• Hands‑on experience with Cloud Composer / Apache Airflow.
• Experience with Dataform and/or dbt.
• Experience with Dataflow / Apache Beam or Dataproc / Spark / PySpark.
• Strong understanding of data modelling and large‑scale datasets.
• Experience with incremental and idempotent processing.
• Understanding of data contracts, schema evolution, data quality, and reconciliation.
• Experience with Git, automated testing, and CI/CD.
• Strong production troubleshooting and problem‑solving skills.

Technical Skills

Programming: Python, SQL

Data Processing: Dataflow, Apache Beam, Dataproc, Spark/PySpark

Orchestration: Cloud Composer, Apache Airflow

Transformation: Dataform, dbt

Engineering: Git, CI/CD, Automated Testing, Monitoring

Data Engineering: ETL/ELT, Batch Processing, Data Modelling, Data Quality, Data Contracts, Metadata & Lineage

Good to Have

• Apache Iceberg or modern lakehouse technologies.
• Streaming or event‑driven processing.
• Data Product / Data Mesh concepts.
• Metadata‑driven or configuration‑driven processing.
• Change Data Capture (CDC).
• Terraform / Infrastructure as Code.
• Experience building reusable components for multiple engineering teams.

What We’re Looking For

A strong ownership mindset, good analytical and debugging skills, ability to independently design solutions, and the confidence to take data engineering solutions through production.