Senior Data Engineer

Quarks Technosoft Pvt. Ltd.; Dadri; On-site; INR 2,500,000 - 4,500,000; Full time

Job summary

Quarks Technosoft Pvt. Ltd. invites a hands-on Senior Data Engineer to lead scalable data pipelines, focusing on migration, validation, and modernization of enterprise workloads on GCP.

You will design, implement, and operate end-to-end ETL/ELT processes, collaborate with architects and client teams, and ensure production-grade reliability across data platforms. The role emphasizes data modeling, quality checks, and efficient data processing, with strong Python/PySpark/SQL skills and experience

Qualifications

• 6–8 years of hands-on data engineering experience.
• Strong hands-on expertise in Python, PySpark, and SQL.
• Strong experience with cloud-based data engineering preferably GCP.
• Experience with services such as GCP services BigQuery, Cloud Storage, Dataflow, Dataproc, or Cloud Composer.
• Strong understanding of ETL/ELT, data migration, data modeling, and data warehousing.
• Experience building and supporting production-grade data pipelines.
• Experience with data validation, reconciliation, matching, and data quality.
• Good understanding of Git, CI/CD, monitoring, and production support.

Responsibilities

• Design,develop, and maintain data pipelines using Python, PySpark, and SQL.
• Workon data migration and modernization initiatives on GCP.
• DevelopETL/ELT pipelines for ingesting, transforming, and processing largedatasets.
• Implementdata validation, reconciliation, matching, and data quality frameworks.
• Applyrule-based and ML-assisted approaches for data validation and matchingwhere applicable.
• Optimizedata processing workloads and troubleshoot pipeline and data qualityissues.
• Workwith GCP services and modern data platforms to build reliableproduction-grade solutions.
• Collaboratewith architects, analysts, and client teams on technical design anddelivery.

Skills

Python

PySpark

SQL

GCP

BigQuery

Data Migration

Data Modeling

Data Warehousing

Git

CI/CD

Data Validation

Airflow/Composer

Tools

Airflow

Dataproc

Cloud Composer

Job description

We are looking for a hands-on Senior Data Engineerwith strong experience in data engineering and migration projects. Therole will focus on building scalable data pipelines preferably GCP, datamigration, reconciliation, validation, and modernization of enterprise dataworkloads.

Key Responsibilities

• Design,develop, and maintain data pipelines using Python, PySpark, and SQL.
• Workon data migration and modernization initiatives on GCP.
• DevelopETL/ELT pipelines for ingesting, transforming, and processing largedatasets.
• Implementdata validation, reconciliation, matching, and data quality frameworks.
• Applyrule-based and ML-assisted approaches for data validation and matchingwhere applicable.
• Optimizedata processing workloads and troubleshoot pipeline and data qualityissues.
• Workwith GCP services and modern data platforms to build reliableproduction-grade solutions.
• Collaboratewith architects, analysts, and client teams on technical design anddelivery.

Mandatory Skills

• 6–8years of hands-on data engineering experience.
• Stronghands-on expertise in Python, PySpark, and SQL.
• Strongexperience with cloud-based data engineering preferably GCP
• Experiencewith services such as GCP services BigQuery, Cloud Storage, Dataflow,Dataproc, or Cloud Composer.
• Strongunderstanding of ETL/ELT, data migration, data modeling, and datawarehousing.
• Experiencebuilding and supporting production-grade data pipelines.
• Experiencewith data validation, reconciliation, matching, and data quality.
• Goodunderstanding of Git, CI/CD, monitoring, and production support.

Preferred Skills

• Experiencewith Snowflake or Databricks in migration or integration scenarios.
• Exposureto ML-based data matching, anomaly detection, or automated data validation.
• Experiencewith large-scale cloud migration or data platform modernization programs.
• Exposureto Airflow/Cloud Composer and orchestration frameworks.