Senior Data Quality Engineer

Papigen; Chennai District; On-site; INR 2,500,000 - 4,500,000; Full time

Job summary

Papigen is seeking a Senior Data Quality Engineer to strengthen our data reliability across enterprise platforms. The role focuses on implementing data quality controls, monitoring pipelines with Monte Carlo Data Observability, and solving data accuracy challenges across SQL and NoSQL environments.

You will work with Azure Databricks for batch and near-real-time processing, validate ETL/ELT flows, profile data, and collaborate across data teams to improve governance, lineage, and stewardship.

Qualifications

• Experience building data quality controls across large datasets.
• Experience with data observability platforms (Monte Carlo).
• Strong SQL for validation, reconciliation, profiling.
• Experience with ETL/ELT pipelines.
• Experience with Azure cloud data platforms.
• Familiarity with data governance and metadata management.

Responsibilities

• Develop and execute data quality rules, validations, and controls across enterprise datasets.
• Monitor data pipelines with Monte Carlo Data Observability.
• Develop complex SQL queries for data validation, reconciliation, profiling, and troubleshooting.
• Work hands‑on with Azure Databricks for batch and near-real‑time data processing.
• Validate ETL/ELT pipelines and identify data quality issues across source-to-target flows.
• Collaborate with Data Engineers, Architects, Data Stewards, and business teams to resolve data quality issues.
• Support Collibra and data governance initiatives including metadata, lineage, and stewardship.
• Document data quality rules, issues, root‑cause analysis, and remediation processes.
• Support production issues and contribute to continuous improvement of data quality and observability processes.

Skills

SQL on large data

Databricks

Monte Carlo

Data quality tools

ETL/ELT

Relational databases

NoSQL databases

Azure data platform

Collibra

Data warehousing concepts

Tools

Databricks

Monte Carlo

Collibra

Job description

Role Overview

We are looking for an experienced Senior Data Quality Engineer with strong expertise in SQL, Databricks, Data Quality, and Data Observability. The candidate will be responsible for implementing data quality controls, monitoring data pipelines, identifying data issues, and ensuring the reliability and accuracy of enterprise data platforms.

Key Responsibilities

• Develop and execute data quality rules, validations, and controls across enterprise datasets.
• Work with Monte Carlo Data Observability to monitor data pipelines, detect anomalies, and identify data reliability issues.
• Develop complex SQL queries for data validation, reconciliation, profiling, and troubleshooting.
• Work hands‑on with Azure Databricks for batch and near‑real‑time data processing.
• Validate ETL/ELT pipelines and identify data quality issues across source-to-target flows.
• Work with relational and NoSQL databases and large‑volume datasets.
• Support Data Warehouse, Data Lake, and Lakehouse environments.
• Perform data profiling and investigate data anomalies, freshness, volume, schema, and lineage issues.
• Collaborate with Data Engineers, Architects, Data Stewards, and business teams to resolve data quality issues.
• Support Collibra and data governance initiatives including metadata, lineage, and stewardship.
• Document data quality rules, issues, root‑cause analysis, and remediation processes.
• Support production issues and contribute to continuous improvement of data quality and observability processes.

Required Skills

• Strong hands‑on SQL experience with large datasets.
• Hands‑on experience with Databricks.
• Mandatory: Practical experience with Monte Carlo Data Observability.
• Experience with Data Quality tools/frameworks.
• Hands‑on experience with ETL/ELT pipelines.
• Experience with relational and NoSQL databases.
• Familiarity with Azure cloud data platforms.
• Familiarity with Collibra.
• Strong understanding of Data Warehouse, Data Lake, and Lakehouse architectures.
• Experience with batch and near‑real‑time data processing.
• Knowledge of dimensional and relational data modeling.
• Strong analytical and problem‑solving skills.

Nice to Have

• Programming experience in Python, Scala, or Java.
• Experience with Airflow or Control-M.
• Knowledge of Data Governance, Metadata Management, and Data Lineage.
• Banking or other regulatory/compliance‑driven data environment experience.
• CDMP, Azure, or AWS certifications.

Skills: metadata management,data warehouse,monte carlo,data observability,scala,azure,data quality,aws,etl,python,databases,sql,data governance,nosql,java