Senior Data QE Engineer

Dataart Technologies; Hyderabad; On-site; INR 1,200,000 - 1,800,000; Full time

Job summary

Dataart Technologies is seeking a senior data quality engineer to own end-to-end validation of data moving from source systems into analytical datasets. You will design and execute ETL quality tests, write SQL checks for Snowflake models, and automate data reconciliation with Python.

Expect collaboration with analytics teams and a CI/CD setup using GitHub Actions. You will validate PostgreSQL sources, monitor pipelines with Grafana and CloudWatch, and contribute to dashboards and quality metrics

Qualifications

• Senior-level data quality engineering experience in ETL testing and data validation.
• Strong SQL and Snowflake data-modelling experience, including layered raw, history, silver, and gold models, reconciliation and drift queries, and slowly changing dimensions.
• Experience with logical and physical data models and assessing downstream impact of schema and data-contract changes.
• Experience validating PostgreSQL sources, including consistent snapshot counts and logical replication.
• Hands-on Python automation for data comparison and reconciliation, using pytest or an equivalent framework.
• Experience running and reading orchestrated pipelines in Airflow or an equivalent scheduler.
• Experience validating data movement, transformation, completeness, history, freshness, and compliance from source systems into analytical datasets.
• Experience diagnosing data quality issues and supporting remediation.
• Experience integrating data quality checks with CI/CD using GitHub Actions or equivalent tools.
• Experience with test management and defect tracking in TestRail and Jira.
• Working knowledge of AWS and Grafana for pipeline observation, including CloudWatch, secrets management, and container platforms.
• Experience designing resilience tests and judging evidence for failover, outage, volume, and recovery objectives.
• Working knowledge of data classification, masking, and retention checks.
• Experience producing data quality metrics, reports, and dashboards.
• Experience with risk-based testing in Agile or Kanban teams.
• Hands-on use of AI coding assistants to author SQL, Python, and quality checks, with human review of generated output.
• Ability to communicate clearly with engineering and business stakeholders.
• Experience providing quality evidence, risks, and readiness input for releases.

Responsibilities

• Own data quality for data moving from source systems into analytical datasets, from test design through execution, investigation, and reporting.
• Design, implement, and execute data quality tests, and publish quality metrics and dashboards that producing and consuming groups can use.
• Validate pipelines end to end, covering movement, transformation, completeness, history, freshness, and compliance.
• Write reconciliation, drift, and quality SQL on Snowflake layered models (raw, history, silver, and gold), including slowly changing dimensions and merge behavior.
• Validate PostgreSQL sources with consistent snapshot counts and checks that logical replication kept the expected data.
• Use logical and physical data models and data contracts to assess downstream impact and set quality coverage.
• Automate data comparison and reconciliation in Python with pytest, and keep the checks in GitHub.
• Trigger pipeline runs in Airflow and read task results into the quality verdict.
• Investigate quality issues and support remediation with the groups that produce and consume the data.
• Design failover, outage, and volume tests and judge the evidence, including recovery time and recovery point. The platform team executes destructive and chaos tests.
• Run read-only checks in pre-production and production.
• Integrate quality checks with CI/CD on GitHub Actions or an equivalent tool.
• Use AWS and Grafana to observe pipeline health, including CloudWatch metrics, secrets metadata, and container platform state.
• Check classification, masking, and retention against the stated data rules.
• Track defects to closure in Jira and keep test cases and results in TestRail.
• Provide quality evidence, risks, and readiness input for releases.
• Use an AI coding assistant to analyze data and author SQL, Python, and quality checks, and review generated output before it is used.

Skills

SQL

Python

Data quality testing

ETL testing

QA automation

Stakeholder communication

Tools

Snowflake

Airflow

GitHub Actions

TestRail

Jira

Grafana

CloudWatch

PostgreSQL

Job description

Responsibilities

• Own data quality for data moving from source systems into analytical datasets, from test design through execution, investigation, and reporting.
• Design, implement, and execute data quality tests, and publish quality metrics and dashboards that producing and consuming groups can use.
• Validate pipelines end to end, covering movement, transformation, completeness, history, freshness, and compliance.
• Write reconciliation, drift, and quality SQL on Snowflake layered models (raw, history, silver, and gold), including slowly changing dimensions and merge behavior.
• Validate PostgreSQL sources with consistent snapshot counts and checks that logical replication kept the expected data.
• Use logical and physical data models and data contracts to assess downstream impact and set quality coverage.
• Automate data comparison and reconciliation in Python with pytest, and keep the checks in GitHub.
• Trigger pipeline runs in Airflow and read task results into the quality verdict.
• Investigate quality issues and support remediation with the groups that produce and consume the data.
• Design failover, outage, and volume tests and judge the evidence, including recovery time and recovery point. The platform team executes destructive and chaos tests.
• Run read-only checks in pre-production and production.
• Integrate quality checks with CI/CD on GitHub Actions or an equivalent tool.
• Use AWS and Grafana to observe pipeline health, including CloudWatch metrics, secrets metadata, and container platform state.
• Check classification, masking, and retention against the stated data rules.
• Track defects to closure in Jira and keep test cases and results in TestRail.
• Provide quality evidence, risks, and readiness input for releases.
• Use an AI coding assistant to analyze data and author SQL, Python, and quality checks, and review generated output before it is used.

Requirements

• Senior-level data quality engineering experience in ETL testing and data validation.
• Strong SQL and Snowflake data-modelling experience, including layered raw, history, silver, and gold models, reconciliation and drift queries, and slowly changing dimensions and merge behavior.
• Experience with logical and physical data models and with assessing the downstream impact of schema and data-contract changes.
• Experience validating PostgreSQL sources, including consistent snapshot counts and logical replication.
• Hands-on Python automation for data comparison and reconciliation, using pytest or an equivalent framework.
• Experience running and reading orchestrated pipelines in Airflow or an equivalent scheduler.
• Experience validating data movement, transformation, completeness, history, freshness, and compliance from source systems into analytical datasets.
• Experience diagnosing data quality issues and supporting remediation.
• Experience integrating data quality checks with CI/CD using GitHub Actions or equivalent tools.
• Experience with test management and defect tracking in TestRail and Jira.
• Working knowledge of AWS and Grafana for pipeline observation, including CloudWatch, secrets management, and container platforms.
• Experience designing resilience tests and judging evidence for failover, outage, volume, and recovery objectives.
• Working knowledge of data classification, masking, and retention checks.
• Experience producing data quality metrics, reports, and dashboards.
• Experience with risk-based testing in Agile or Kanban teams.
• Hands-on use of AI coding assistants (Cursor, GitHub Copilot, Claude Code, Codex, or equivalent) to author SQL, Python, and quality checks, with human review of generated output.
• Ability to communicate clearly with engineering and business stakeholders.
• Experience providing quality evidence, risks, and readiness input for releases.

Nice to have

• Experience observing Snowflake Openflow or NiFi connector state well enough to recognize a stalled load.
• Experience with custom cross-table data quality checks that publish results for dashboarding.
• Familiarity with GenAI, LLMs, and agentic workflows for maintaining data quality checks.
• Familiarity with RAG and MCP concepts for QE tooling and for generating test data.
• Knowledge of AI-driven test optimization and coverage analytics (SeaLights or equivalent).
• Experience with service virtualization using WireMock or an equivalent tool, so checks can run when a dependency is unavailable.
• Experience specifying and reviewing dbt tests. Developers author those tests.