Data Quality Engineer

Qentelli; Hyderabad; On-site; INR 1,800,000 - 4,000,000; Full time

Job summary

Qentelli is seeking a Data Engineer Quality Engineer with strong expertise in Data Quality Testing, ETL/ELT validation, AWS, Kafka, Databricks, SQL, and Python. The role ensures reliability, accuracy, scalability, and performance of enterprise data platforms, pipelines, and analytics workflows.

You will collaborate with Data Engineering, Platform Engineering, Analytics, and DevOps teams to build trusted data solutions and maintain high-quality data ecosystems.

Qualifications

• 6+ years of experience in QA, SDET, Data Quality Engineering, or Data Testing.
• Strong hands-on experience with SQL for complex data validation and data analysis.
• Proficiency in Python for test automation, validation frameworks, and data quality checks.
• Experience testing ETL/ELT pipelines and enterprise data platforms.
• Hands-on experience with Databricks and Apache Spark (PySpark preferred).
• Strong knowledge of AWS Data Services including S3, Glue, Lambda, Redshift, Athena, and EMR.
• Experience with Kafka or other streaming platforms.
• Expertise in validating batch and real-time streaming data pipelines.
• Experience in data reconciliation, schema validation, and data transformation testing.
• Knowledge of CI/CD integration for automated testing.
• Experience working with large datasets and distributed data systems.
• Strong analytical, debugging, and problem-solving skills.

Responsibilities

• Validate batch and streaming data pipelines for correctness, completeness, consistency, and timeliness.
• Develop and maintain data quality checks covering null values, duplicates, schema drift, and referential integrity.
• Perform SQL-based validation of business rules, data transformations, and analytics workflows.
• Design and execute test strategies for Databricks-based ETL/ELT pipelines and reporting solutions.
• Establish end-to-end data reconciliation and traceability across source and target systems.
• Test ETL/ELT workflows built using AWS services such as Glue, Lambda, EMR, and Step Functions.
• Validate data ingestion, enrichment, aggregation, and publishing processes.
• Test historical data loads, reprocessing, and backfill scenarios.
• Validate Kafka-based streaming applications, including data integrity, ordering, and event processing.
• Test producer/consumer workflows, schema evolution, topic configurations, offsets, and retention policies.
• Validate AWS data services including S3, Redshift, Athena, Kinesis, DynamoDB, and related components.
• Verify IAM permissions, secure data access, encryption, and storage lifecycle policies.
• Develop and maintain automated data testing frameworks using Python.
• Integrate automated validation suites into CI/CD pipelines and implement data quality alerting mechanisms.
• Execute performance, scalability, regression, failover, and recovery testing for large-scale data platforms.
• Monitor pipeline health using CloudWatch, Grafana, Prometheus, or similar monitoring tools.
• Collaborate with stakeholders to define and validate Data SLAs and SLOs.
• Participate in incident management, root cause analysis, and resolution of data quality issues.

Skills

SQL validation

Python

Data Quality

ETL/ELT testing

Databricks

Apache Spark

AWS Data Services

Kafka

Data reconciliation

CI/CD

Large datasets

Analytics debugging

Tools

Databricks

Spark PySpark

Kafka

Job description

Job Summary

We are looking for a Data Engineer Quality Engineer with strong expertise in Data Quality Testing, ETL/ELT Validation, AWS, Kafka, Databricks, SQL, and Python. The role is responsible for ensuring the reliability, accuracy, scalability, and performance of enterprise data platforms, data pipelines, and analytics workflows. The candidate will work closely with Data Engineering, Platform Engineering, Analytics, and DevOps teams to build trusted data solutions and maintain high-quality data ecosystems.

Roles and Responsibilities

• Validate batch and streaming data pipelines for correctness, completeness, consistency, and timeliness.


• Develop and maintain data quality checks covering null values, duplicates, schema drift, and referential integrity.


• Perform SQL-based validation of business rules, data transformations, and analytics workflows.


• Design and execute test strategies for Databricks-based ETL/ELT pipelines and reporting solutions.


• Establish end-to-end data reconciliation and traceability across source and target systems.


• Test ETL/ELT workflows built using AWS services such as Glue, Lambda, EMR, and Step Functions.


• Validate data ingestion, enrichment, aggregation, and publishing processes.


• Test historical data loads, reprocessing, and backfill scenarios.


• Validate Kafka-based streaming applications, including data integrity, ordering, and event processing.


• Test producer/consumer workflows, schema evolution, topic configurations, offsets, and retention policies.


• Validate AWS data services including S3, Redshift, Athena, Kinesis, DynamoDB, and related components.


• Verify IAM permissions, secure data access, encryption, and storage lifecycle policies.


• Develop and maintain automated data testing frameworks using Python.


• Integrate automated validation suites into CI/CD pipelines and implement data quality alerting mechanisms.


• Execute performance, scalability, regression, failover, and recovery testing for large-scale data platforms.


• Monitor pipeline health using CloudWatch, Grafana, Prometheus, or similar monitoring tools.


• Collaborate with stakeholders to define and validate Data SLAs and SLOs.


• Participate in incident management, root cause analysis, and resolution of data quality issues.

Mandatory Skills

• 6+ years of experience in QA, SDET, Data Quality Engineering, or Data Testing.


• Strong hands‑on experience with SQL for complex data validation and data analysis.


• Proficiency in Python for test automation, validation frameworks, and data quality checks.


• Experience testing ETL/ELT pipelines and enterprise data platforms.


• Hands‑on experience with Databricks and Apache Spark (PySpark preferred).


• Strong knowledge of AWS Data Services including S3, Glue, Lambda, Redshift, Athena, and EMR.


• Experience with Kafka or other streaming platforms.


• Expertise in validating batch and real‑time streaming data pipelines.


• Experience in data reconciliation, schema validation, and data transformation testing.


• Knowledge of CI/CD integration for automated testing.


• Experience working with large datasets and distributed data systems.


• Strong analytical, debugging, and problem‑solving skills.

Preferred Skills

• Experience with Apache Spark (PySpark/Scala).


• Knowledge of Kafka serialization formats such as Avro, JSON, and Protobuf.


• Familiarity with monitoring tools such as CloudWatch, Grafana, and Prometheus.


• Experience in performance, load, soak, and failover testing of data platforms.


• Exposure to DataOps and DevOps practices.


• Knowledge of data governance, data security, and compliance best practices.


• Experience with synthetic test data generation and reusable test utilities.


• Understanding of data SLAs, SLOs, and observability frameworks.


• Exposure to Agile/Scrum development methodologies.


• AWS certifications or Databricks certifications are a plus.