Sr. Data QA Engineer

R3 Consultant; United States; Remote; USD 16,000 - 31,000; Full time

Job summary

R3 Consultant is seeking an experienced Sr. QA Data Engineer to own the quality of data pipelines and data-driven systems, working with Data Engineers, Software Engineers, and Analysts to ensure data integrity across sources and stages.

The role emphasizes hands-on testing of ETL/ELT processes, SQL validation, and automation using Python, Spark, Databricks, and Delta Lake, in a remote India setting with IST shift timings.

Qualifications

• Bachelor's degree in Computer Science, Engineering, Information Technology, or a related field.
• 5+ years of QA/testing experience, preferably working with Data Engineering, Big Data, or data-intensive applications.
• Strong hands-on ETL/ELT testing and data validation.
• Experience with Apache Spark/PySpark and Databricks.
• Excellent English communication and problem-solving skills.

Responsibilities

• Design and execute functional, integration, regression, and end-to-end testing for data pipelines and platforms.
• Validate ETL/ELT workflows, data transformations, and source-to-target data accuracy.
• Test Spark/PySpark, Databricks, and Delta Lake workflows.
• Perform SQL-based data validation and data reconciliation.
• Develop Python-based test automation for data validation and regression.
• Collaborate with Data Engineers, Analysts, Data Scientists and Product teams.
• Maintain test cases, automation suites, and release validation.

Skills

Data Testing

ETL/ELT Testing

SQL

Python

Apache Spark

Databricks

Delta Lake

Data Validation

APIs Testing

Linux/Unix

Shell Scripting

Git

CI/CD QA

Agile/Scrum

Troubleshooting

Education

Bachelor's degree in Computer Science, Engineering, Information Technology, or a related field

Tools

Databricks

Delta Lake

Apache Spark

PySpark

Job description

Sr. QA Data Engineer

Employment Type: Full-Time, Permanent

Location: Remote, Pan-India

Shift Timings: 11:00 AM 08:00 PM IST

Reporting To: Sr. Technical Manager / Director of Engineering, as assigned by Management

Position Overview

We are looking for an experienced Sr. QA Data Engineer to join our Data Engineering team and take ownership of the quality, accuracy, consistency, and reliability of data pipelines and data-driven systems.

The ideal candidate will have strong hands‑on experience in Data Testing, ETL/ELT Testing,

Database Testing, SQL, Python, Apache Spark, and Databricks. The candidate should be comfortable validating complex data transformations, investigating data discrepancies, and testing end-to-end data workflows across multiple sources, processing layers, and downstream systems.

You will work closely with Data Engineers, Software Engineers, Database Architects, Data Analysts, Data Scientists, and Product teams to ensure data is processed correctly and delivered reliably to downstream consumers.

This role requires strong analytical and problem-solving skills, attention to detail, and the ability to independently understand complex data flows, develop comprehensive test strategies, automate data validation, investigate defects, and drive quality across the data engineering ecosystem.

Key Responsibilities

• Design and execute functional, integration, regression, and end-to-end testing for data pipelines and data platforms.
• Validate ETL/ELT workflows, data transformations, business rules, and source-to-target data accuracy.
• Test Apache Spark/PySpark, Databricks, and Delta Lake data processing workflows.
• Perform advanced SQL-based data validation, reconciliation, and data-quality testing.
• Validate data for accuracy, completeness, consistency, uniqueness, integrity, and schema compliance.
• Test APIs and integrations across data sources, databases, pipelines, and downstream systems.
• Develop and maintain Python/SQL-based test automation for data validation and regression testing.
• Design and execute positive, negative, edge-case, failure, retry, and recovery scenarios across data workflows.
• Investigate data discrepancies and work with Data Engineers, Developers, Analysts, and Data Scientists to identify root causes and resolve issues.
• Review requirements, technical designs, and data flows to identify QA risks, dependencies, and coverage gaps early in the development lifecycle.
• Maintain comprehensive test cases, automation suites, defect documentation, test evidence, and release validation for assigned data products.
• Drive continuous improvement of QA processes, data validation frameworks, automation coverage, and overall data quality.

Required Qualifications & Skills

• Bachelor's degree in Computer Science, Engineering, Information Technology, or a related field.
• 5+ years of relevant QA/testing experience, preferably working with Data Engineering, Big Data, or data-intensive applications.
• Strong hands‑on experience in ETL/ELT testing, database testing, and data validation.
• Advanced SQL skills, including:
• Complex joins
• Subqueries
• CTEs
• Window functions
• Aggregations
• Data reconciliation
• Practical experience testing Apache Spark/PySpark data processing workflows.
• Hands-on experience with Databricks and Delta Lake.
• Strong understanding of Big Data concepts, distributed data processing, data pipelines, and data architecture.
• Experience validating Parquet, JSON, CSV, and other structured and semi-structured data formats.
• Strong Python programming/scripting skills for test automation and data validation.
• Experience with API and integration testing.
• Working knowledge of Linux/Unix environments and shell scripting.
• Understanding of:
• Data Lakes
• Lakehouse Architecture
• Data Warehousing
• ETL/ELT Pipelines
• Experience with Git, defect management tools, and CI/CD-based testing workflows.
• Experience working in an Agile/Scrum environment.
• Strong analytical, troubleshooting, and root-cause analysis skills.
• Ability to work independently, take ownership, and collaborate effectively with cross-functional teams.
• Excellent written and verbal communication skills in English.

Core Technology Areas

Data Testing | ETL/ELT | SQL | Python | Apache Spark | PySpark | Databricks | Delta Lake | Database Testing | API Testing | Data Quality | Data Lakes | Lakehouse | Data Warehousing | Linux/Unix | Shell Scripting | Git | CI/CD