Python QA - (Pyspark with Databricks)

TestYantra Software Solutions; Bengaluru; On-site; INR 1,500,000 - 3,200,000; Full time

Job summary

TestYantra Software Solutions in Bengaluru is seeking an experienced QA Automation Engineer to lead automation for enterprise data pipelines. You will design, implement, and maintain robust automation frameworks using Python, PySpark, and Databricks to validate large-scale ETL processes.

The role emphasizes data quality, CI/CD integration, and collaboration with data engineers and analysts to ensure reliable data platforms and timely deliveries.

Qualifications

• 8+ years of experience in Software Testing or Test Automation.
• Hands-on Python expertise driving automation frameworks.
• Deep knowledge of PySpark and Databricks for data pipelines.
• Proficient in SQL and ETL/data validation techniques.
• Experience validating large-scale data pipelines and data platforms.
• Familiarity with data quality frameworks such as Great Expectations, Soda, Deequ, or dbt Tests.
• Experience working in Agile/Scrum environments and CI/CD integration.

Responsibilities

• Design, develop, and maintain automated test frameworks using Python.
• Validate ETL pipelines, data transformations, and ingestion using PySpark and Databricks.
• Perform data validation, reconciliation, and quality checks across datasets.
• Develop automated test scripts for functional, integration, regression, and end-to-end testing of data platforms.
• Integrate automated tests into CI/CD pipelines and support Agile delivery.
• Collaborate with Data Engineers, Developers, and Analysts to ensure data quality and timely deliveries.
• Analyse defects, perform root cause analysis, and support issue resolution.

Skills

Python

PySpark

Databricks

SQL

Data validation

Great Expectations

dbt Tests

Agile/Scrum

Tools

Great Expectations

Soda

Deequ

dbt

Job description

About the Role

We are looking for an experienced QA Automation Engineer with strong expertise in Python, PySpark, and Databricks to support enterprise data engineering and analytics initiatives. The ideal candidate should have hands‑on experience in validating large‑scale data pipelines, ETL processes, and data platforms while building robust automation frameworks to ensure data quality and reliability.

Key Responsibilities

• Design, develop, and maintain automated test frameworks using Python.
• Validate ETL pipelines, data transformations, and data ingestion processes built using PySpark and Databricks.
• Perform data validation, reconciliation, and quality checks across structured and semi‑structured datasets.
• Develop automated test scripts for functional, integration, regression, and end‑to‑end testing of data platforms.
• Validate large datasets using SQL and PySpark.
• Build and maintain automated data quality checks and monitoring processes.
• Collaborate with Data Engineers, Developers, and Business Analysts to understand data requirements and ensure high‑quality deliveries.
• Analyse defects, perform root cause analysis, and support issue resolution.
• Integrate automated tests into CI/CD pipelines and support Agile delivery.
• Contribute to QA best practices, automation strategy, and continuous improvement initiatives.

Skills & Qualifications

• 8+ years of experience in Software Testing or Test Automation.
• Strong hands‑on experience with Python.
• Solid expertise in PySpark and Databricks.
• Strong experience in SQL and ETL/data validation.
• Experience testing large‑scale data pipelines and data engineering workflows.
• Hands‑on experience with data quality frameworks or libraries such as Great Expectations, Soda, Deequ, dbt Tests, or similar.
• Strong understanding of data validation, reconciliation, and quality assurance methodologies.
• Experience working in Agile/Scrum environments.
• Excellent analytical, troubleshooting, and problem‑solving skills.