Python QA - (Pyspark with Databricks)

Test Yantra; Bengaluru; On-site; INR 1,800,000 - 2,400,000; Full time

Job summary

Test Yantra is seeking an experienced QA Automation Engineer to bolster our data engineering and analytics capabilities. You will design and maintain automated test frameworks in Python, validate large-scale ETL pipelines with PySpark and Databricks, and ensure data quality across diverse datasets.

You will work with Data Engineers, Developers, and Analysts in an Agile setting, implementing automated checks, contributing to CI/CD pipelines, and advancing QA best practices to deliver reliable

Qualifications

• 8+ years in software testing or test automation.
• Hands-on Python experience.
• Experience with PySpark and Databricks.
• SQL and ETL/data validation experience.
• Agile/Scrum experience.

Responsibilities

• Design, develop, and maintain automated test frameworks using Python.
• Validate ETL pipelines, data transformations, and data ingestion processes built using PySpark and Databricks.
• Perform data validation, reconciliation, and quality checks across structured and semi-structured datasets.
• Develop automated test scripts for functional, integration, regression, and end-to-end testing of data platforms.
• Validate large datasets using SQL and PySpark.
• Build and maintain automated data quality checks and monitoring processes.
• Collaborate with Data Engineers, Developers, and Business Analysts to understand data requirements and ensure high-quality deliveries.
• Analyse defects, perform root cause analysis, and support issue resolution.
• Integrate automated tests into CI/CD pipelines and support Agile delivery.
• Contribute to QA best practices, automation strategy, and continuous improvement initiatives.

Skills

Python

PySpark

Databricks

SQL

ETL

Data validation

Agile/Scrum

CI/CD

Tools

Great Expectations

Soda

Deequ

dbt Tests

Job description

About The Role

We are looking for an experienced QA Automation Engineer with strong expertise in Python, PySpark, and Databricks to support enterprise data engineering and analytics initiatives. The ideal candidate should have hands‑on experience in validating large-scale data pipelines, ETL processes, and data platforms while building robust automation frameworks to ensure data quality and reliability.

Key Responsibilities

• Design, develop, and maintain automated test frameworks using Python.
• Validate ETL pipelines, data transformations, and data ingestion processes built using PySpark and Databricks.
• Perform data validation, reconciliation, and quality checks across structured and semi‑structured datasets.
• Develop automated test scripts for functional, integration, regression, and end‑to‑end testing of data platforms.
• Validate large datasets using SQL and PySpark.
• Build and maintain automated data quality checks and monitoring processes.
• Collaborate with Data Engineers, Developers, and Business Analysts to understand data requirements and ensure high‑quality deliveries.
• Analyse defects, perform root cause analysis, and support issue resolution.
• Integrate automated tests into CI/CD pipelines and support Agile delivery.
• Contribute to QA best practices, automation strategy, and continuous improvement initiatives.

Skills & Qualifications

• 8+ years of experience in Software Testing or Test Automation.
• Strong hands‑on experience with Python.
• Solid expertise in PySpark and Databricks.
• Strong experience in SQL and ETL/data validation.
• Experience testing large‑scale data pipelines and data engineering workflows.
• Hands‑on experience with data quality frameworks or libraries such as Great Expectations, Soda, Deequ, dbt Tests, or similar.
• Strong understanding of data validation, reconciliation, and quality assurance methodologies.
• Experience working in Agile/Scrum environments.
• Excellent analytical, troubleshooting, and problem‑solving skills.