ETL Tester with Python and Databricks Experience

Infobeans; Indore District, Pune District, Bengaluru; Hybrid; INR 600,000 - 1,000,000; Full time

Job summary

Infobeans is seeking an ETL QA Engineer to ensure data integrity on the Databricks platform, validating pipelines using SQL and Python while collaborating with global teams on data solutions.

The role requires flexibility to overlap US EST hours for real-time coordination and team participation, with responsibilities across data validation and DBT/Databricks testing.

Qualifications

• Advanced SQL with windows functions, complex joins and CTEs for data validation.
• ETL and Data Warehouse concepts understanding.

Responsibilities

• Design, develop, and execute ETL test cases to validate ETL/ELT processes.
• Write and execute complex SQL queries for data validation across multiple hops.
• Validate Bronze, Silver, and Gold layers in Databricks architecture.

Skills

SQL

Python

Databricks

ETL testing

Data validation

Job description

Job Title

ETL QA Tester with Python and Databricks experience

Summary

We are seeking a meticulous ETL QA Engineer to ensure data integrity across our Databricks platform.

You will validate complex data pipelines using SQL and Python, working closely with global teams to deliver high-quality data solutions.

This role requires flexibility to ensure overlap with US EST working hours for real-time coordination and team collaboration.

Key Responsibilities

• Design, develop, and execute ETL test cases to validate data extraction, transformation, and loading processes.
• Write and execute complex SQL queries to perform data validation between multiple hops.
• Medallion Architecture Testing: Validate data integrity as it moves through the Bronze, Silver, and Gold layers within Databricks.
• Write and execute advanced SQL and Python scripts to validate large datasets and identify defects.
• Design, develop, and execute UI Front end manual test cases to validate the functionalities of the software.
• Identify, document, and track defects, collaborate with developers to ensure timely resolution.
• Participate in project meetings and be available for discussions in US EST working hours as needed

Required Skills & Qualifications

Must-have

• Advanced SQL: Proficiency in windows functions, complex joins and CTEs for deep data validation.
• ETL & DWH: Deep understanding of ETL and Data Warehouse concepts.

Good to have

• Databricks Experience: Hands-on experience with Databricks Notebooks and cluster management
• Programming: Strong skills in Python, specifically PySpark and Pandas for large-scale data manipulation.
• Cloud Platform Familiarity: Experience working within Azure or AWS environments.
• Agile Methodologies and Defect Tracking Tools: Experience with Jira for defect tracking and working with Agile/Scrum framework.
• CI/CD (Additional Plus): Familiarity with automated deployment pipelines and version control (Git).