Senior Data Tester - Databricks & Big Data QA

Compunnel, Inc.; Louisville (KY); On-site; USD 90,000 - 110,000; Full time

Job summary

A leading data solutions provider in Louisville, KY is seeking an experienced Data Tester. In this role, you will validate data pipelines and ensure data integrity while utilizing your expertise in Databricks, PySpark, and SQL. Candidates should have extensive experience in cloud environments and a strong background in data lake testing. Join us to contribute to high-quality data processes in an Agile environment.

Qualifications

• 8+ years of experience in data testing or QA in enterprise data environments.
• 5+ years of experience testing ETL/Big Data pipelines and validating data transformations.
• 4+ years of hands-on experience with Databricks, including notebook execution and job scheduling.
• 4+ years of experience in PySpark (DataFrame APIs, UDFs, joins, transformations).
• 5+ years of strong proficiency in SQL for complex data validation.
• 3+ years of experience with Delta Lake or data lake testing.

Responsibilities

• Validate end-to-end data pipelines developed in Databricks and PySpark.
• Develop and execute test plans, test cases, and automated scripts for ETL and data quality validation.
• Perform data validation, reconciliation, and regression testing using SQL, Python, and PySpark DataFrame APIs.
• Verify data transformations, aggregations, and schema consistency across various layers.
• Test Delta Lake tables for schema evolution and performance.

Skills

Data testing

ETL validation

Databricks

PySpark

SQL

Delta Lake testing

Cloud platforms

Python scripting

Automation frameworks

Agile methodologies

Tools

Azure

AWS

GCP

pytest

Job description

A leading data solutions provider in Louisville, KY is seeking an experienced Data Tester. In this role, you will validate data pipelines and ensure data integrity while utilizing your expertise in Databricks, PySpark, and SQL. Candidates should have extensive experience in cloud environments and a strong background in data lake testing. Join us to contribute to high-quality data processes in an Agile environment.