Data Tester

Compunnel, Inc.; Louisville (KY); On-site; USD 90,000 - 110,000; Full time

Job summary

A leading data solutions provider in Louisville, KY is seeking an experienced Data Tester. In this role, you will validate data pipelines and ensure data integrity while utilizing your expertise in Databricks, PySpark, and SQL. Candidates should have extensive experience in cloud environments and a strong background in data lake testing. Join us to contribute to high-quality data processes in an Agile environment.

Qualifications

• 8+ years of experience in data testing or QA in enterprise data environments.
• 5+ years of experience testing ETL/Big Data pipelines and validating data transformations.
• 4+ years of hands-on experience with Databricks, including notebook execution and job scheduling.
• 4+ years of experience in PySpark (DataFrame APIs, UDFs, joins, transformations).
• 5+ years of strong proficiency in SQL for complex data validation.
• 3+ years of experience with Delta Lake or data lake testing.

Responsibilities

• Validate end-to-end data pipelines developed in Databricks and PySpark.
• Develop and execute test plans, test cases, and automated scripts for ETL and data quality validation.
• Perform data validation, reconciliation, and regression testing using SQL, Python, and PySpark DataFrame APIs.
• Verify data transformations, aggregations, and schema consistency across various layers.
• Test Delta Lake tables for schema evolution and performance.

Skills

Data testing

ETL validation

Databricks

PySpark

SQL

Delta Lake testing

Cloud platforms

Python scripting

Automation frameworks

Agile methodologies

Tools

Azure

AWS

GCP

pytest

Job description

We are seeking an experienced Data Tester with strong expertise in Databricks, PySpark, and Big Data ecosystems.

This role focuses on validating data pipelines, ETL workflows, and analytical models to ensure data integrity, accuracy, and performance across distributed systems.

The ideal candidate will have hands-on experience in cloud environments and automation frameworks, with a deep understanding of data lake testing and SQL-based validation.

Key Responsibilities

• Validate end-to-end data pipelines developed in Databricks and PySpark.
• Develop and execute test plans, test cases, and automated scripts for ETL and data quality validation.
• Perform data validation, reconciliation, and regression testing using SQL, Python, and PySpark DataFrame APIs.
• Verify data transformations, aggregations, and schema consistency across raw, curated, and presentation layers.
• Test Delta Lake tables for schema evolution, partitioning, versioning, and performance.
• Collaborate with data engineers, analysts, and DevOps teams to ensure high-quality data delivery.
• Analyze Databricks job logs, Spark execution plans, and cluster metrics to troubleshoot issues.
• Participate in Agile/Scrum ceremonies and contribute to sprint planning and defect triage.
• Maintain documentation for test scenarios, execution reports, and data lineage verification.

Required Qualifications

• 8+ years of experience in data testing or QA in enterprise data environments.
• 5+ years of experience testing ETL/Big Data pipelines and validating data transformations.
• 4+ years of hands-on experience with Databricks, including notebook execution and job scheduling.
• 4+ years of experience in PySpark (DataFrame APIs, UDFs, joins, transformations).
• 5+ years of strong proficiency in SQL for complex data validation.
• 3+ years of experience with Delta Lake or data lake testing.
• 3+ years of experience in Python scripting for automation.
• 3+ years of experience with cloud platforms (Azure, AWS, or GCP).
• 2+ years of experience in test automation using tools like pytest or custom Python utilities.
• 4+ years of experience in data warehousing, data modeling, and data quality frameworks.
• 4+ years of experience with Agile/SAFe methodologies.
• 6+ years of analytical and debugging experience for data pipeline issues.

Preferred Qualifications

• Experience with CI/CD tools for Databricks or data testing (e.g., GitHub Actions, Jenkins, Azure DevOps).
• Exposure to BI validation tools (e.g., Power BI, Tableau, Looker).
• Knowledge of REST APIs for metadata or integration testing.
• Familiarity with big data tools such as Hive, Spark SQL, Snowflake, and Airflow.

Certifications

• Microsoft Azure Data Engineer Associate or AWS Big Data Specialty (preferred).