Senior ETL Tester with AI Testing Experience

Confiar Services; Chicago (IL); On-site; USD 140,000 - 190,000; Full time

Job summary

Confiar Services seeks a Senior QA / ETL Test Lead to own ETL testing strategy and validate data integrity across legacy and modern platforms. You will lead testing for AI-generated queries, ensure privacy and multi-tenant isolation, and mentor QA engineers.

The role requires expertise in SQL, Azure Databricks, PySpark, and automation frameworks, with a focus on data quality and regulatory compliance in Healthcare data environments.

Qualifications

• 10+ years of experience in data/ETL testing and ownership of test strategy for a major migration or re-platforming project.
• Strong SQL skills with complex queries and data reconciliation.
• Hands-on experience with Azure Databricks, PySpark, and Python automation.
• Experience with data quality and automation frameworks such as Great Expectations, dbt tests, pytest, or equivalent.
• Expertise in source-to-target validation, data integrity, regression testing, and test strategy development.
• Experience testing healthcare data, PHI-scoped, de-identified, or synthetic datasets.
• Experience with at least two data platforms (Databricks, SQL Server, Vertica).
• Strong leadership, defect management, stakeholder communication, and reporting skills.
• Experience with web testing and test automation tools.

Responsibilities

• Own the overall ETL test strategy, test planning, reconciliation methodology, and risk-based test coverage.
• Design source-to-target validation across data ingestion, transformation, and refined data layers.
• Build and maintain golden reference datasets to evaluate AI-generated queries and transformation logic.
• Develop automated regression frameworks to compare new pipeline outputs with existing production data.
• Perform row-count, aggregate, field-level, data-type, null, duplicate, and referential integrity validations.
• Evaluate AI agents for accuracy, consistency, hallucinations, incorrect joins, missing fields, and confidence calibration.
• Use MLflow or similar tools to track evaluation results, compare model versions, and establish regression gates.
• Validate healthcare data privacy, PHI protection, multi-tenant isolation, and facility-level data access controls.
• Validate data-sharing contracts and ensure generated queries comply with permitted access restrictions.
• Perform query optimization and acquisition-cost validation against baseline queries.
• Lead and mentor QA engineers, review test coverage, manage defects, and report quality metrics.
• Ensure compatibility across Azure Databricks, SQL Server, Vertica, and other data platforms.
• Prepare test evidence and quality reports for client and executive reviews.

Skills

ETL testing

SQL

Azure Databricks

PySpark

Python automation

Great Expectations

dbt tests

pytest

data quality

leadership

stakeholder communication

healthcare data

two data platforms

test strategy

web testing

automation frameworks

Tools

MLflow

LangSmith

Weights & Balance

GenAI evaluation tools

Caboodle/Clarity (Epic EHR)

Job description

We are seeking an experienced Senior QA / ETL Test Lead with strong expertise in ETL testing, SQL, Azure Databricks, Python automation, and Agentic AI evaluation. The ideal candidate will lead testing strategies, validate data integrity across legacy and modern data platforms, and ensure AI-generated queries and transformation logic meet accuracy, security, and compliance standards.

Roles and Responsibilities

• Own the overall ETL test strategy, test planning, reconciliation methodology, and risk-based test coverage.
• Design source-to-target validation across data ingestion, transformation, and refined data layers.
• Build and maintain golden reference datasets to evaluate AI-generated queries and transformation logic.
• Develop automated regression frameworks to compare new pipeline outputs with existing production data.
• Perform row-count, aggregate, field-level, data-type, null, duplicate, and referential integrity validations.
• Evaluate AI agents for accuracy, consistency, hallucinations, incorrect joins, missing fields, and confidence calibration.
• Use MLflow or similar tools to track evaluation results, compare model versions, and establish regression gates.
• Validate healthcare data privacy, PHI protection, multi-tenant isolation, and facility-level data access controls.
• Validate data-sharing contracts and ensure generated queries comply with permitted access restrictions.
• Perform query optimization and acquisition-cost validation against baseline queries.
• Lead and mentor QA engineers, review test coverage, manage defects, and report quality metrics.
• Ensure compatibility across Azure Databricks, SQL Server, Vertica, and other data platforms.
• Prepare test evidence and quality reports for client and executive reviews.

Mandatory Skills

• 10+ years of experience in data / ETL testing, including test strategy ownership for a major migration or re-platforming project.
• Strong SQL skills, including complex queries and data reconciliation.
• Hands-on experience with Azure Databricks, PySpark, and Python automation.
• Experience with data quality and automation frameworks such as Great Expectations, dbt tests, pytest, or equivalent.
• Expertise in source-to-target validation, data integrity, regression testing, and test strategy development.
• Experience testing healthcare data, including PHI-scoped, de-identified, or synthetic datasets.
• Experience with at least two data platforms, such as Databricks, SQL Server, or Vertica.
• Strong leadership, defect management, stakeholder communication, and reporting skills.
• Experience with web testing and test automation tools.

Preferred Skills

• Experience testing Agentic AI, LLM, or GenAI systems.
• Familiarity with MLflow, LangSmith, Weights & Balance, or similar evaluation tools.
• Knowledge of LLM-as-a-judge techniques, statistical sampling, confidence intervals, and model calibration.
• Epic EHR experience, particularly Caboodle or Clarity.
• Familiarity with healthcare coding standards such as ICD, CPT, SNOMED, and LOINC.
• Experience with multi-tenant data isolation and SQL-generation system testing.
• Experience presenting test evidence and quality findings to client executives.
• Knowledge of software design principles, SOLID, and Domain-Driven Design.