ETL Tester with AI Experience

AddSource

Irving (TX)

Hybrid

USD 120,000 - 150,000

Full time

42 hours ago
Be an early applicant

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Job summary

AddSource is seeking a senior ETL Tester with AI experience to lead test strategy and hands-on validation across ETL workflows, agentic AI evaluation, and automated regression for large data migrations. The role focuses on data integrity, compliance, and scalable validation across Databricks, SQL Server, and Vertica.

The candidate will drive testing standards and liaise with executives. The position is based in Chicago, IL with a hybrid work arrangement and requires 10+ years in data/ETL

Qualifications

  • 10+ years of experience in data and ETL testing, including test strategy ownership on at least one large migration or re-platforming program.
  • Proven experience in reconciliation and equivalence testing, demonstrating that rebuilt pipelines produce outputs equivalent to legacy systems.
  • Expert-level SQL skills, including independently writing complex analytical validation queries against unfamiliar schemas.
  • Experience with Azure Databricks and PySpark for large-volume data validation.
  • Python programming for test automation and test harness development.
  • Experience with data quality and test automation frameworks such as Great Expectations, dbt tests, pytest, or equivalent.
  • Experience in test strategy development, coverage models, risk-based prioritization, entry and exit criteria, and defect classification.
  • Multi-platform validation experience across at least two of Databricks, SQL Server, Vertica, or comparable platforms.
  • Experience testing PHI-scoped healthcare data, including de-identified or synthetic test data.
  • Defect management and reporting using Azure Boards.
  • Team leadership experience, including setting testing standards and reviewing team members' test coverage.
  • Strong communication skills and the ability to present and defend quality findings to senior stakeholders.

Responsibilities

  • Test AI agents that generate extraction queries and transformation logic, along with the workbench through which humans review and approve their outputs.
  • Validate agent-generated metadata executed by the client’s existing data platform.
  • Understand and validate the redesigned data processing workflow across raw ingestion, consolidated transformation, and domain-refined layers.
  • Own the test strategy and reconciliation methodology.
  • Define equivalence standards, identifying what must match exactly, what may vary within tolerance, and at which layer each applies.
  • Design source-to-target validation across the redesigned data processing workflow.
  • Establish risk-based prioritization, as the schedule does not permit exhaustive coverage of hundreds of tables.
  • Author the test strategy and test plan for review with the client in Phase 2.
  • Build and maintain a golden reference set using existing production extracts, validated with the Epic SME.
  • Maintain strict separation between evaluation data and data used to develop or tune the agents.
  • Establish the evaluation harness in MLflow, including scoring runs, versioning, and candidate comparison.
  • Define and report key metrics, including mapping precision and recall, override rate, confidence calibration, run-to-run consistency, escalation correctness, and cost per case.
  • Operate the regression gate to ensure no prompt, skill, or model change is promoted without passing evaluation.
  • Identify and characterize agent failures, including hallucinated tables, incorrect column selection, omission of required fields, and incorrectly selected join grain.
  • Own the automated regression harness.
  • Automate comparison of new-path outputs against current production extracts.
  • Validate row counts, aggregate reconciliation, field-level comparisons, data types, precision, null and duplicate profiling, and referential integrity.
  • Establish repeatable execution so every agent revision can be revalidated efficiently.
  • Design control-critical test suites for member, health-system, and facility isolation.
  • Validate facility mapping changes and ensure one member cannot access another member's data.
  • Perform contract-scope validation.
  • Verify that generated queries comply with data-sharing restrictions at facility, column, and product levels.
  • Validate acquisition costs by confirming optimized queries access fewer source tables than the baseline and that facility consolidation works as expected.
  • Directly lead two QA engineers.
  • Establish testing standards, review test coverage, and manage defect classification.
  • Confirm downstream compatibility across Databricks, SQL Server, and Vertica.
  • Produce evidence packs suitable for client executive review.
  • Defend testing methodology, sample sizes, and statistical findings when required.

Skills

ETL
SQL
Azure Databricks
Agentic AI Testing
Web Testing
Data Automation Tools

Education

Engineering degree BE/ME/BTech/MTech/BSc/MSc

Tools

Python
Great Expectations
dbt tests
pytest
Azure Boards

Job description

Job Description

Job Title: ETL Tester with AI Experience

Location: Chicago, IL (Hybrid 3 days' work from office)

Experience: 10+ Years

Skills: ETL, SQL, Azure Databricks, Agentic AI Testing, Web Testing, Data Automation Tools

We are looking for a senior QA professional skilled in ETL, SQL, Azure Databricks, agentic AI evaluation, web testing, and test automation tools. The ideal candidate will drive data integrity, compliance, and scalable validation strategies.

Roles and Responsibilities Agentic AI Testing and Evaluation
  • Test AI agents that generate extraction queries and transformation logic, along with the workbench through which humans review and approve their outputs.
  • Validate agent-generated metadata executed by the client's existing data platform.
  • Understand and validate the redesigned data processing workflow across raw ingestion, consolidated transformation, and domain-refined layers.
Test Strategy and Reconciliation
  • Own the test strategy and reconciliation methodology.
  • Define equivalence standards, identifying what must match exactly, what may vary within tolerance, and at which layer each applies.
  • Design source-to-target validation across the redesigned data processing workflow.
  • Establish risk-based prioritization, as the schedule does not permit exhaustive coverage of hundreds of tables.
  • Author the test strategy and test plan for review with the client in Phase 2.
Agent Evaluation
  • Build and maintain a golden reference set using existing production extracts, validated with the Epic SME.
  • Maintain strict separation between evaluation data and data used to develop or tune the agents.
  • Establish the evaluation harness in MLflow, including scoring runs, versioning, and candidate comparison.
  • Define and report key metrics, including mapping precision and recall, override rate, confidence calibration, run-to-run consistency, escalation correctness, and cost per case.
  • Operate the regression gate to ensure no prompt, skill, or model change is promoted without passing evaluation.
  • Identify and characterize agent failures, including hallucinated tables, incorrect column selection, omission of required fields, and incorrectly selected join grain.
Automated Regression Testing
  • Own the automated regression harness.
  • Automate comparison of new-path outputs against current production extracts.
  • Validate row counts, aggregate reconciliation, field-level comparisons, data types, precision, null and duplicate profiling, and referential integrity.
  • Establish repeatable execution so every agent revision can be revalidated efficiently.
Data Security and Control Testing
  • Design control-critical test suites for member, health-system, and facility isolation.
  • Validate facility mapping changes and ensure one member cannot access another member's data.
  • Perform contract-scope validation.
  • Verify that generated queries comply with data-sharing restrictions at facility, column, and product levels.
  • Validate acquisition costs by confirming optimized queries access fewer source tables than the baseline and that facility consolidation works as expected.
Team Leadership and Reporting
  • Directly lead two QA engineers.
  • Establish testing standards, review test coverage, and manage defect classification.
  • Confirm downstream compatibility across Databricks, SQL Server, and Vertica.
  • Produce evidence packs suitable for client executive review.
  • Defend testing methodology, sample sizes, and statistical findings when required.
Educational Qualifications
  • Engineering degree BE, ME, BTech, MTech, BSc, or MSc.
  • Technical certifications in multiple technologies are desirable.
Mandatory Skills
  • 10+ years of experience in data and ETL testing, including test strategy ownership on at least one large migration or re-platforming program.
  • Proven experience in reconciliation and equivalence testing, demonstrating that rebuilt pipelines produce outputs equivalent to legacy systems.
  • Expert-level SQL skills, including independently writing complex analytical validation queries against unfamiliar schemas.
  • Experience with Azure Databricks and PySpark for large-volume data validation.
  • Python programming for test automation and test harness development.
  • Experience with data quality and test automation frameworks such as Great Expectations, dbt tests, pytest, or equivalent.
  • Experience in test strategy development, coverage models, risk-based prioritization, entry and exit criteria, and defect classification.
  • Multi-platform validation experience across at least two of Databricks, SQL Server, Vertica, or comparable platforms.
  • Experience testing PHI-scoped healthcare data, including de-identified or synthetic test data.
  • Defect management and reporting using Azure Boards.
  • Team leadership experience, including setting testing standards and reviewing team members' test coverage.
  • Strong communication skills and the ability to present and defend quality findings to senior stakeholders.
Preferred Qualifications
  • Experience evaluating LLM or GenAI systems using MLflow, LangSmith, Weights & Biases, or equivalent tools.
  • Experience with LLM-as-judge design and understanding its limitations and failure modes.
  • Applied statistics, including sampling design, confidence intervals, significance testing, and calibration measurement.
  • Epic EHR data experience.
  • Experience with Caboodle or Clarity.
  • Knowledge of clinical coding systems such as ICD, CPT, SNOMED, and LOINC.
  • Experience with multi-tenant data isolation testing.
  • Experience testing code-generation or SQL-generation systems.
  • Previous experience presenting test evidence directly to client executives.
Software Design and Best Practices
  • Apply software design principles such as SOLID and Domain-Driven Design.
  • Understand software code patterns and best practices.
  • Maintain up-to-date technical skills and industry knowledge.
Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Senior AI Systems Quality Engineer
Senior AI Systems Quality Engineer

Abacus Insights • United States

On-site
USD 120,000 - 180,000
ETL QA -Databricks
ETL QA -Databricks

VeeRteq Solutions LLC • Irving (TX)

On-site
USD 120,000 - 150,000
Agentic Testing
Agentic Testing

CitiusTech • Chicago (IL)

Hybrid
USD 120,000 - 165,000
EDW QA & ETL Tester — Data Quality & Automation
EDW QA & ETL Tester — Data Quality & Automation

Vergence Group • United States

On-site
USD 70,000 - 90,000
Quality Assurance Analyst
Quality Assurance Analyst

eSense Incorporated • United States

On-site
USD 85,000 - 110,000
ETL Manual Tester
ETL Manual Tester

Quality Ai • United States

Remote
USD 90,000 - 100,000
Competitive compensation
Remote work within US
401(k) plan
+2
Data QA Engineer/Analyst
Data QA Engineer/Analyst

TechDigital Group • Reston (VA)

On-site
USD 80,000 - 100,000
Test Architect
Test Architect

TechDigital Group • Torrance (CA)

Hybrid
USD 120,000 - 150,000
ETL Tester
ETL Tester

Virtusa Corporation • New York (NY)

On-site
USD 110,000 - 160,000
Quality Engineering Lead-Data, AI & Testing
Quality Engineering Lead-Data, AI & Testing

Mastek • United States

On-site
USD 150,000 - 210,000