Senior QA Engineer: ETL, AI Evaluation & Databricks

TalentOla; Chicago (IL); On-site; USD 120,000 - 180,000; Full time

Job summary

TalentOla in Chicago is seeking a Senior QA professional to lead data validation for AI-assisted extraction and transformation. You will own test strategy across ETL, validation and automation, ensuring data integrity and regulatory compliance in a scalable path.

You will design source-to-target validation, establish risk-based priorities, and build automated regression harnesses across Databricks, SQL Server and Vertica, with emphasis on precision and recall metrics and governance.

Qualifications

• 10+ years in data / ETL testing with strategy ownership on large migrations.
• Proven reconciliation and equivalence testing defending conclusions to stakeholders.
• Expert SQL with complex validation queries against unfamiliar schemas.
• Databricks / PySpark for validation; Python for test automation.
• Experience testing PHI-scoped healthcare data and synthetic data helpful.
• Defect management and reporting in Azure Boards; team leadership experience.
• Preferred: evaluating LLM/GenAI systems with MLflow, LangSmith, Weights & Biases.

Responsibilities

• Own test strategy and reconciliation methodology for data validation.
• Define equivalence standards and where they apply in the data path.
• Design source-to-target validation across ingestion and transformation.
• Establish risk-based prioritisation and build a test plan for client review.
• Author test strategy and plan; evaluate agents with a golden reference set.
• Stand up an evaluation harness in MLflow for scoring and comparison.
• Define measures for precision, recall, override rate, calibration and cost.
• Operate the regression gate; ensure no prompt or model change passes without evaluation.
• Characterise failure modes: hallucinated outputs, wrong column selection, data leaks.
• Lead automated regression harness; repeatable execution for agent revisions.
• Design cross-member isolation tests; ensure regulatory-compliant data access.
• Produce evidence packs for executive review; defend methodology and sample sizes.

Skills

ETL testing

SQL

Azure Databricks

Agentic AI evaluation

Web testing

Test automation tools

Tools

Great Expectations

dbt tests

pytest

MLflow

LangSmith

Weights & Biases

Databricks

SQL Server

Vertica

Azure Boards

Job description

TalentOla in Chicago is seeking a Senior QA professional to lead data validation for AI-assisted extraction and transformation. You will own test strategy across ETL, validation and automation, ensuring data integrity and regulatory compliance in a scalable path.

You will design source-to-target validation, establish risk-based priorities, and build automated regression harnesses across Databricks, SQL Server and Vertica, with emphasis on precision and recall metrics and governance.