Agentic Testing

CitiusTech; Chicago (IL); Hybrid; USD 120,000 - 165,000; Full time

Job summary

CitiusTech Inc is seeking a Senior QA professional to own test strategy for ETL pipelines, leveraging SQL, Azure Databricks, and AI evaluation to ensure data integrity. You will build automated regression harnesses, define equivalence across raw to domain layers, and lead a team of QA engineers through complex healthcare data validation.

Hybrid role in Chicago, IL, with a focus on scalable validation strategies, risk-based testing, and client-facing artifacts to demonstrate methodology and

Qualifications

• 10+ years in data / ETL testing with strategy ownership on large migrations.
• Proven reconciliation and equivalence testing of rebuilt pipelines.
• Expert SQL with complex validation against unfamiliar schemas.
• Databricks / PySpark for validation at data volume, Python for automation.
• Experience testing PHI-scoped healthcare data and working with EHR data.
• Defect management and reporting; team leadership and coverage reviews.

Responsibilities

• Own test strategy and reconciliation methodology across data pipelines.
• Define equivalence standards and layer-specific validation rules.
• Design source-to-target validation for ingestion, transformation, and domain refinement.
• Prioritise risks and author Test Strategy & Plan for client review.
• Maintain a golden reference set and stand up an evaluation harness in MLflow.
• Define measures: precision, recall, escalation rate, and cost per case.
• Lead automated regression harness and cross-system validation across Databricks, SQL Server, Vertica.

Skills

ETL testing

SQL

Azure Databricks

Agentic AI testing

Web testing

Test automation

Education

Engineering degree

Tools

MLflow

LangSmith

Weights & Biases

Azure Boards

Great Expectations

dbt tests

pytest

Job description

CitiusTech, we constantly strive to solve the industry's greatest challenges with technology, creativity, and agility. With over 8,500 healthcare technology professionals worldwide, CitiusTech powers healthcare digital innovation, business transformation, and industry-wide convergence for over 140 organizations through next-generation technologies, solutions, and products. We aim to accelerate the transition to a human-first, sustainable, and digital healthcare ecosystem with the world's leading Healthcare and life sciences organizations and our partners.

Here is an opportunity for you to make a difference and collaborate with global leaders to shape the future of healthcare and positively impact human lives.

Our vision: -

To inspire new possibilities for the health ecosystem with technology and human ingenuity.

What is in it for you?

Looking for a Senior QA professional skilled in ETL, SQL, Azure Databricks, Agentic AI evaluation, web testing, and test automation tools, driving data integrity, compliance, and scalable validation strategies to join our team in Chicago, IL

Roles and Responsibilities: -

• We are building AI agents that generate the extraction queries and transformation logic instead plus the Workbench through which humans review and approve everything they produce.
• The agents emit metadata; the client's existing data platform executes it. What you’ll do Own test strategy and reconciliation methodology Define the equivalence standard: what must match exactly, what may vary within tolerance, and at which layer each applies Design source-to-target validation across the redesigned path — raw ingestion, consolidated transformation, domain refined
• Establish risk-based prioritisation; the schedule does not permit exhaustive coverage of hundreds of tables Author the Test Strategy and Test Plan for review with the client in Phase 2 Own agent evaluation Build and maintain a golden reference set from existing production extracts, validated with the Epic SME, with strict separation between evaluation data and anything used to develop or tune the agents Stand up the evaluation harness in MLflow — scoring runs, versioning, candidate comparison
• Define and report the measures that matter mapping precision and recall, override rate, confidence calibration, run-to-run consistency, escalation correctness and cost per case Operate the regression gate — no prompt, skill or model change is promoted without passing evaluation.
• A prompt change can silently degrade quality with no data-level symptom Characterise how agents fail, not just how often: hallucinated tables, plausible-but-wrong column selection, silent omission of required fields, mis-selected join grain.
• Own the automated regression harness Automated comparison of new-path output against current production extracts Row counts, aggregate reconciliation, field-level comparison, type and precision validation, null and duplicate profiling, referential integrity Repeatable execution so every agent revision can be re-validated cheaply
• Design the control-critical test suites Member, health-system and facility isolation. Facility mapping is moving downstream, creating real regression risk against the guarantee that one member cannot see another member's data.
• A cross-member leak is the highest-severity failure mode on this engagement Contract-scope validation.
• Data-sharing contracts restrict access at facility, column and product level. Verifying that generated queries stay inside permitted scope is a testable control with legal consequences if missed Acquisition cost validation.
• Confirm optimised queries touch fewer source tables than baseline and that facility consolidation genuinely works Lead and report Direct two QA engineers; set standards, review coverage, own the defect taxonomy
• Confirm downstream compatibility across Databricks, SQL Server and Vertica Produce evidence packs suitable for client executive review, and defend methodology, sample sizes and statistical claims under scrutiny.

Experience: -

10+ Years

Location: -

Chicago, IL (Hybrid - 3 days' Work from Office)

Duration: -

12+ Months

Educational Qualifications: -

Engineering Degree - BE/ME/BTech/MTech/BSc/MSc.

Technical certification in multiple technologies is desirable.

Skills: -

Mandatory skills

• 10+ years in data / ETL testing, including strategy ownership on at least one large migration or re-platforming programme
• Proven reconciliation and equivalence testing — demonstrating that a rebuilt pipeline produces output equivalent to a legacy one, and defending that conclusion to stakeholders
• Expert SQL — complex analytical validation queries written independently against unfamiliar schemas Databricks / PySpark for validation at data volume Python for test automation and harness development
• Data quality and test automation frameworks — Great Expectations, dbt tests, pytest or equivalent Test strategy authorship — coverage models, risk-based prioritisation, entry and exit criteria, defect taxonomy multi-target validation across at least two of Databricks, SQL Server, Vertica or comparable
• Experience testing PHI-scoped healthcare data, including de-identified or synthetic test data Defect management and reporting in Azure Boards Team leadership — setting standards and reviewing others' coverage
• Ability to state an uncomfortable quality position to senior stakeholders and hold it Preferred Experience evaluating LLM or GenAI systems — MLflow, LangSmith, Weights & Biases or equivalent; LLM-as-judge design and its failure modes Applied statistics — sampling design, confidence intervals, significance testing, calibration measurement Epic EHR data experience
• Caboodle or Clarity Clinical coding systems: ICD, CPT, SNOMED, LOINC Multi-tenant data isolation testing Experience testing code- or SQL-generation systems Prior work where test evidence was presented directly to client executives
• Software Design and Best Practices: Apply software design principles such as SOLID and Domain-Driven Design.
• Understand code patterns and practices.
• Maintain currency in technical skills and industry trends.

Skills

ETL, SQL, Azure Databricks, Agentic AI testing, Web testing Any Data Automation tool.

Life at CitiusTech

We focus on building highly motivated engineering teams and thought leaders with an entrepreneurial mindset, centred on our core values of Passion, Respect, Openness, Unity, and Depth (PROUD) of knowledge. Our success lies in creating a fun, transparent, non-hierarchical, diverse work culture that focuses on continuous learning and work-life balance.

Rated by our employees as the 'Great Place to Work' for' according to the Great Place to Work survey. We offer you a comprehensive set of benefits to ensure that you have a long and rewarding career with us.

Equal Opportunity Employer

CitiusTech is an Equal Opportunity Employer. We celebrate diversity and are committed to creating an inclusive environment for all employees. All qualified applicants will receive consideration for employment without regard to race, colour, religion, gender, gender identity or expression, sexual orientation, national origin, genetics, disability, age, or veteran status.

Our EVP

Be You Be Awesome is our EVP and it reflects our continuing efforts to create CitiusTech as a great place to work where our employees can thrive, both personally and professionally. It encompasses the unique benefits and opportunities we offer to support your growth, well-being, and success throughout your journey with us and beyond. Together with our clients, we are solving some of the greatest healthcare challenges and positively impacting human lives. Welcome to the world of Faster Growth, Higher Learning, and Stronger Impact.