AI Quality Engineering Lead

Mastercard; O’Fallon (MO); On-site; USD 140,000 - 231,000; Full time

Benefits offered by this job

Annual bonus

401k match

Generous leave

On-site fitness facilities

Job summary

Mastercard's AI Engineering leadership seeks a Manager to direct AI testing for Generative AI systems within the Operational Intelligence Program. You will define evaluation frameworks, red team strategies, and LLMOps quality gates to ensure accuracy, safety, and enterprise readiness.

You will build test suites for multi-agent workflows, lead adversarial testing, and oversee bias audits, partnering with Gen AI engineers and product stakeholders to embed quality from day one.

Qualifications

• Master's/Bachelor's degree in Computer Science, AI/ML, or Software Engineering, with hands-on AI/ML quality engineering or LLM testing leadership experience.

Responsibilities

• Design and own end-to-end LLM evaluation frameworks, including prompt regression, scoring, benchmarking, and hallucination detection.

Skills

Python

SQL

LLM testing

MLOps/LLMOps

AI testing frameworks

RAGAS

TruLens

DeepEval

LangChain

Education

Master's/ Bachelor's in CS/AI or Software Engineering

Tools

Databricks

Grafana

Datadog

CloudWatch

Job description

Mastercard's AI Engineering leadership seeks a Manager to direct AI testing for Generative AI systems within the Operational Intelligence Program. You will define evaluation frameworks, red team strategies, and LLMOps quality gates to ensure accuracy, safety, and enterprise readiness.

You will build test suites for multi-agent workflows, lead adversarial testing, and oversee bias audits, partnering with Gen AI engineers and product stakeholders to embed quality from day one.