Senior AI Ops Engineer: Production Monitoring & Governance

Mastercard; Helena (MT); On-site; USD 120,000 - 190,000; Full time

Job summary

Mastercard is seeking a Lead AI Ops Engineer to own end-to-end production monitoring for AI agents. You will track metrics, SLAs, and guardrails while providing L1/L2 support as new agentic services come online.

You will diagnose issues via eval scores, drift alerts, and root-cause analysis, ensuring context retrieval remains current and learning loops stay properly governed in production.

Qualifications

• Experience monitoring production AI/ML systems and observability.
• Strong analytical skills to diagnose root causes across models.
• Understanding of governance and responsible AI practices is a plus.

Responsibilities

• Own end to end production monitoring for one or more AI agents, tracking key metrics, KPIs, accuracy, latency, SLA breaches, guardrail triggers, and out of scope rates through dedicated observability dashboards
• Serve as L1/L2 production support, providing first response monitoring, triage, and escalation to tech teams as new agentic services go live
• Identify and diagnose issues by monitoring eval scores and drift alerts, inspecting failing traces and patterns, and classifying root causes such as intent, tool, parameter, or hallucination errors
• Evaluate whether the agent’s reasoning remains correct in production, checking whether context retrieval stays current, whether resolutions match ground truth, and whether the learning loop is drifting
• Monitor and manage post production reliability, including API and authentication failures and issues originating from external AI endpoints
• Partner with AI Engineering to prioritize fixes, run experiments on prompts, context, and routing in Dev, and validate improvements via evals before deployment
• Embed governance within the agent’s learning loop so that approval workflows and audit logging travel with every production change

Job description

Mastercard is seeking a Lead AI Ops Engineer to own end-to-end production monitoring for AI agents. You will track metrics, SLAs, and guardrails while providing L1/L2 support as new agentic services come online.

You will diagnose issues via eval scores, drift alerts, and root-cause analysis, ensuring context retrieval remains current and learning loops stay properly governed in production.