Principal AI Engineer

Mastercard; Singapore; On-site; SGD 180,000 - 260,000; Full time

Job summary

Mastercard seeks a Principal AI Engineer on the AI Foundations team to advance production-grade agentic systems, memory-driven workflows, and safe tool use. You will shape technical direction, design scalable pipelines, and mentor engineers while delivering measurable business outcomes at scale.

Responsibilities include leading design and deployment of multi-agent solutions, establishing LLMOps practices, and collaborating with platform, security, governance, and product stakeholders to ensure

Qualifications

• Bachelor's degree in Computer Science, Engineering, Data Science, Applied Mathematics, or related technical field.
• Strong foundation in software engineering, distributed systems, and applied machine learning for production AI systems.
• Demonstrated understanding of responsible AI, model/system risk, privacy/security considerations, and governance requirements for enterprise deployments.

Responsibilities

• Serve as an established subject matter expert in AI Engineering, influencing stakeholders and shaping technical direction across multiple initiatives.
• Architect, design, develop, and maintain advanced AI/ML systems, with emphasis on complex agentic solutions (multi-agent orchestration, tool/function-calling, memory, reflection, autonomy policies).
• Lead production implementation of agentic AI systems, including scalable training and evaluation pipelines, deployment frameworks, and runtime orchestration patterns.
• Define and implement safe tool-use patterns: structured outputs, robust error handling, permissioning and auditability, HITL approval steps for sensitive actions, and guardrail enforcement.
• Establish end-to-end AgentOps/LLMOps practices for agentic systems: release pipelines for prompts/tools/policies, canary strategies, safe rollback mechanisms, and continuous regression/safety evaluations as release gates.
• Build and optimize data ingestion, preprocessing, feature/embedding engineering, and retrieval/memory workflows to improve grounding quality and reduce failure modes.
• Own production observability for agentic systems: trace capture, cost/token telemetry, latency and reliability SLOs, and incident response practices for agent failures.
• Implement drift detection and performance decay monitoring (data drift, concept drift), and automate model/agent retraining, policy updates, and redeployment to maintain output quality over time.
• Drive measurable improvements in system effectiveness, safety, and efficiency by defining success metrics and continuously improving evaluation coverage.
• Mentor and grow senior and junior engineers through design reviews, code reviews, hands-on coaching, and the creation of reusable patterns, playbooks, and standards for agentic delivery.

Skills

Agentic System Architecture

Production Engineering for Agents

AgentOps / LLMOps

Evaluation & Safety

Observability & Reliability

Data/Feature/Retrieval Engineering

Cross-Functional Influence

Education

Bachelor's degree in Computer Science, Engineering or related field

Advanced degree preferred

Tools

CI/CD for prompts/tools/policies

Vector search

Job description

Our Purpose

Mastercard powers economies and empowers people in 200+ countries and territories worldwide. Together with our customers, we’re helping build a sustainable economy where everyone can prosper. We support a wide range of digital payments choices, making transactions secure, simple, smart and accessible. Our technology and innovation, partnerships and networks combine to deliver a unique set of products and services that help people, businesses and governments realize their greatest potential.

Title and Summary

Principal AI Engineer

Overview

As a Principal AI Engineer on the AI Foundations team, you are an established subject matter expert in AI Engineering who applies expert knowledge and experience to drive achievement of key area goals and initiatives by making significant improvements to new or existing products, services, and/or processes. You lead the design and operationalization of complex, production-grade agentic systems—particularly multi‑agent, multi‑tool solutions that plan, call tools safely, maintain memory, and continuously improve through evaluation and feedback. You influence technical direction across programs, set engineering standards for reliability and responsible AI, and partner with platform, security, governance, and product stakeholders to ship measurable business outcomes.

Key Responsibilities

• Serve as an established subject matter expert in AI Engineering, influencing stakeholders and shaping technical direction across multiple initiatives.
• Architect, design, develop, and maintain advanced AI/ML systems, with emphasis on complex agentic solutions (multi‑agent orchestration, tool/function‑calling, memory, reflection/self‑correction, and autonomy policies).
• Lead production implementation of agentic AI systems, including scalable training and evaluation pipelines, deployment frameworks, and runtime orchestration patterns.
• Define and implement safe tool‑use patterns: structured outputs, robust error handling, permissioning and auditability, human‑in‑the‑loop (HITL) approval steps for sensitive actions, and guardrail enforcement.
• Establish end‑to‑end AgentOps/LLMOps practices for agentic systems: release pipelines for prompts/tools/policies, canary strategies, safe rollback mechanisms, and continuous regression/safety evaluations as release gates.
• Build and optimize data ingestion, preprocessing, feature/embedding engineering, and retrieval/memory workflows to improve grounding quality and reduce failure modes.
• Own production observability for agentic systems: trace capture, cost/token telemetry, latency and reliability SLOs, and incident response practices for agent failures.
• Implement drift detection and performance decay monitoring (data drift, concept drift), and automate model/agent retraining, policy updates, and redeployment to maintain output quality over time.
• Drive measurable improvements in system effectiveness, safety, and efficiency by defining success metrics (task success, intervention rate, policy violations, cost and latency per task) and continuously improving evaluation coverage.
• Mentor and grow senior and junior engineers through design reviews, code reviews, hands‑on coaching, and the creation of reusable patterns, playbooks, and standards for agentic delivery.

Key Skills

• Agentic System Architecture: multi‑agent orchestration, planning and goal decomposition, tool/function‑calling, memory and retrieval patterns, and behavior optimization.
• Production Engineering for Agents: scalable deployment frameworks, high‑availability runtime design, robust failure handling, and operational readiness.
• AgentOps / LLMOps: CI/CD for prompts/tools/policies, release governance, experiment management, canaries, feature flags, and safe rollbacks.
• Evaluation & Safety: automated eval pipelines (behavioral, regression, adversarial), HITL workflows, guardrails, policy enforcement, and red‑team testing as shipping gates.
• Observability & Reliability: distributed tracing for reasoning/tool calls, telemetry, cost and latency management, SLOs/SLIs, and incident response for agent systems.
• Data/Feature/Retrieval Engineering: ingestion and preprocessing pipelines, feature/embedding workflows, vector search and reranking, caching, and trace stores.
• Cross‑Functional Influence: ability to align platform, security, governance, and product partners; communicate tradeoffs; and drive adoption of standards across teams.

Qualifications

• Bachelor’s degree in Computer Science, Engineering, Data Science, Applied Mathematics, or related technical field; advanced degree preferred.
• Strong foundation in software engineering, distributed systems, and applied machine learning relevant to production AI systems.
• Demonstrated understanding of responsible AI, model/system risk, privacy/security considerations, and governance requirements for enterprise deployments.

Experience Requirements (Production‑Focused)

• Demonstrated, sustained ownership of production AI/ML systems, including design, build, deployment, and ongoing lifecycle operations.
• Real‑world experience shipping complex agentic systems into production, including multi‑agent coordination and multi‑tool integration with safe action policies.
• Hands‑on experience building production pipelines for evaluation, monitoring, versioning, and continuous improvement (including retraining or policy/guardrail updates).
• Proven ability to define and operationalize observability and reliability practices for agentic systems (traceability, telemetry, SLOs, incident management).
• Track record of influencing architecture and standards across multiple teams or programs, and mentoring engineers to raise overall engineering rigor.

Corporate Security Responsibility

All Activities Involving Access To Mastercard Assets, Information, And Networks Comes With An Inherent Risk To The Organization And, Therefore, It Is Expected That Every Person Working For, Or On Behalf Of, Mastercard Is Responsible For Information Security And Must:

• Abide by Mastercard’s security policies and practices;
• Ensure the confidentiality and integrity of the information being accessed;
• Report any suspected information security violation or breach, and
• Complete all periodic mandatory security trainings in accordance with Mastercard’s guidelines.