Head of AI Agent Evaluation & Instrumentation

Singtel; Singapore; On-site; Confidential; Full time

Job summary

Singtel, powered by AIDA, seeks a leader to head the Agent Evaluation & Instrumentation function as the independent quality authority for the telco’s production AI estate. Own the Day 1 → Day 2 handover process and hold gate authority over every AI agent and ML model entering production.

Define what “good” means through metrics, instrumentation standards, and ensure reliability before and after launch. Lead a team of evaluators and analysts, safeguarding independence from delivery pressure.

Qualifications

• Degree in Computer Science, Data Science, Engineering, or related field.
• 10+ years in software/ML/quality with 3+ years leading technical teams.
• Experience operating or evaluating ML/LLM systems in production.
• LLM/agent evaluation methods (offline & online, LLM-as-judge).
• Metrics design and observability/instrumentation for AI systems.
• Understanding of RAG, SQL agents, autonomous agents and AI/ML model lifecycles.
• Strong stakeholder management and the independence to hold a line.
• Clear written communication for gate findings and executive reporting.
• People leadership and coaching.

Responsibilities

• Own, publish, and continuously improve the Day 1 → Day 2 handover process, operability gates, and shadow-run exit criteria across all agent archetypes (RAG, SQL, task-based autonomous, AI/ML models, voice overlay).
• Chair operability gate reviews and issue independent go/no-go decisions with written findings; track remediation of conditional passes to closure.
• Set instrumentation and telemetry standards (trace schema, required events, dashboards) and certify telemetry completeness before shadow-run exit.
• Lead, coach the Agent Eval & Instrumentation team to build specific AI/Agent use case operational and evaluation dashboards in consultation with Agent Capabilities Services and Business/Product Owners.
• Drive the improvement engine: convert production signals into a prioritised optimisation backlog routed to Agent Capabilities Services, Data & Harness Engineering, and AI/Agent Operations, and evidence that shipped improvements moved the target metric.
• Represent Agent Eval & Instrumentation with cross-functional partners such as (Agent Capabilities Services, Data & Harness Engineering, Central AI Platform & Ops, AI Lab) and provide quality and reliability inputs to CAIO/board reporting.
• Lead, coach, and develop the Agent Eval & Instrumentation team; manage evaluation independence, workload prioritisation, and performance.

Skills

Software/ML/Quality leadership

Team leadership

Production ML/LLM experience

LLM/agent evaluation methods

Metrics & observability

RAG and SQL agents understanding

Stakeholder management

Executive communication

People leadership

Education

Degree in Computer Science, Data Science, Engineering, or related field

Job description

Singtel, powered by AIDA, seeks a leader to head the Agent Evaluation & Instrumentation function as the independent quality authority for the telco’s production AI estate. Own the Day 1 → Day 2 handover process and hold gate authority over every AI agent and ML model entering production.

Define what “good” means through metrics, instrumentation standards, and ensure reliability before and after launch. Lead a team of evaluators and analysts, safeguarding independence from delivery pressure.