Lead Agent Evaluation & Instrumentation #AIDA

Singtel Group; Singapore; On-site; SGD 120,000 - 180,000; Full time

Job summary

Singtel is hiring a Lead for Agent Evaluation & Instrumentation to become the independent quality authority for the telco’s AI estate. You will own the Day 1–Day 2 handover, define metrics and instrumentation, and ensure safe, reliable production AI deployments.

You will lead a team of engineers and analysts, maintain independence, chair gate reviews, and drive improvements through telemetry and dashboards in collaboration with multiple internal partners.

Qualifications

• Degree in Computer Science, Data Science, Engineering, or related field
• 10+ years in software/ML/quality with 3+ years leading technical teams
• Experience operating or evaluating ML/LLM systems in production
• Metrics design and observability/instrumentation for AI systems
• Understanding of RAG, SQL agents, autonomous agents and AI/ML model lifecycles
• Strong stakeholder management and the independence to hold a line
• Clear written communication for gate findings and executive reporting
• People leadership and coaching

Responsibilities

• Lead the Agent Evaluation & Instrumentation function as the independent quality authority for the telco’s production AI estate.
• Own the Day 1 → Day 2 handover process and gate authority over every AI agent and AI/ML model entering production.
• Define metrics catalogue, evaluation suites, and instrumentation standards, proving agent quality, safety, and reliability.
• Lead a team of evaluation and instrumentation engineers and analysts, safeguarding independence from delivery pressure.
• Chair operability gate reviews and issue independent go/no-go decisions with findings; track remediation to closure.
• Set instrumentation and telemetry standards and certify telemetry completeness before shadow-run exit.
• Lead, coach the team; build AI/Agent dashboards in consultation with other services and product owners.
• Drive the improvement engine by routing production signals into a prioritised backlog and evidencing improvements.
• Represent the function with cross-functional partners and provide quality inputs to reporting.

Education

Degree in Computer Science, Data Science, Engineering, or related field

Job description

Select how often (in days) to receive an alert:

Lead Agent Evaluation & Instrumentation #AIDA

To lead the next phase of our AI evolution, we’ve launched a new business unitAIDA–Artificial Intelligence & Data Analytics– a strategic engine driving our transformation designed to scale our AI ambitions with precision and purpose.This marks apivotal shift in how we operate, innovate, and serve to embed intelligence into every layer of our business.

AtSingtel, this is more than a technology upgrade. It’s astrategic transformationthat redefines how value is created across the enterprise core—augmenting human capabilitiesand unlocking entirely new potential. It is a transformation journey by aligningpeople, platforms, and processesunder one cohesive strategy. Our mission is to buildAI literacy, and foster a culture whereintelligence empowers people.

We welcome you to join us on a transformational journey that’s reshaping the telecommunications industry — and redefining what’s possible with AI at its core. Grow with us in a workplace that champions innovation, embraces agility, and puts human potential at the heart of everything we do.

Role Summary:

Lead the Agent Evaluation & Instrumentation function as the independent quality authority for the telco’s production AI estate. Own the Day 1 → Day 2 handover process and hold go/no-go gate authority over every AI agent and AI/ML model entering production.

Define what “good” means through the metrics catalogue, evaluation suites, and instrumentation standards, and ensure the organisation can prove agent quality, safety, and reliability before and after launch.

Lead a team of evaluation and instrumentation engineers and analysts, safeguarding evaluation independence from delivery pressure.

How You will Make An Impact:

• Own, publish, and continuously improve the Day 1 → Day 2 handover process, operability gates, and shadow-run exit criteria across all agent archetypes (RAG, SQL, task-based autonomous, AI/ML models, voice overlay).
• Chair operability gate reviews and issue independent go/no-go decisions with written findings; track remediation of conditional passes to closure.
• Set instrumentation and telemetry standards (trace schema, required events, dashboards) and certify telemetry completeness before shadow-run exit.
• Lead, coach the Agent Eval & Instrumentation team to build specific AI/Agent use case operational and evaluation dashboards in consultation with Agent Capabilities Services and Business/Product Owners.
• Drive the improvement engine: convert production signals into a prioritised optimisation backlog routed to Agent Capabilities Services, Data & Harness Engineering, and AI/Agent Operations, and evidence that shipped improvements moved the target metric.
• Represent Agent Eval & Instrumentation with cross-functional partners such as (Agent Capabilities Services, Data & Harness Engineering, Central AI Platform & Ops, AI Lab) and provide quality and reliability inputs to CAIO/board reporting.
• Lead, coach and develop the Agent Eval & Instrumentation team; manage evaluation independence, workload prioritisation, and performance.

Skills for Success:

• Degree in Computer Science, Data Science, Engineering, or related field
• 10+ years in software/ML/quality with 3+ years leading technical teams
• Experience operating or evaluating ML/LLM systems in production
• Metrics design and observability/instrumentation for AI systems
• Understanding of RAG, SQL agents, autonomous agents and AI/ML model lifecycles
• Strong stakeholder management and the independence to hold a line
• Clear written communication for gate findings and executive reporting
• People leadership and coaching

Are you ready to say hello to BIG Possibilities?

Join Singtel to shape what's next and accelerate your career through meaningful work, continuous learning, and real impact.