Associate Director/Senior Manager, Agent Evaluation & Instrumentation (AIDA)

Singtel; Singapore; On-site; Confidential; Full time

Job summary

Singtel is building an AI-first telco and seeks a senior leader to own Agent Evaluation & Instrumentation. You will govern quality for production AI estates, define success metrics, and ensure go/no-go decisions for AI agents and models.

You will lead a team of engineers and analysts, champion independent evaluation, and collaborate with global partners to raise reliability and safety. In this role you will shape telemetry, dashboards, and operational dashboards across agent archetypes, drive

Qualifications

• Degree in Computer Science, Data Science, Engineering, or related field.
• 10+ years in software/ML/quality with 3+ years leading technical teams.
• Experience operating or evaluating ML/LLM systems in production.
• Metrics design and observability/instrumentation for AI systems.
• Understanding of RAG, SQL agents, autonomous agents and AI/ML model lifecycles.
• Strong stakeholder management and the independence to hold a line.
• Clear written communication for gate findings and executive reporting.

Responsibilities

• Lead the Agent Evaluation & Instrumentation function as the independent quality authority for production AI estate.
• Own the Day 1 → Day 2 handover process and go/no-go gate authority over every AI agent and AI/ML model entering production.
• Define what “good” means through metrics catalog, evaluation suites, and instrumentation standards.
• Lead a team of evaluation and instrumentation engineers and analysts, safeguarding independence from delivery pressure.
• Set instrumentation and telemetry standards and certify telemetry completeness before shadow-run exit.
• Lead, coach the team to build AI/Agent dashboards in consultation with stakeholders.
• Drive the improvement engine by translating production signals into a prioritised backlog.
• Represent with cross-functional partners and provide quality inputs to reporting.
• Lead and develop the team; manage independence, workload prioritisation, and performance.

Skills

Experience in ML/LLM systems

Metrics design

Observability/instrumentation

Stakeholder management

Gate findings & executive reporting

People leadership & coaching

Strong written communication

Education

Degree in Computer Science / Data Science / Engineering

Job description

Singtel is where Technology meets Purpose.

We're building an AI-first telco for the future—connecting people, businesses and communities through trusted networks, intelligent digital services and next-generation technology.

From transforming everyday experiences in mobile, broadband, entertainment and lifestyle services to enabling enterprises with AI, 5G+, cloud, cybersecurity, data centres and digital platforms, we create the technology that powers how millions live, work and connect.

At Singtel, your work creates real impact. You'll solve meaningful challenges, shape critical digital infrastructure and help build secure, resilient and AI-enabled solutions that serve businesses, digital ecosystems and communities across the globe.

This is where BIG Possibilities become reality. Whether you're engineering intelligent networks, advancing sovereign AI cloud platforms, strengthening cybersecurity, designing seamless customer experiences or driving sustainable innovation, you'll be part of teams shaping the future of technology.

Backed by decades of expertise and driven by a culture of continuous learning and innovation, Singtel offers the scale, opportunities and support to help you grow your career while making a difference where it matters most.

Role Summary:

To lead the Agent Evaluation & Instrumentation function as the independent quality authority for the telco’s production AI estate. Own the Day 1 → Day 2 handover process and hold go/no-go gate authority over every AI agent and AI/ML model entering production.

Define what “good” means through the metrics catalogue, evaluation suites, and instrumentation standards, and ensure the organisation can prove agent quality, safety, and reliability before and after launch.

Lead a team of evaluation and instrumentation engineers and analysts, safeguarding evaluation independence from delivery pressure.

How You will Make An Impact:

• Own, publish, and continuously improve the Day 1 → Day 2 handover process, operability gates, and shadow-run exit criteria across all agent archetypes (RAG, SQL, task-based autonomous, AI/ML models, voice overlay).
• Chair operability gate reviews and issue independent go/no-go decisions with written findings; track remediation of conditional passes to closure.
• Set instrumentation and telemetry standards (trace schema, required events, dashboards) and certify telemetry completeness before shadow-run exit.
• Lead, coach the Agent Eval & Instrumentation team to build specific AI/Agent use case operational and evaluation dashboards in consultation with Agent Capabilities Services and Business/Product Owners.
• Drive the improvement engine: convert production signals into a prioritised optimisation backlog routed to Agent Capabilities Services, Data & Harness Engineering, and AI/Agent Operations, and evidence that shipped improvements moved the target metric.
• Represent Agent Eval & Instrumentation with cross-functional partners such as (Agent Capabilities Services, Data & Harness Engineering, Central AI Platform & Ops, AI Lab) and provide quality and reliability inputs to CAIO/board reporting.
• Lead, coach, and develop the Agent Eval & Instrumentation team; manage evaluation independence, workload prioritisation, and performance.

Skills for Success:

• Degree in Computer Science, Data Science, Engineering, or related field
• 10+ years in software/ML/quality with 3+ years leading technical teams
• Experience operating or evaluating ML/LLM systems in production
• Metrics design and observability/instrumentation for AI systems
• Understanding of RAG, SQL agents, autonomous agents and AI/ML model lifecycles
• Strong stakeholder management and the independence to hold a line
• Clear written communication for gate findings and executive reporting
• People leadership and coaching

Are you ready to say hello to BIG Possibilities?

Join Singtel to shape what's next and accelerate your career through meaningful work, continuous learning, and real impact.