Senior AI Agent Operations & Evaluation Engineer

Singtel; Singapore; On-site; Confidential; Full time

Job summary

Singtel invites you to join its AI & Data Analytics unit, AIDA, to lead the live GenAI agents and build the evaluation and observability that proves accuracy, safety, and improvement. You will ensure agents stay healthy, triage incidents, and minimize downtime while driving drift detection and quality assurance.

Work closely with Day 1 build teams, solution architects and business owners across Singtel Singapore to deliver measurable outcomes and reliable AI services.

Qualifications

• Degree in Computer Science, Engineering, Data Science, or a related field.
• 6-10 years across software / SRE / data or ML engineering.
• Hands-on production operations experience and evaluation, data-analysis or quality experience.
• Experience with LLM / agentic systems.
• Monitoring, alerting and observability; log / trace analysis.
• Evaluation methods and metric design for AI systems.
• Dashboarding tools and Python.
• Understanding of LLM / agent architectures and failure modes.
• LLM observability and evaluation tooling (e.g. tracing, eval harnesses).
• Incident management practice.

Responsibilities

• Build and maintain the operational observability dashboard for each AI / agent and Classical AI ML use case, across system health, performance, quality, safety and business-outcome metrics.
• Define metrics with build teams and product owners; consume telemetry from the platform.
• Build and run evaluation suites (offline and in-production) and detect drift and quality decay.
• Monitor live agents via dashboards and automated safeguards; act on alerts.
• Perform first-level triage and fixes (restart, config, cache, routing).
• Manage incidents with full evidence bundles and end-to-end ownership.
• Contribute to the handover gate to ensure operability and evaluation evidence.

Skills

LLM / agentic systems

Monitoring, alerting & observability

Dashboarding tools

Python

Incident management

Technical communication

Data analysis

Gate findings writing

Education

Degree in CS/Engineering/Data Science

Tools

LLM observability tooling

Tracing

Eval harnesses

Job description

Singtel invites you to join its AI & Data Analytics unit, AIDA, to lead the live GenAI agents and build the evaluation and observability that proves accuracy, safety, and improvement. You will ensure agents stay healthy, triage incidents, and minimize downtime while driving drift detection and quality assurance.

Work closely with Day 1 build teams, solution architects and business owners across Singtel Singapore to deliver measurable outcomes and reliable AI services.