Senior AI Agent Ops & Evaluation Engineer

Singtel Group; Singapore; On-site; SGD 180,000 - 240,000; Full time

Job summary

Singtel is seeking a Senior Agent Operations & Evaluation Engineer to lead live GenAI agents and build robust evaluation/observability to ensure accuracy, safety, and continuous improvement. You will oversee agent health, triage incidents, and minimize downtime while collaborating with Day 1 build teams and business owners.

The role emphasizes measuring technical and outcome metrics, implementing proactive drift detection, and delivering insights through dashboards.

Qualifications

• Degree in Computer Science, Engineering, Data Science, or a related field.
• 6 - 10 years across software / SRE / data or ML engineering.
• Hands-on production operations experience AND hands-on evaluation, data-analysis or quality experience.
• Experience with LLM / agentic systems.
• Monitoring, alerting and observability; log / trace analysis.
• Evaluation methods and metric design for AI systems.
• Dashboarding tools and Python.
• Understanding of LLM / agent architectures and failure modes.
• LLM observability and evaluation tooling (e.g. tracing, eval harnesses).
• Incident management practice.
• Composure and clear communication under incident pressure.
• Analytical rigour and attention to detail.
• Ability to context-switch and prioritise across two disciplines.
• Clear technical writing for gate findings.
• Ability to influence build teams on quality.
• Understanding of telco customer intents and journeys.

Responsibilities

• Build and maintain the operational observability dashboard for each AI / agent and Classical AI ML use case, across system health, performance, quality, safety and business-outcome metrics.
• Define, with build teams and solution architects (technical metrics) and product / business owners (outcome metrics), what needs to be measured per use case; consume the telemetry the platform provides.
• Build and run the evaluation suites (offline before launch and continuous evaluation on sampled production traffic after) and drive proactive detection of drift and quality decay.
• Monitor live agents, using the dashboards and automated safeguards, and act on alerts and proactive signals.
• Perform first-level triage and in-scope fixes (restart, safe-listed configuration, cache reset, routing fallback) per the runbooks.
• Manage incidents — classify severity, route by fault domain, elevate with a full evidence bundle, and verify-and-close in production — while owning the incident throughout.
• Contribute to the handover gate — verifying operability and evaluation evidence before an agent is accepted into full operation.

Skills

LLM systems

Observability

Monitoring

Python

Dashboarding

Data analysis

SRE

Incident management

Communication

Technical writing

Quality assurance

Stakeholder collaboration

Education

Degree in Computer Science, Engineering, Data Science, or a related field

Job description

Singtel is seeking a Senior Agent Operations & Evaluation Engineer to lead live GenAI agents and build robust evaluation/observability to ensure accuracy, safety, and continuous improvement. You will oversee agent health, triage incidents, and minimize downtime while collaborating with Day 1 build teams and business owners.

The role emphasizes measuring technical and outcome metrics, implementing proactive drift detection, and delivering insights through dashboards.