Senior AI Evaluation Architect

Singtel Group; Singapore; On-site; SGD 120,000 - 180,000; Full time

Job summary

Singtel is seeking a Senior Technical Evaluator to design, calibrate, and adjudicate evaluation suites across agent archetypes, and lead gate reviews while owning regression-pack methodology.

In this role you will design offline evaluation suites, conduct operability gate reviews, calibrate production-aligned thresholds, and maintain drift hygiene. You will mentor teammates and ensure high-quality results that inform AI/ML deployments.

Qualifications

• Bachelor's or Master's degree in Computer Science or a related field.
• 6+ years in ML/data/software with a strong evaluation or quality focus.
• Hands-on experience evaluating LLM or ML systems.
• LLM/agent evaluation design and statistical rigour.
• Proficient in Python and evaluation tooling (promptfoo, DeepEval, or custom harnesses).
• Data analysis and metric interpretation.
• Observability and tracing tooling.
• Adversarial testing/red-teaming experience.
• Clear technical writing for gate findings.
• Ability to influence build teams on quality.

Responsibilities

• Design and maintain offline evaluation suites and the continuous-evaluation scoring pipeline across archetypes.
• Conduct operability gate reviews: assess evidence packs, reproduce evaluation results, and document findings.
• Own the model-update regression pack methodology and adjudicate regression runs with AIML Operations.
• Calibrate gate thresholds against production reality and maintain evaluation drift hygiene.
• Produce the monthly quality report per agent: eval trends, failure-mode taxonomy, and defect clusters with reproduction traces.
• Mentor members of the team and review their work for quality and consistency.

Skills

ML evaluation

Quality assurance

LLM evaluation

Python

Evaluation tooling

Data analysis

Observability

Adversarial testing

Technical writing

Stakeholder influence

Education

Bachelor’s or Master’s degree in Computer Science or related field

Tools

promptfoo

DeepEval

custom harnesses

Job description

Singtel is seeking a Senior Technical Evaluator to design, calibrate, and adjudicate evaluation suites across agent archetypes, and lead gate reviews while owning regression-pack methodology.

In this role you will design offline evaluation suites, conduct operability gate reviews, calibrate production-aligned thresholds, and maintain drift hygiene. You will mentor teammates and ensure high-quality results that inform AI/ML deployments.