Senior AI Evaluation Engineer - Gatekeeper of Model Quality

Singapore Telecommunications Limited; Singapore; On-site; SGD 120,000 - 180,000; Full time

Job summary

Singtel, a leader in AI & Data Analytics, seeks a senior technical evaluator to design, calibrate, and adjudicate evaluation suites for agent archetypes. You will lead gate reviews under the Team Lead and own the regression-pack methodology, ensuring quality and drift hygiene for AI/ML changes.

You will mentor teammates, produce monthly quality reports, and influence build teams on evaluation rigor. 6+ years in ML/data/software with hands-on AI evaluation is required.

Qualifications

• Bachelor's or Master's in CS or related field.
• Experience with ML/AI evaluation, quality assurance, or models.
• Strong ability to design and run evaluation suites for AI systems.
• Proficiency in Python and evaluation tooling.
• Experience with LLM or agent evaluation.

Responsibilities

• Design and maintain offline evaluation suites and scoring pipelines.
• Conduct operability gate reviews with documented findings.
• Own regression pack methodology and adjudicate against baselines.
• Calibrate gate thresholds to production reality and manage drift.
• Produce monthly quality reports with trends and defect clusters.
• Mentor team members and ensure work quality.

Skills

LLM Evaluation Design

Statistical rigour

Python & tooling

Data analysis & metrics

Tracing/observability

Adversarial testing

Gate-review leadership

Technical writing

Education

Bachelor's or Master's in Computer Science or related field

Tools

promptfoo

DeepEval

Custom evaluation harnesses

Reproducibility tooling

Job description

Singtel, a leader in AI & Data Analytics, seeks a senior technical evaluator to design, calibrate, and adjudicate evaluation suites for agent archetypes. You will lead gate reviews under the Team Lead and own the regression-pack methodology, ensuring quality and drift hygiene for AI/ML changes.

You will mentor teammates, produce monthly quality reports, and influence build teams on evaluation rigor. 6+ years in ML/data/software with hands-on AI evaluation is required.