LLM Quality & Evaluation Engineer — Regional AI Center

Argyll Scott; Hong Kong; On-site; HKD 900,000 - 1,300,000; Full time

Job summary

Regional AI Centre of Excellence in Hong Kong is seeking an AI Quality & Evals Engineer to build evaluation systems for generative AI and agentic workflows across regional businesses. You will develop evaluation infrastructure, streaming monitors and governance evidence while collaborating with Risk, Security and Legal teams.

Ideal candidates bring 4+ years in software/ML/data engineering, with 2+ years in production LLM or agent evaluation, and strong Python/CI/CD discipline.

Qualifications

• 4+ years in software, ML or data engineering, including 2+ years focused on production LLM or agent quality, evaluation or safety.
• Strong Python and engineering discipline across testing, typing, versioning and CI/CD.

Responsibilities

• Build evaluation frameworks using reference datasets, production failure cases, synthetic tests and versioned evaluation criteria.
• Design and calibrate LLM-based judges against human assessments, monitoring their consistency as models evolve.
• Test agent-specific failure modes, including fabricated tool arguments, retrieval failures, uncontrolled loops and unsafe actions.
• Develop adversarial and red-team suites covering prompt injection, data-exfiltration attempts and privilege escalation.
• Integrate regression evaluations into release pipelines for changes to prompts, models, tools, retrieval and policies.
• Build streaming evaluations and risk-based monitors for quality, task success, latency, cost and behavioural drift.
• Implement guardrails, intervention thresholds, kill switches and safe interruption mechanisms.
• Automate evaluation results, incidents, asset inventories and control evidence into governance platforms, partnering with Risk, Security, Legal and Internal Audit.
• Share operational responsibility for the agent control platform, including on-call response and decisions to constrain or halt agent activity.

Skills

Python

CI/CD

Online evaluation

Testing

Tools

Vertex AI

Job description

Regional AI Centre of Excellence in Hong Kong is seeking an AI Quality & Evals Engineer to build evaluation systems for generative AI and agentic workflows across regional businesses. You will develop evaluation infrastructure, streaming monitors and governance evidence while collaborating with Risk, Security and Legal teams.

Ideal candidates bring 4+ years in software/ML/data engineering, with 2+ years in production LLM or agent evaluation, and strong Python/CI/CD discipline.