LLM Quality & Evaluation Engineer – Production Safety

Argyll Scott; Hong Kong; Sur place; HKD 500 000 - 900 000; Plein temps

Résumé du poste

Argyll Scott in Hong Kong is partnering with a global life insurer to establish a regional AI Centre of Excellence. The AI Quality & Evals Engineer will help build evaluation infrastructure, monitor live AI activity and intervene when needed, spanning governance, safety and cost controls.

You will work hands-on on evaluation frameworks, streaming monitors, and incident reporting, collaborating with Risk, Security and Legal teams, contributing to robust, auditable AI systems across Asia.

Qualifications

• 4+ years in software, ML or data engineering.
• 2+ years focused on production LLM or agent quality, evaluation or safety.

Responsabilités

• Build evaluation frameworks using reference datasets, production failure cases, synthetic tests and versioned evaluation criteria.
• Design and calibrate LLM-based judges against human assessments, monitoring their consistency as models evolve.
• Test agent-specific failure modes, including fabricated tool arguments, retrieval failures, uncontrolled loops and unsafe actions.
• Develop adversarial and red-team suites covering prompt injection, data-exfiltration attempts and privilege escalation.
• Integrate regression evaluations into release pipelines for changes to prompts, models, tools, retrieval and policies.
• Build streaming evaluations and risk-based monitors for quality, task success, latency, cost and behavioural drift.
• Implement guardrails, intervention thresholds, kill switches and safe interruption mechanisms.
• Automate evaluation results, incidents, asset inventories and control evidence into governance platforms, partnering with Risk, Security, Legal and Internal Audit.
• Share operational responsibility for the agent control platform, including on-call response and decisions to constrain or halt agent activity.

Description du poste

Argyll Scott in Hong Kong is partnering with a global life insurer to establish a regional AI Centre of Excellence. The AI Quality & Evals Engineer will help build evaluation infrastructure, monitor live AI activity and intervene when needed, spanning governance, safety and cost controls.

You will work hands-on on evaluation frameworks, streaming monitors, and incident reporting, collaborating with Risk, Security and Legal teams, contributing to robust, auditable AI systems across Asia.