Global Life Insurer - Asia AI Quality Evaluation Engineer

Argyll Scott; Hong Kong; In loco; HKD 500.000 - 900.000; Tempo pieno

Descrizione del lavoro

Argyll Scott in Hong Kong is partnering with a global life insurer to establish a regional AI Centre of Excellence. The AI Quality & Evals Engineer will help build evaluation infrastructure, monitor live AI activity and intervene when needed, spanning governance, safety and cost controls.

You will work hands-on on evaluation frameworks, streaming monitors, and incident reporting, collaborating with Risk, Security and Legal teams, contributing to robust, auditable AI systems across Asia.

Competenze

• 4+ years in software, ML or data engineering.
• 2+ years focused on production LLM or agent quality, evaluation or safety.

Mansioni

• Build evaluation frameworks using reference datasets, production failure cases, synthetic tests and versioned evaluation criteria.
• Design and calibrate LLM-based judges against human assessments, monitoring their consistency as models evolve.
• Test agent-specific failure modes, including fabricated tool arguments, retrieval failures, uncontrolled loops and unsafe actions.
• Develop adversarial and red-team suites covering prompt injection, data-exfiltration attempts and privilege escalation.
• Integrate regression evaluations into release pipelines for changes to prompts, models, tools, retrieval and policies.
• Build streaming evaluations and risk-based monitors for quality, task success, latency, cost and behavioural drift.
• Implement guardrails, intervention thresholds, kill switches and safe interruption mechanisms.
• Automate evaluation results, incidents, asset inventories and control evidence into governance platforms, partnering with Risk, Security, Legal and Internal Audit.
• Share operational responsibility for the agent control platform, including on-call response and decisions to constrain or halt agent activity.

Descrizione del lavoro

AI Quality & Evals Engineer

Regional AI Centre of Excellence | Global Life Insurer

Location: Hong Kong; candidates based in the Philippines or India may also be considered

Experience: 4+ years in software, ML or data engineering, including 2+ years focused on production LLM or agent quality, evaluation or safety

The opportunity

We are partnered with a global life insurer to build out its regional AI Centre of Excellence, supporting the adoption of generative AI and agentic systems across its Asian businesses. As these capabilities take on more complex workflows, engineering robust evaluation and operational controls will be essential to their success.

As an AI Quality & Evals Engineer, you will build the systems that assess AI behaviour, monitor live activity and intervene when necessary. This is a hands-on engineering role spanning evaluation infrastructure, streaming monitors, runtime controls and automated governance evidence.

What you will do

• Build evaluation frameworks using reference datasets, production failure cases, synthetic tests and versioned evaluation criteria.
• Design and calibrate LLM-based judges against human assessments, monitoring their consistency as models evolve.
• Test agent‑specific failure modes, including fabricated tool arguments, retrieval failures, uncontrolled loops and unsafe actions.
• Develop adversarial and red‑team suites covering prompt injection, data‑exfiltration attempts and privilege escalation.
• Integrate regression evaluations into release pipelines for changes to prompts, models, tools, retrieval and policies.
• Build streaming evaluations and risk‑based monitors for quality, task success, latency, cost and behavioural drift.
• Implement guardrails, intervention thresholds, kill switches and safe interruption mechanisms.
• Automate evaluation results, incidents, asset inventories and control evidence into governance platforms, partnering with Risk, Security, Legal and Internal Audit.
• Share operational responsibility for the agent control platform, including on‑call response and decisions to constrain or halt agent activity.

What you will bring

• Strong Python and engineering discipline across testing, typing, versioning and CI/CD.
• Proven experience building online evaluation over live production traffic.

Preferred: GCP/Vertex AI; event‑driven architectures; MCP server development; AI performance and cost dashboards; and AI governance knowledge.

Why this role

You will help establish the production foundations of a regional AI capability, combining deep engineering ownership with exposure to modern ML, LLM and agent technologies in a complex enterprise environment.