LLM Agent Reliability & Evaluation Architect

Shopee; Singapore; On-site; SGD 90,000 - 130,000; Full time

Job summary

Shopee is seeking an expert to design and own the evaluation system for LLM Agents, including benchmark construction and LLM-as-Judge with human annotation. You will build reliability analyses across multi-step trajectories and lead evaluation-driven optimization of agents and tools.

The role involves leading post-training efforts for LLMs, building RAG and tool-layers, and translating research into deployed solutions with a focus on frontier AI and end-to-end quality.

Qualifications

• Master’s degree or above in Artificial Intelligence, Computer Science, Mathematics, or related field.
• At least 1 year of full-time industry experience building and deploying production LLM Agent systems with evaluation design.
• Experience fine-tuning LLMs via SFT and DPO, and building RAG/retrieval systems.
• Proficient in Python; familiarity with PyTorch.
• Strong analytical skills and experimental design; collaborative mindset.

Responsibilities

• Design and own the evaluation system for LLM Agents and benchmarks.
• Build Agent reliability analysis with error attribution across multi-step trajectories.
• Drive evaluation-guided Agent optimization including Harness design, tool calling, and memory.
• Lead evaluation-driven post-training of LLMs for vertical domains (SFT, DPO, RL).
• Build and optimize RAG/retrieval systems and the Agent tool layer; assess impact.
• Explore frontier research and drive a full loop from research to deployment.

Skills

LLM Agent design

Evaluation metrics

Python programming

Statistics

Experiment design

Team collaboration

Education

Master’s degree in AI/CS/Math

Tools

verl

vLLM

FSDP

LLaMA-Factory

Job description

Shopee is seeking an expert to design and own the evaluation system for LLM Agents, including benchmark construction and LLM-as-Judge with human annotation. You will build reliability analyses across multi-step trajectories and lead evaluation-driven optimization of agents and tools.

The role involves leading post-training efforts for LLMs, building RAG and tool-layers, and translating research into deployed solutions with a focus on frontier AI and end-to-end quality.