LLM Agent Reliability & Evaluation Engineer

SHOPEE IP SINGAPORE PRIVATE LIMITED; Singapore; On-site; SGD 120,000 - 180,000; Full time

Job summary

SHOPEE IP SINGAPORE PRIVATE LIMITED is seeking a specialist to design and own the evaluation framework for LLM Agents, including benchmarks and human-annotated judgments, with continuous monitoring to guide optimization.

You will lead reliability analyses over multi-step agent trajectories, drive evaluation-driven optimization (Harness design, tool calling, context engineering, memory), and oversee post-training in SFT/DPO/RL with data pipelines and reward models, plus RAG systems and tool-layer

Qualifications

• Master’s degree or above in AI/CS/Math or related field.
• ≥1 year of full-time industry experience building and deploying production LLM Agent systems (planning, orchestration, tool calling) with evaluation metrics to drive iteration.
• Hands-on experience fine-tuning LLMs via SFT and DPO; building and optimizing RAG/retrieval systems (recall, ranking, embedding fine-tuning).
• Strong Python programming; SQL and PyTorch are a plus; familiar with training/inference frameworks such as verl, vLLM, FSDP, or LLaMA-Factory.
• Strong analytical skills and statistics, with ability to design experiments and collaborate across teams to close the loop from research to deployment.

Responsibilities

• Design and own the evaluation system for LLM Agents, including benchmark construction, LLM-as-Judge with human annotation, and regression/comparison pipelines.
• Build Agent reliability analysis, including error attribution and uncertainty estimation across multi-step trajectories, and turn failure cases into targeted improvements.
• Drive evaluation-guided Agent optimization, including Harness design (planning, verification, self-refine), tool calling, context engineering, and memory.
• Lead evaluation-driven post-training of LLMs for vertical domains (SFT, DPO, RL), including data pipelines and reward model design.
• Build and optimize RAG/retrieval systems and the Agent tool layer (search, code interpreters, sandboxes, MCP) and evaluate their impact on end-to-end Agent quality.
• Explore frontier research in Agent reliability and evaluation and drive a full closed loop from research to deployment.

Skills

Python

SQL

PyTorch

LLM fine-tuning

Education

Master’s degree or above in AI/CS/Math

Tools

verl

vLLM

FSDP

LLaMA-Factory

Job description

SHOPEE IP SINGAPORE PRIVATE LIMITED is seeking a specialist to design and own the evaluation framework for LLM Agents, including benchmarks and human-annotated judgments, with continuous monitoring to guide optimization.

You will lead reliability analyses over multi-step agent trajectories, drive evaluation-driven optimization (Harness design, tool calling, context engineering, memory), and oversee post-training in SFT/DPO/RL with data pipelines and reward models, plus RAG systems and tool-layer