AI Evaluation Engineer: LLM Benchmarking & Safety

SHOPEE SINGAPORE PRIVATE LIMITED; Singapore; On-site; SGD 120,000 - 180,000; Full time

Job summary

Shopee Singapore Private Limited is seeking an AI evaluation engineer to design robust evaluation frameworks and drive safety testing for AI models. You will collaborate with AI and engineering teams to diagnose failures, optimize prompts, and build automated tooling that accelerates product iteration.

Strong Python, Linux, and shell scripting skills are required. The role emphasizes QA automation, data pipelines, and rigorous analysis of model behavior in a fast-growing e-commerce platform

Qualifications

• Strong software engineering fundamentals.
• Experience with evaluating AI models or NLP systems is a plus.

Responsibilities

• Design and build evaluation frameworks for multi-dimensional AI assessment.
• Perform red-team testing and safety controls on AI models to identify bias, hallucinations, and policy violations.
• Conduct root-cause analysis of incorrect model outputs and collaborate with engineers to optimize prompts and fine-tune models.
• Develop automated evaluation tooling to streamline product iterations via LLM-as-a-Judge methodologies.
• Research user psychology in social/companion-style products and translate insights into measurable metrics.

Skills

Python

Shell scripting

Linux

Education

Master’s degree in Computer Science

Tools

Linux

Job description

Shopee Singapore Private Limited is seeking an AI evaluation engineer to design robust evaluation frameworks and drive safety testing for AI models. You will collaborate with AI and engineering teams to diagnose failures, optimize prompts, and build automated tooling that accelerates product iteration.

Strong Python, Linux, and shell scripting skills are required. The role emphasizes QA automation, data pipelines, and rigorous analysis of model behavior in a fast-growing e-commerce platform