LLM Evaluation Engineer: AI Safety & Metrics

Shopee; Singapore; On-site; SGD 120,000 - 180,000; Full time

Job summary

Shopee's Engineering and Technology team in Singapore seeks an experienced individual contributor to design and implement robust evaluation frameworks for AI models. You will conduct red-team tests, root-cause analyses of model outputs, and build automated tooling that accelerates product iterations.

Role requires strong software engineering fundamentals, Python, Linux, and exposure to LLM-based evaluation approaches, with a Master's degree and at least 3 years in related AI/NLP work.

Qualifications

• Requires strong Python programming and shell scripting

Responsibilities

• Design and implement automated evaluation metrics for AI models
• Perform red-team testing and safety controls for AI systems
• Investigate model errors with engineers to guide prompt optimization
• Build automated evaluation pipelines using LLM-as-a-Judge
• Conduct user experience research to quantify social interactions with AI tools

Skills

Python

Shell scripting

Linux

LLM evaluation

Software engineering fundamentals

Education

Master's degree in Computer Science

Tools

Self-Instruct

Evol-Instruct

Synthetic Data RL

Job description

Shopee's Engineering and Technology team in Singapore seeks an experienced individual contributor to design and implement robust evaluation frameworks for AI models. You will conduct red-team tests, root-cause analyses of model outputs, and build automated tooling that accelerates product iterations.

Role requires strong software engineering fundamentals, Python, Linux, and exposure to LLM-based evaluation approaches, with a Master's degree and at least 3 years in related AI/NLP work.