Agent Algorithm Evaluation Engineer

SHOPEE SINGAPORE PRIVATE LIMITED; Singapore; On-site; SGD 120,000 - 180,000; Full time

Job summary

Shopee Singapore Private Limited is seeking an AI evaluation engineer to design robust evaluation frameworks and drive safety testing for AI models. You will collaborate with AI and engineering teams to diagnose failures, optimize prompts, and build automated tooling that accelerates product iteration.

Strong Python, Linux, and shell scripting skills are required. The role emphasizes QA automation, data pipelines, and rigorous analysis of model behavior in a fast-growing e-commerce platform

Qualifications

• Strong software engineering fundamentals.
• Experience with evaluating AI models or NLP systems is a plus.

Responsibilities

• Design and build evaluation frameworks for multi-dimensional AI assessment.
• Perform red-team testing and safety controls on AI models to identify bias, hallucinations, and policy violations.
• Conduct root-cause analysis of incorrect model outputs and collaborate with engineers to optimize prompts and fine-tune models.
• Develop automated evaluation tooling to streamline product iterations via LLM-as-a-Judge methodologies.
• Research user psychology in social/companion-style products and translate insights into measurable metrics.

Skills

Python

Shell scripting

Linux

Education

Master’s degree in Computer Science

Tools

Linux

Job description

The Engineering and Technology team is at the core of the Shopee platform development. The team is made up of a group of passionate engineers from all over the world, striving to build the best systems with the most suitable technologies. Our engineers do not merely solve problems at hand; We build foundations for a long-lasting future. We don't limit ourselves on what we can or can't do; we take matters into our own hands even if it means drilling down to the bottom layer of the computing platform. Shopee's hyper-growing business scale has transformed most "innocent" problems into huge technical challenges, and there is no better place to experience it first-hand if you love technologies as much as we do.

About the Team:

About Us

Sea Group is establishing a brand-new, strategic AI department. This department is dedicated to exploring the transformative potential of generative AI in revolutionizing human connection, self-expression and communication diversity, and social interaction. We are building the next generation of AI-native applications and a comprehensive Model-as-a-Service (MaaS) product support system. Based on massive multi-country data, we are building a leading multilingual AI ecosystem from the ground up. We look forward to more outstanding talents joining us to build leading Southeast Asian multilingual models and explore innovative AI-native applications.

The AI application team focuses on the intersection of social connectivity and artificial intelligence. Our mission is to leverage LLMs to create digital personas that can act as personal assistants and social bridges. This team operates with a startup's agility backed by our Group's robust resources, aiming to define how humans interact in the AI era.

Job Description:

• Evaluation Framework Design: Design and build multi-dimensional evaluation metric systems tailored to diverse business scenarios.
• Red-Team Testing & Safety Control: Simulate complex, extreme real-world user scenarios to conduct stress testing and red-teaming of AI models; identify and drive remediation of issues related to bias, hallucinations, policy-violating content, and values alignment.
• In-Depth Bad Case Analysis: Conduct root-cause analysis of erroneous model outputs and collaborate with algorithm engineers (Model/SFT/RLHF) to drive prompt optimization and fine-tuning improvements.

Automated Evaluation Tooling: Leverage LLM-as-a-Judge methodologies to build automated evaluation pipelines that enhance product iteration efficiency.

• User Experience Insights: Conduct in-depth research into user psychology within social and companion-style products, translating subjective, qualitative experience into quantifiable metrics to continuously enhance product experience.

Requirements:

• Proven experience enhancing automated evaluation workflows, including but not limited to conventional evaluation methods, LLM-as-a-Judge, and automated weakness mining.
• Proven experience enhancing automated data production pipelines, including but not limited to Self-Instruct, Evol-Instruct, and Synthetic Data RL.
• Strong software engineering fundamentals, with proficiency in Python and Shell scripting and hands-on experience in Linux environments.
• Master's degree or above in Computer Science or a related field, with at least 3 years' working experience.

Nice-to-Have

• Prior background in AI evaluation, QA automation, or NLP algorithm development at a major internet company.
• Extensive experience as a power user or developer of mainstream AI social products.