Agent Algorithm Evaluation Engineer

Shopee; Singapore; On-site; SGD 120,000 - 180,000; Full time

Job summary

Shopee's Engineering and Technology team in Singapore seeks an experienced individual contributor to design and implement robust evaluation frameworks for AI models. You will conduct red-team tests, root-cause analyses of model outputs, and build automated tooling that accelerates product iterations.

Role requires strong software engineering fundamentals, Python, Linux, and exposure to LLM-based evaluation approaches, with a Master's degree and at least 3 years in related AI/NLP work.

Qualifications

• Requires strong Python programming and shell scripting

Responsibilities

• Design and implement automated evaluation metrics for AI models
• Perform red-team testing and safety controls for AI systems
• Investigate model errors with engineers to guide prompt optimization
• Build automated evaluation pipelines using LLM-as-a-Judge
• Conduct user experience research to quantify social interactions with AI tools

Skills

Python

Shell scripting

Linux

LLM evaluation

Software engineering fundamentals

Education

Master's degree in Computer Science

Tools

Self-Instruct

Evol-Instruct

Synthetic Data RL

Job description

Department Engineering and TechnologyLevelExperienced (Individual Contributor)LocationSingaporeThe Engineering and Technology team is at the core of the Shopee platform development. The team is made up of a group of passionate engineers from all over the world, striving to build the best systems with the most suitable technologies. Our engineers do not merely solve problems at hand; We build foundations for a long-lasting future. We don't limit ourselves on what we can or can't do; we take matters into our own hands even if it means drilling down to the bottom layer of the computing platform. Shopee's hyper-growing business scale has transformed most \"innocent\" problems into huge technical challenges, and there is no better place to experience it first-hand if you love technologies as much as we do.About the Team:Sea Group is establishing a brand-new, strategic AI department. This department is dedicated to exploring the transformative potential of generative AI in revolutionizing human connection, self-expression and communication diversity, and social interaction. We are building the next generation of AI-native applications and a comprehensive Model-as-a-Service (MaaS) product support system. Based on massive multi-country data, we are building a leading multilingual AI ecosystem from the ground up. We look forward to more outstanding talents joining us to build leading Southeast Asian multilingual models and explore innovative AI-native applications.The AI application team focuses on the intersection of social connectivity and artificial intelligence. Our mission is to leverage LLMs to create digital personas that can act as personal assistants and social bridges. This team operates with a startup's agility backed by our Group's robust resources, aiming to define how humans interact in the AI era.Job Description:Evaluation Framework Design: Design and build multi-dimensional evaluation metric systems tailored to diverse business scenarios.Red-Team Testing & Safety Control: Simulate complex, extreme real-world user scenarios to conduct stress testing and red-teaming of AI models; identify and drive remediation of issues related to bias, hallucinations, policy-violating content, and values alignment.In-Depth Bad Case Analysis: Conduct root-cause analysis of erroneous model outputs and collaborate with algorithm engineers (Model/SFT/RLHF) to drive prompt optimization and fine-tuning improvements.Automated Evaluation Tooling: Leverage LLM-as-a-Judge methodologies to build automated evaluation pipelines that enhance product iteration efficiency.User Experience Insights: Conduct in-depth research into user psychology within social and companion-style products, translating subjective, qualitative experience into quantifiable metrics to continuously enhance product experience.Requirements:Proven experience enhancing automated evaluation workflows, including but not limited to conventional evaluation methods, LLM-as-a-Judge, and automated weakness mining.Proven experience enhancing automated data production pipelines, including but not limited to Self-Instruct, Evol-Instruct, and Synthetic Data RL.Strong software engineering fundamentals, with proficiency in Python and Shell scripting and hands-on experience in Linux environments.Master's degree or above in Computer Science or a related field, with at least 3 years' working experience.Nice-to-HavePrior background in AI evaluation, QA automation, or NLP algorithm development at a major internet company.Extensive experience as a power user or developer of mainstream AI social products.