Agent Evaluation Engineer

Manus AI; Singapore; On-site; SGD 90,000 - 150,000; Full time

Benefits offered by this job

Equity plan

Unlimited Manus Tokens

Frontier AI projects

Job summary

Manus AI in Singapore seeks an experienced evaluator to design metrics and experiments for evaluating AI agents and models. You will translate user tasks into measurable criteria, design tasks around planning and tool use, and support post-training comparisons to ensure robust evaluation results.

You will work with product, engineering and model teams to analyze results, identify gaps, and develop new methods that address capability and user-value issues, ensuring rigorous validation.

Qualifications

• Practical experience designing metrics for AI evaluations.
• Evidence of designing and validating evaluation metrics.
• Experience analyzing AI model behavior and system performance.

Responsibilities

• Design online and offline metrics translating user tasks into measurable criteria.
• Create evaluation tasks for agent planning, tool use, context and feedback.
• Support post-training evaluations comparing model quality and use cases.
• Develop new evaluation methods and test bias and reproducibility.
• Analyze results, distinguish score changes from true capability changes.

Skills

Metric design

Experiment design

Post-training evaluation

Bias analysis

Uncertainty analysis

Communication

Job description

Develop evaluations grounded in product needs, user tasks and how AI models and agents work. Use metrics, experiments and failure analysis to assess capability changes and investigate gaps in existing evaluations, informing system development, post-training and model selection.

Responsibilities

• Design online and offline metrics that translate user tasks, output quality and practical value into measurable, testable evaluation criteria.
• Design evaluation tasks and experiments around agent planning, tool use, context and feedback, comparing performance before and after system changes and identifying what affects results.
• Support post-training evaluations and comparisons of third-party model quality, defining use cases, metric definitions and the conditions to which results apply.
• Develop new tasks, metrics or experimental methods for capabilities and user experience issues that existing evaluations miss, and test their validity, bias and reproducibility.
• Analyze evaluation results and failures, distinguish score changes from real capability changes, and validate improvements and refine methods with product, engineering and model teams.

Requirements

• Practical experience evaluating agent products or model post-training, with concrete evidence of metric design and validation skills.
• Deep understanding of AI model and agent mechanisms, including task planning, tool use, context management and feedback, and the ability to use this knowledge to analyze system behavior.
• Data analysis, engineering and research skills, including the ability to design experiments, process evaluation data and analyze uncertainty in results.
• Ability to investigate open-ended problems, develop well-reasoned new evaluation methods, examine experimental bias and how samples support conclusions, and test judgments through experiments.
• Independent judgment on AI output quality and real user value, with the ability to explain findings, their applicability and limitations clearly.
• Familiarity with user research methods such as interviews, observation or usability testing, and the ability to translate findings into evaluation tasks and criteria, are preferred.

Manus excels at various tasks in work and life, getting everything done while you rest at Manus AI.

What we offer:

• Build at the Frontier of AI Agents - Work with a fast-paced team to turn frontier AI breakthroughs into real-world impact.
• Equity & Shared success - Our share incentive plan enables employees to participate in Manus’s long-term growth and share in the value we create together.
• Unlimited Manus Tokens - Enjoy unlimited Manus tokens to experiment, build, and supercharge your productivity.

If you're passionate about cutting-edge technology and making a real impact, we’d love to hear from you!