Agent Evaluation Engineer

Manus AI; Singapore; On-site; SGD 90,000 - 150,000; Full time

Job summary

Manus AI specializes in evaluating AI agents and models, grounding evaluations in user tasks and how systems operate.

This role designs online and offline metrics, develops experiments around planning, tool use and feedback, and helps compare model quality after changes.

You will collaborate with product, engineering and research teams to ensure robust post-training evaluations and trustworthy results.

Qualifications

• Practical experience evaluating agent products or model post-training with metric design and validation evidence.
• Deep understanding of AI model and agent mechanisms including task planning, tool use, context management and feedback.

Responsibilities

• Design online and offline metrics translating user tasks and output quality into measurable evaluation criteria.
• Design evaluation tasks and experiments around agent planning, tool use, context and feedback, comparing performance before and after system changes.
• Support post-training evaluations and comparisons of third-party model quality, defining use cases and metric definitions.
• Develop new tasks, metrics or experimental methods for capabilities and user experience issues that existing evaluations miss and test validity, bias and reproducibility.
• Analyze evaluation results and failures, distinguish score changes from real capability changes and validate improvements with product, engineering and model teams.

Skills

Metric design

Experiment design

Post-training evaluation

AI model understanding

Data analysis

Uncertainty analysis

Results interpretation

User research methods

Job description

Develop evaluations grounded in product needs, user tasks and how AI models and agents work. Use metrics, experiments and failure analysis to assess capability changes and investigate gaps in existing evaluations, informing system development, post-training and model selection.

Responsibilities

• Design online and offline metrics that translate user tasks, output quality and practical value into measurable, testable evaluation criteria.
• Design evaluation tasks and experiments around agent planning, tool use, context and feedback, comparing performance before and after system changes and identifying what affects results.
• Support post-training evaluations and comparisons of third-party model quality, defining use cases, metric definitions and the conditions to which results apply.
• Develop new tasks, metrics or experimental methods for capabilities and user experience issues that existing evaluations miss, and test their validity, bias and reproducibility.
• Analyze evaluation results and failures, distinguish score changes from real capability changes, and validate improvements and refine methods with product, engineering and model teams.

Requirements

• Practical experience evaluating agent products or model post-training, with concrete evidence of metric design and validation skills.
• Deep understanding of AI model and agent mechanisms, including task planning, tool use, context management and feedback, and the ability to use this knowledge to analyze system behavior.
• Data analysis, engineering and research skills, including the ability to design experiments, process evaluation data and analyze uncertainty in results.
• Ability to investigate open-ended problems, develop well-reasoned new evaluation methods, examine experimental bias and how samples support conclusions, and test judgments through experiments.
• Independent judgment on AI output quality and real user value, with the ability to explain findings, their applicability and limitations clearly.
• Familiarity with user research methods such as interviews, observation or usability testing, and the ability to translate findings into evaluation tasks and criteria, are preferred.