AI Agent Evaluation Scientist

Manus AI; Singapore; On-site; SGD 90,000 - 150,000; Full time

Job summary

Manus AI specializes in evaluating AI agents and models, grounding evaluations in user tasks and how systems operate.

This role designs online and offline metrics, develops experiments around planning, tool use and feedback, and helps compare model quality after changes.

You will collaborate with product, engineering and research teams to ensure robust post-training evaluations and trustworthy results.

Qualifications

• Practical experience evaluating agent products or model post-training with metric design and validation evidence.
• Deep understanding of AI model and agent mechanisms including task planning, tool use, context management and feedback.

Responsibilities

• Design online and offline metrics translating user tasks and output quality into measurable evaluation criteria.
• Design evaluation tasks and experiments around agent planning, tool use, context and feedback, comparing performance before and after system changes.
• Support post-training evaluations and comparisons of third-party model quality, defining use cases and metric definitions.
• Develop new tasks, metrics or experimental methods for capabilities and user experience issues that existing evaluations miss and test validity, bias and reproducibility.
• Analyze evaluation results and failures, distinguish score changes from real capability changes and validate improvements with product, engineering and model teams.

Skills

Metric design

Experiment design

Post-training evaluation

AI model understanding

Data analysis

Uncertainty analysis

Results interpretation

User research methods

Job description

Manus AI specializes in evaluating AI agents and models, grounding evaluations in user tasks and how systems operate.

This role designs online and offline metrics, develops experiments around planning, tool use and feedback, and helps compare model quality after changes.

You will collaborate with product, engineering and research teams to ensure robust post-training evaluations and trustworthy results.