AI Agent Evaluation Architect

Manus AI; Singapore; On-site; SGD 90,000 - 150,000; Full time

Benefits offered by this job

Equity plan

Unlimited Manus Tokens

Frontier AI projects

Job summary

Manus AI in Singapore seeks an experienced evaluator to design metrics and experiments for evaluating AI agents and models. You will translate user tasks into measurable criteria, design tasks around planning and tool use, and support post-training comparisons to ensure robust evaluation results.

You will work with product, engineering and model teams to analyze results, identify gaps, and develop new methods that address capability and user-value issues, ensuring rigorous validation.

Qualifications

• Practical experience designing metrics for AI evaluations.
• Evidence of designing and validating evaluation metrics.
• Experience analyzing AI model behavior and system performance.

Responsibilities

• Design online and offline metrics translating user tasks into measurable criteria.
• Create evaluation tasks for agent planning, tool use, context and feedback.
• Support post-training evaluations comparing model quality and use cases.
• Develop new evaluation methods and test bias and reproducibility.
• Analyze results, distinguish score changes from true capability changes.

Skills

Metric design

Experiment design

Post-training evaluation

Bias analysis

Uncertainty analysis

Communication

Job description

Manus AI in Singapore seeks an experienced evaluator to design metrics and experiments for evaluating AI agents and models. You will translate user tasks into measurable criteria, design tasks around planning and tool use, and support post-training comparisons to ensure robust evaluation results.

You will work with product, engineering and model teams to analyze results, identify gaps, and develop new methods that address capability and user-value issues, ensuring rigorous validation.