AI Product Evaluator

Confidential; United States; Remote; USD 34,000 - 69,000; Full time

Job summary

Zekai is seeking an AI Product Evaluator for a 100% remote contractor role across global teams. You will apply expertise to train next-generation AI systems by evaluating models and agents against predefined scenarios.

You will document findings, reset usage limits, and contribute to datasets on training platforms, ensuring high accuracy, repeatability, and strict adherence to project milestones.

Qualifications

• Experience evaluating AI products or AI models.
• Ability to follow complex instructions with precision.
• Strong documentation and reporting skills.
• Daily use of AI tools is noted as a plus.

Responsibilities

• Evaluate AI models and agents per predefined scenarios.
• Perform repeated evaluations within project timeframes.
• Document observations and anomalies in structured reports.
• Reset usage limits and resume evaluations per protocol.
• Submit evaluation datasets to training platforms per milestones.
• Maintain high attention to detail across tasks.

Skills

AI product evaluation

QA testing

Documentation

Instruction following

Attention to detail

Job description

Role Title: AI Product Evaluator

Role Type: Contractor

Location: India, Japan or United Kingdom, Remote

our partner is engaging expert AI Product Evaluators to contribute directly to a customer's high-impact AI project. In this role, you'll apply your expertise to help train next-generation AI systems. Your work will shape how models learn, reason, and perform through high-quality, real-world input. This operationally intensive engagement involves handling AI model and agent evaluations, often within strict timeframes and usage limit resets. The project benefits greatly from contributions of multiple specialists, each bringing a meticulous approach to instruction-following and quality assurance.

Scope of Work

• Evaluate AI models and agents according to detailed, pre-defined scenarios and guidelines.
• Execute a series of operational tasks designed to test model accuracy, reasoning, and responsiveness.
• Document findings, observations, and any anomalies in structured written reports.
• Reset usage limits and repeat evaluations as required by project protocols.
• Record findings and submit evaluation datasets on AI training platforms in line with project standards and milestones.
• Maintain a high standard of attention to detail and rigor across all evaluation tasks.

Preferred Qualifications

• Experience with AI products, AI models, or AI agent evaluation and testing.
• Demonstrated ability to follow complex instructions with precision and consistency.
• Exceptional attention to detail and operational thoroughness.
• Someone who already uses AI in daily life.
• Comfort working with repetitive or high-volume operational tasks within strict timeframes.
• Prior participation in digital product, software QA, or AI research projects is a plus.
• Detail-oriented, great at following instructions and documenting each step.

This is a 100% remote contract role, open to applicants worldwide in the countries the employer accepts.