AI Evaluation Specialist | $35/hr Remote

Crossing Hurdles; India; On-site; INR 600,000 - 900,000; Full time

Job summary

Crossing Hurdles in India seeks an experienced evaluator to design and implement self-contained evaluation tasks for AI-driven workflows. You will develop prompts, supporting files, and detailed grading rubrics to benchmark performance across scenarios.

_Define clear, unambiguous criteria for task success_ and meticulously observe AI behaviors, producing precise English reports. Collaborate with the customer team to drive continuous improvement in evaluation techniques.

Qualifications

• Experience in roles emphasizing written precision and structured thinking.
• Fluent in English writing with succinct, unambiguous observations.
• Skill in rubric-based evaluation or structured scoring frameworks.
• High attention to detail and ability to notice subtle patterns.
• Excellent written and verbal communication for documenting observations and feedback.

Responsibilities

• Design and implement evaluation tasks with prompts, supporting files, and detailed rubrics.
• Define clear, unambiguous criteria for task success across scenarios.
• Observe AI agent behaviors and produce precise summaries in high-quality English.
• Iterate and refine evaluation tasks and rubrics based on feedback and collaboration.
• Collaborate with the customer team to share insights and drive continuous improvement.

Skills

Evaluation design

Rubric development

Observational reporting

Feedback iteration

Team collaboration

Job description

• Design and implement self-contained evaluation tasks, including prompts, supporting files, and detailed grading rubrics to assess AI performance on practical computer-based workflows.
• Define clear, unambiguous written criteria for successful and unsuccessful task completion across diverse administrative and workflow scenarios.
• Meticulously observe and document AI agent behaviors, producing crisp, precise summaries and reports in high-quality English.
• Iterate and refine evaluation tasks and rubrics based on feedback and team collaboration to ensure robust benchmarking methodologies.
• Collaborate with the customer's team to share insights and help drive continuous improvement in AI evaluation techniques.

Requirements

• Have a minimum of of experience in roles emphasizing written precision and structured thinking, such as paralegal, executive assistant, junior analyst, librarian, document archival specialist, research assistant, technical writer, or QA analyst.
• Be native or fluent in English writing, with a demonstrated ability to produce observations that are succinct, specific, and unambiguous.
• Proven skill in designing or applying rubric-based evaluation, grading against set criteria, or building structured scoring frameworks is required.
• Possess high attention to detail and the ability to notice subtle patterns or inconsistencies others might miss.
• Exceptional written and verbal communication skills are necessary, especially for documenting nuanced observations and feedback.