AI Benchmark Quality Reviewer

Braintrust; United States; Remote; USD 26,000 - 37,000; Full time

Job summary

Turing is seeking a remote contractor to support AI evaluation efforts for eight weeks, working 40 hours per week with eight hours of overlap with Pacific Time. The role focuses on verifying task instructions, reference materials, and evaluation criteria for accuracy and completeness.

Ideal candidates have 5+ years of technical or analytical experience and can read Python, SQL, and shell scripts, review execution traces, and provide concise, reproducible feedback with clear uncertainty

Qualifications

• 5+ years of relevant technical or analytical experience.
• Ability to read Python, SQL, shell scripts, and structured data and execution logs.
• Strong written English, analytical judgment and attention to detail.
• Ability to give specific, reproducible feedback and explain uncertainty clearly.
• Experience in AI evaluation or data analysis is helpful but not required.

Responsibilities

• Check task instructions, source materials, reference solutions and evaluation criteria for consistency and completeness.
• Review model execution traces, tool calls and deliverables to assess whether successes and failures are justified.
• Identify brittle grading checks, unsupported criteria and valid alternative solutions that may have been marked incorrect.
• Investigate discrepancies and distinguish model limitations from task, grader, tool or environment issues.
• Write concise, evidence-backed findings and verify that revisions address the issues found.

Skills

Analytical experience

Read Python

Read SQL

Shell scripts

Execution logs

Strong written English

Feedback quality

AI evaluation

Tools

Harbor task setup

Job description

• Rate: $23/hr
• Hours: 40 hours / week
• Experience: 5 - 10 years
• Location: Argentina | Belize + 16 more

What you'll do

• Check task instructions, source materials, reference solutions and evaluation criteria for consistency and completeness.
• Review model execution traces, tool calls and deliverables to assess whether successes and failures are justified.
• Identify brittle grading checks, unsupported criteria and valid alternative solutions that may have been marked incorrect.
• Investigate discrepancies and distinguish model limitations from task, grader, tool or environment issues.
• Write concise, evidence-backed findings and verify that revisions address the issues found.

What we're looking for

• At least five years of relevant technical or analytical experience.
• Ability to read Python, SQL, shell scripts, structured data and execution logs to understand task setup and grading behavior.
• Strong written English, analytical judgment and attention to detail.
• Ability to give specific, reproducible feedback and explain uncertainty clearly.
• Experience in AI evaluation, technical QA, data analysis or benchmark development is helpful, but not required. Familiarity with Harbor task setup is a plus.

Engagement

• Remote contractor assignment for eight weeks.
• 40 hours per week, including eight hours of daily overlap with Pacific Time.
• Shortlisted applicants may be asked to complete an interest form before final review.

Compensation

• $23 per hour.

Company: Turing