AI Benchmark Task Author · Turing Turing · Varies · remote · 10h ago Varies 10h ago

Benture; San Francisco (CA); Remote; USD 55,000 - 110,000; Part time

Job summary

Turing, based in San Francisco, is hiring an analytical researcher and red-teaming specialist to build evidence-based benchmark tasks for exposing weaknesses in state-of-the-art AI models. You will verify facts, draft precise prompts, and test tasks against leading models, documenting sources and results with primary evidence.

Remote contractor role with PST overlap. Ideal candidates have 1+ years in AI research, OSINT, and benchmark authoring, plus a Bachelor's degree in a relevant engineering

Qualifications

• Bachelor's degree in a relevant engineering field is required.
• 1+ years of investigative research, AI benchmark task authoring, red-teaming, or technical QA review.
• Strong OSINT and web-sourcing abilities.
• Ability to inspect model reasoning chains, logs and source selection to diagnose failures.
• Clear, unambiguous benchmark prompts with primary-sourced answer keys.

Responsibilities

• Research niche topics to uncover factual ground truths not easily surfaced by straightforward queries.
• Verify facts against primary or highly credible sources and document exact URLs.
• Draft clear benchmark tasks to expose weaknesses in model research strategies.
• Run tasks against SOTA models to confirm failure modes and document results.
• Trace model run logs and search trajectories to pinpoint failure points, vs. ambiguous prompts.
• Review tasks authored by peers and apply validated research patterns across domains.

Skills

Investigative research

Fact-checking

AI benchmark task authoring

Red-teaming

Technical QA review

OSINT

Education

Bachelor's degree (engineering)

Tools

GAIA

BrowseComp

SimpleQA

SWE-bench

Terminal-Bench

Humanity's Last Exam (HLE)

Job description

AI Benchmark Task Author | Contractor | Remote | Rate varies | Up to 40 hrs/week with 4 hrs PST overlap

Turing, based in San Francisco, works with frontier AI labs and enterprises on AI research, data, and training. It is hiring an analytical researcher and red-teaming specialist to build evidence-based web-research benchmark tasks that expose weaknesses in state-of-the-art AI models (e.g., Claude Opus 5 Max). You will discover hard-to-surface facts, build precise research tasks around them, test them directly against leading LLMs, and diagnose exactly how the model's search and reasoning paths fail.

What the Work Involves

• Research niche topics to find factual ground truths that cannot be surfaced with simple, direct search queries.
• Verify facts against primary or highly credible sources, cross-check for contradictory evidence, and document exact source URLs.
• Draft clear, answerable benchmark tasks designed to expose structural weaknesses in model research strategies.
• Run tasks against designated SOTA models to confirm they reliably trigger the intended model failures.
• Trace model run logs, search trajectories, and intermediate outputs to pinpoint where and why the model failed, distinguishing genuine model limitations from ambiguous prompts.
• Apply validated "model-breaking" research patterns across new domains; strong performers also review tasks authored by peer trainers.

Requirements

• 1+ years in investigative research, fact-checking, AI benchmark task authoring, red-teaming, or technical QA review.
• Experience authoring, testing, or reviewing tasks for AI benchmarks such as GAIA, BrowseComp, SimpleQA, SWE-bench, Terminal-Bench, or Humanity's Last Exam (HLE).
• Intermediate-to-advanced web search and OSINT skills; ability to navigate academic archives, local records, or obscure databases.
• Ability to inspect model reasoning chains, search logs, and source selection to diagnose root causes of failure.
• Precision writing: clear, unambiguous benchmark prompts with bulletproof, primary-sourced answer keys.
• Bachelor's degree or higher in a relevant engineering domain (described as a must in the listing).

Preferred Qualifications

• Experience designing prompt-injection or reasoning-trap tasks that break SOTA models.
• Advanced OSINT capabilities or published investigative/academic research demonstrating deep source verification.
• Niche domain expertise in areas rich with hard-to-find records (e.g., local government filings, regional sports history, legal archives, historical patents).
• Experience as a lead reviewer, benchmark auditor, or quality owner on AI evaluation projects.

Schedule and Engagement

• Up to 40 hours per week, with 4 hours of overlap with PST.
• Contractor engagement, up to 4 weeks long.
• Hourly rate is not stated in the listing.