AI Benchmark & Red-Teaming Research Lead

Turing; United States; À distance; USD 83 000 - 124 000; Plein temps

Résumé du poste

Turing, based in San Francisco, seeks an analytical researcher to design and test benchmark tasks that expose weaknesses in state-of-the-art AI models. You will identify hard-to-surface facts, craft precise prompts, and test against leading LLMs to diagnose failure modes in reasoning paths.

This contractor role emphasizes rigorous source verification, evidence-based task construction, and the ability to document primary sources and exact URLs, with about 40 hours per week and a four-week

Qualifications

• Core Experience: 1+ years in investigative research, fact-checking, AI benchmark task authoring, red-teaming, or technical QA review.
• Benchmark Familiarity: Experience with AI benchmarks such as GAIA, BrowseComp, SimpleQA, SWE-bench, Terminal-Bench, or Humanity’s Last Exam (HLE).
• Analytical & OSINT Skills: Intermediate-to-advanced web search and OSINT; ability to navigate academic archives, local records, or obscure databases.
• Model Trajectory Debugging: Analyze model reasoning chains, search logs, and source selection to diagnose failures.
• Precision Writing: Write clear, unambiguous benchmark prompts with primary-sourced answer keys.

Responsabilités

• Obscure Fact Discovery: Deeply research niche topics to identify factual ground truths that cannot be surfaced via simple, direct search queries.
• Source Verification: Rigorously verify facts against primary or highly credible sources, cross-check for contradictory evidence, and document exact source URLs.
• Task Construction: Draft clear, answerable benchmark tasks designed to expose structural weaknesses in model research strategies.
• Red-Teaming & Model Testing: Execute tasks against designated SOTA models to confirm they reliably trigger intended model failures.
• Failure Diagnosis & Debugging: Trace model run logs, search trajectories, and intermediate outputs to pinpoint exactly where and why the model failed.
• Pattern Scaling & Review: Apply validated 'model-breaking' research patterns across new domains and, for strong performers, review tasks authored by peer trainers.

Connaissances

LLM

Quality Management

QA

Description du poste

Turing, based in San Francisco, seeks an analytical researcher to design and test benchmark tasks that expose weaknesses in state-of-the-art AI models. You will identify hard-to-surface facts, craft precise prompts, and test against leading LLMs to diagnose failure modes in reasoning paths.

This contractor role emphasizes rigorous source verification, evidence-based task construction, and the ability to document primary sources and exact URLs, with about 40 hours per week and a four-week