Frontier AI Benchmarking & Red-Teaming Analyst

HumanitApp; San Francisco (CA); Presencial; USD 83 000 - 138 000; Tempo parcial

Resumo da oferta

Turing, based in San Francisco, CA, is seeking an analytical researcher and red-teaming specialist to design evidence-based web-research benchmarks that reveal weaknesses in state-of-the-art AI models. You will craft precise tasks, test them against leading LLMs, and diagnose failure modes in reasoning paths.

This contractor role offers up to 40 hours/week with four hours of overlap with PST and runs for up to four weeks. Compensation details are provided separately by the partner platform.

Qualificações

• 1+ years in investigative research, fact-checking, AI benchmark task authoring, red-teaming, or technical QA review.
• Experience authoring, testing, or reviewing AI benchmarks (GAIA, BrowseComp, SimpleQA, SWE-bench, Terminal-Bench, HLE).
• Strong OSINT and web research skills with ability to locate obscure sources.
• Ability to inspect model reasoning chains and diagnose failure causes from logs or traces.
• Clear, precise benchmark prompts with primary-sourced answer keys.

Responsabilidades

• Obscure Fact Discovery: research niche topics to identify factual truths not easily surfaced by simple queries.
• Source Verification: verify facts against primary sources and document sources with URLs.
• Task Construction: draft clear, answerable benchmark tasks to expose weaknesses.
• Red-Teaming & Model Testing: run tasks against SOTA models to trigger failures.
• Failure Diagnosis & Debugging: trace logs to locate where and why the model failed.
• Pattern Scaling & Review: apply patterns across domains and review peers' tasks.

Conhecimentos

Investigative research

OSINT

Analytical thinking

Benchmark task authoring

Red-teaming

Technical QA review

Formação académica

Bachelor or higher degree in engineering or related field

Descrição da oferta de emprego

Turing, based in San Francisco, CA, is seeking an analytical researcher and red-teaming specialist to design evidence-based web-research benchmarks that reveal weaknesses in state-of-the-art AI models. You will craft precise tasks, test them against leading LLMs, and diagnose failure modes in reasoning paths.

This contractor role offers up to 40 hours/week with four hours of overlap with PST and runs for up to four weeks. Compensation details are provided separately by the partner platform.