AI Benchmark & Red-Teaming Specialist

HumanitApp; San Francisco (CA); On-site; USD 83,000 - 138,000; Part time

Job summary

Turing, based in San Francisco, CA, is seeking an analytical researcher and red-teaming specialist to design evidence-based web-research benchmarks that reveal weaknesses in state-of-the-art AI models. You will craft precise tasks, test them against leading LLMs, and diagnose failure modes in reasoning paths.

This contractor role offers up to 40 hours/week with four hours of overlap with PST and runs for up to four weeks. Compensation details are provided separately by the partner platform.

Qualifications

• 1+ years in investigative research, fact-checking, AI benchmark task authoring, red-teaming, or technical QA review.
• Experience authoring, testing, or reviewing AI benchmarks (GAIA, BrowseComp, SimpleQA, SWE-bench, Terminal-Bench, HLE).
• Strong OSINT and web research skills with ability to locate obscure sources.
• Ability to inspect model reasoning chains and diagnose failure causes from logs or traces.
• Clear, precise benchmark prompts with primary-sourced answer keys.

Responsibilities

• Obscure Fact Discovery: research niche topics to identify factual truths not easily surfaced by simple queries.
• Source Verification: verify facts against primary sources and document sources with URLs.
• Task Construction: draft clear, answerable benchmark tasks to expose weaknesses.
• Red-Teaming & Model Testing: run tasks against SOTA models to trigger failures.
• Failure Diagnosis & Debugging: trace logs to locate where and why the model failed.
• Pattern Scaling & Review: apply patterns across domains and review peers' tasks.

Skills

Investigative research

OSINT

Analytical thinking

Benchmark task authoring

Red-teaming

Technical QA review

Education

Bachelor or higher degree in engineering or related field

Job description

Future projects, not immediate vacancies. Check each network's country and experience requirements.

Based in San Francisco, California, Turing is the world’s leading research accelerator for frontier AI labs and a trusted partner for global enterprises deploying advanced AI systems. Turing supports customers in two ways: first, by accelerating frontier research with high-quality data, advanced training pipelines, plus top AI researchers who specialize in coding, reasoning, STEM, multilinguality, multimodality, and agents; and second, by applying that expertise to help enterprises transform AI from proof of concept into proprietary intelligence with systems that perform reliably, deliver measurable impact, and drive lasting results on the P&L.

Role Overview

We are seeking an analytical researcher and red-teaming specialist to build evidence-based web-research benchmark tasks that reliably expose weaknesses in state-of-the-art (SOTA) AI models (e.g., Claude Opus 5 Max). In this role, you will discover hard-to-surface facts, build precise research tasks around them, test them directly against leading LLMs, and diagnose the exact failure mechanisms in the model's search and reasoning paths.

Your work will directly shape AI evaluation frameworks, ensuring next-generation frontier models are tested against rigorous, non-trivial real-world research scenarios.

Job Requirements

Core Experience: 1+ years in investigative research, fact-checking, AI benchmark task authoring, red-teaming, or technical QA review.

Benchmark Familiarity: Proven experience authoring, testing, or reviewing tasks for AI benchmarks such as GAIA, BrowseComp, SimpleQA, SWE- bench, Terminal-Bench, or Humanity’s Last Exam (HLE).

Analytical & OSINT Skills: Intermediate-to-advanced web search and Open Source Intelligence (OSINT) skills; ability to navigate academic archives, local records, or obscure databases.

Model Trajectory Debugging: Strong analytical capability to inspect model reasoning chains, search logs, and source selection to diagnose root causes of failure.

Precision Writing: Ability to author clear, unambiguous benchmark prompts with bulletproof, primary-sourced answer keys.

Job Responsibilities

Obscure Fact Discovery: Deeply research niche topics to identify factual ground truths that cannot be surfaced via simple, direct search queries.

Source Verification: Rigorously verify facts against primary or highly credible sources, cross-check for contradictory evidence, and document exact source URLs.

Task Construction: Draft clear, answerable benchmark tasks designed specifically to expose structural weaknesses in model research strategies.

Red-Teaming & Model Testing: Execute tasks against designated SOTA models to confirm they reliably trigger intended model failures.

Failure Diagnosis & Debugging: Trace model run logs, search trajectories, and intermediate outputs to pinpoint exactly where and why the model failed (distinguishing genuine model limitations from ambiguous prompts).

Pattern Scaling & Review: Apply validated model-breaking research patterns across new domains and, for strong performers, review tasks authored by peer trainers.

Strong Plus (Preferred Qualifications)

Bachelor or higher degree from relevant engineering domain is a must for this position.

Prior experience specifically designing prompt-injection or reasoning-trap tasks that break SOTA models.

Advanced OSINT capabilities or published investigative/academic research demonstrating deep source verification.

Niche domain expertise in areas rich with hard-to-find records (e.g., local government filings, regional sports history, legal archives, historical patents).

Experience serving as a lead reviewer, benchmark auditor, or quality owner on AI evaluation projects.

Commitments Required: Upto 40 hours per week with 4 hours of overlap with PST.

Engagement type: Contractor

Engagement Length: upto 4 weeks

Evaluation Process

Evaluation Process -

LLM Quality Management QA

Compensation context

Compensation is not stated in the current feed. Confirm the rate and workload before accepting an assessment or project.

• Review the official description below for experience and equipment requirements.
• Confirm location eligibility, language requirements and weekly availability with the partner.

Not the right fit?

Use your CV to find relevant roles on HumanitApp. This is separate from a partner application.

Compensation is not stated in the current feed. Confirm the rate and workload before accepting an assessment or project.

How should I compare the listed pay?

Compensation is not stated in the current feed. Confirm the rate and workload before accepting an assessment or project.

No. HumanitApp independently curates the opportunity. The partner platform manages applications and hiring decisions.

Is HumanitApp the employer?

No. HumanitApp independently curates the opportunity and the partner platform manages the hiring decisions.