AI Benchmark Researcher

Braintrust; Puerto Rico; On-site; Full time

Job summary

Braintrust seeks experts to identify complex problems in their field and translate them into Linux-based tasks for testing advanced AI agents. You will design challenges evaluated against multiple models, ensuring difficulty and novelty.

Applicants should have 10+ years in technical domains, publications, and be comfortable using a Linux terminal, with the posting noting residency in the United States.

Qualifications

• Master’s degree or higher required; candidates with a bachelor’s degree plus 10+ years experience may be considered.
• At least 10 years of professional experience in a technical domain is required.
• Expertise in coding, machine learning, systems, cybersecurity or hardware.
• At least one academic or professional publication is required.
• Practical comfort working in a Linux terminal.
• Currently based in the United States.

Responsibilities

• Identify difficult, meaningful problems in your area of expertise and translate them into tasks.
• Design challenges tested against three leading AI models with three attempts per model.
• Prove tasks are difficult enough that models mostly fail.

Skills

Coding expertise

Machine learning

Systems

Cybersecurity

Hardware

Education

Master’s degree or higher

Bachelor’s degree + 10 years experience may be considered

Tools

Linux terminal

Job description

Bring your expertise to the problems today’s most advanced AI agents still struggle to solve.

What you’ll do

• Identify difficult, meaningful problems in your area of expertise.
• Translate those problems into tasks that can be solved in a Linux terminal.
• Design challenges that expose limitations in advanced AI agents. Each task is tested against three leading models, with three attempts per model, and must prove difficult enough that the models mostly fail.

Who we’re looking for

• A completed master’s degree or higher. Candidates with a bachelor’s degree, at least 10 years of relevant professional experience and a publication may also be considered.
• At least 10 years of professional experience in a technical domain.
• Expertise in coding, machine learning, systems, cybersecurity or hardware.
• At least one academic or professional publication.
• Practical comfort working in a Linux terminal.
• Currently based in the United States.

Project details

• Task-based project expected to run for at least seven weeks.
• Onboarding includes two courses and a graded assessment, taking approximately 90 minutes.
• A live onboarding webinar is also required.

Compensation

• Earn up to $1,350 per task, with opportunities for higher rates based on task quality and submission volume.
• Tasks take six to seven hours on average, although initial tasks may take longer as you get familiar with the process.