Applied AI Researcher - Benchmarking & Task Design

raydar; New York (NY); On-site; USD 160,000 - 200,000; Full time

Benefits offered by this job

Health coverage

Job summary

Raydar is seeking an Applied Researcher to design and run evaluations that test how AI models handle demanding technical work. You will craft task suites, define inputs, tools and verification steps, and turn findings into training signals for model improvement.

You will partner with domain specialists, review model outputs, and report findings with actionable next steps. This role blends task design, scoring and repeatable experiment pipelines across domains.

Qualifications

• Track record in building test suites and scoring methods for model evaluation.
• Hands-on experience designing benchmarks and assessing AI systems.
• Sufficient machine learning background to judge model capabilities.
• Comfort with Python and SQL.

Responsibilities

• Set up task suites mirrored to investigator needs with documented methods, reference scores and withheld data.
• Turn domain data and procedures into runnable tasks with inputs, permitted tools, target results and verification steps.
• Write scoring criteria and automated verifiers and probe them for vagueness and loopholes.
• Mine error patterns for the training tasks worth pursuing and check gains on unseen problems.
• Stand up repeatable systems that generate tasks, run models, score outputs and summarize results with logs.
• Review model outputs and tool usage, classifying misses as model shortfalls or task flaws.
• Partner with domain specialists to shape task construction and checking, and agree on success criteria.
• Report on gains, open questions and the next experiment or engineering step.

Skills

Python

SQL

Education

Biology, Biomedical or Chemical Engineering, Physics degree

Job description

Raydar is seeking an Applied Researcher to design and run evaluations that test how AI models handle demanding technical work. You will craft task suites, define inputs, tools and verification steps, and turn findings into training signals for model improvement.

You will partner with domain specialists, review model outputs, and report findings with actionable next steps. This role blends task design, scoring and repeatable experiment pipelines across domains.