AI Research Engineer: Agent Evaluation & Tools (Remote)

Wwshemi; New York, San Francisco, Los Angeles (NY, CA, CA); Hybrid; USD 175,000 - 195,000; Full time

Benefits offered by this job

Remote first - US

WFH stipend

100% paid medical, dental, vision

US$1,000 WFH stipend

Internet & phone reimbursement

Unlimited vacation

Parental leave

Fertility benefits

Equity incentive

Job summary

Gauntlet is hiring an AI Research Engineer to advance evaluation frameworks for AI agents and to improve how agents are measured in real tasks. You will work with engineers and researchers to translate observations into experiments and scalable improvements across production agents and tooling.

You will contribute to Aria and other agents, while growing into more open-ended research on agent behavior, evaluation and infrastructure.

Qualifications

• Hands-on experience building, evaluating or improving LLM models or agents.
• Strong Python skills and production-quality code practices.
• Experience with applied research, machine learning or data science; designing experiments.

Responsibilities

• Build evaluation frameworks for AI agents, datasets and metrics for real-world tasks.
• Design experiments to compare models, agent harnesses, tools and environments.
• Collaborate with Gauntlet teams to turn feedback into measurable outcomes.
• Investigate agent failures and improve performance across models, harness and tooling.
• Build and improve agent harnesses, internal tools and integrations, contributing to Aria and others.
• Develop shared knowledge and reusable skills across agents and systems.
• Ensure agents operate reliably and safely within production.

Skills

LLM models

Agent frameworks

Python

Applied research

Experiment design

Software engineering

Open-source contributions

Independent work

Job description

Gauntlet is hiring an AI Research Engineer to advance evaluation frameworks for AI agents and to improve how agents are measured in real tasks. You will work with engineers and researchers to translate observations into experiments and scalable improvements across production agents and tooling.

You will contribute to Aria and other agents, while growing into more open-ended research on agent behavior, evaluation and infrastructure.