Remote AI Research Engineer — Agent Evaluation & Frameworks

Gauntlet; New York, San Francisco, Los Angeles (NY, CA, CA); Hybrid; USD 175,000 - 195,000; Full time

Benefits offered by this job

Remote first

Company retreats

Medical, dental, vision insurance

WFH stipend

Internet stipend

Unlimited vacation

Parental leave

Fertility benefits

Incentive compensation

Job summary

Gauntlet is seeking an AI Research Engineer to advance evaluation, training and deployment of AI agents across the business. You will help build systems to measure performance on real tasks and turn observations into experiments that improve our agents and tooling.

The role blends engineering and research, starting with hands-on contributions to Aria and internal tooling, then expanding toward open-ended research on evaluation, agent behavior and infrastructure. Remote, US-friendly.

Qualifications

• Hands-on experience building or evaluating LLM models or agents, in research, industry, or open-source contexts.
• Ability to design controlled experiments to measure agent performance and compare approaches.
• Proven track record turning qualitative feedback into actionable research questions and measurable outcomes.

Responsibilities

• Build evaluation frameworks for AI agents, including datasets, sandboxes and metrics for real tasks.
• Design experiments to compare models, agent harnesses and environments, identifying regressions.
• Collaborate with Gauntlet teams to translate qualitative feedback into research questions and metrics.
• Investigate agent failures to pinpoint model, harness, tooling or prompting issues and drive improvements.
• Develop and improve agent harnesses, internal tools and integrations, contributing to Aria and other agents.
• Help create shared knowledge and reusable skills across agents for ongoing learning.

Skills

LLM evaluation

Python

Experiment design

Research

Automation

Education

Advanced degree in CS/ML

Tools

Hermes

OpenClaw

AI tooling

Job description

Gauntlet is seeking an AI Research Engineer to advance evaluation, training and deployment of AI agents across the business. You will help build systems to measure performance on real tasks and turn observations into experiments that improve our agents and tooling.

The role blends engineering and research, starting with hands-on contributions to Aria and internal tooling, then expanding toward open-ended research on evaluation, agent behavior and infrastructure. Remote, US-friendly.