Research Engineer - Evals

AGI Inc; San Francisco (CA); On-site; USD 120,000 - 160,000; Full time

Benefits offered by this job

Competitive cash and equity

Top-tier relocation support

Job summary

AGI Inc, located in San Francisco, is seeking a dedicated professional for a full-time role to evaluate agent models and develop practical assessment rubrics. This position offers competitive cash and equitable stock options along with top-tier relocation support.

After 30 days, you will audit evaluations and pinpoint improvements; by 60 days, you will implement new evaluation tools to aid decision-making. This role is pivotal to ensure product quality and enhance the research strategy.

Qualifications

• Experience in auditing evaluation metrics and producing documentation.
• Ability to implement new evaluation surfaces and support researchers.

Responsibilities

• Audit every evaluation metric and identify gaps after 30 days.
• Establish a new eval surface for decision-making after 60 days.
• Ensure releases meet evaluation standards and influence research roadmap after 90 days.

Skills

Measuring non-deterministic systems

Pushing back on gamed metrics

Job description

We're building useful AGI for everyday life

At AGI, Inc., we are looking for passionate builders, innovators, and dreamers to join our journey to bring genuinely useful AGI into everyday life.

Employment Type

Full time

Tasks you will own

• The eval suites that gate every model and agent release — capability, behavior, regressions, and human-rated rubrics that catch what automated evals miss
• The dashboards and tooling that make researcher experiment loops fast and leadership decisions easy
• The bar — what counts as ready to ship, and how we know

Areas where you will assist

• Research, by making sure what we measure is what we want
• Product engineers, by instrumenting real-user behavior on real devices
• Partnerships, by translating "did it get better" into language an OEM partner can hold us to

Skills you'll be expected to teach

• How to measure non-deterministic systems — agent eval, tool use, long-horizon tasks, multilingual behavior
• How to push back on a metric that's being gamed without breaking the team

Skills you'll be expected to learn

• On-device perf trade-offs and how they show up in real-user evals
• What QA-ing AI at OEM scale actually looks like
• The realities of shipping consumer agents to production partners

Timeline of success

After 30 days — You've audited every eval we run today and produced a sharp doc on what's good, what's noise, and what's missing. You've fixed the most embarrassing gap.

After 60 days — You've stood up a new eval surface — agentic, on-device, or behavioral — and the team is making real decisions on its output. Researchers come to you before launching a run, not after.

After 90 days — Releases now ship against your eval bar, not a vibe-check. You've caught a regression that would have shipped, and cleared a launch the team was nervous about. You're shaping the research roadmap by surfacing where we're flat, where we're climbing, and where we're lying to ourselves.

Competitive cash and meaningful equity. Top-tier relocation and immigration support. SF, in person.