Staff AI Quality Engineer — RAG & Agent Testing

ID.me; San Francisco (CA); On-site; USD 218,000 - 271,000; Full time

Benefits offered by this job

Medical insurance

Dental insurance

Vision insurance

Health savings account

Flexible spending accounts

401(k) with company match

Parental leave

Unlimited paid time off

Disability insurance

Life and AD&D insurance

Learning and development benefit

Job summary

ID.me is seeking a Staff Engineer to define AI quality standards, build eval infrastructure, and ensure production reliability of agentic AI features. You will mentor engineers, shape testing strategies, and collaborate with product and security teams to embed quality into every stage of development.

Based in Mountain View, CA, this role emphasizes non-deterministic AI systems, retrieval-augmented generation, and scalable test architectures to reduce hallucinations and improve user trust.

Qualifications

• Bachelor's degree in Computer Science, Engineering, or equivalent experience.
• 8+ years building and operating production software systems.
• Experience evaluating or testing LLM-powered features or autonomous agents in production.
• Proficiency with AI-assisted development tools and building with AI daily.
• Strong backend engineering fundamentals in Python, Java, Go, or equivalent.
• Experience designing test infrastructure, CI/CD quality gates, or evaluation pipelines at scale.
• Experience improving developer experience and building internal tooling.

Responsibilities

• Define AI Quality Standards: framework for evaluating, validating, and monitoring AI agents across features and workflows.
• Build Eval Infrastructure: pipelines for LLM outputs, agent behavior, tool use, and multi-turn interactions.
• Production Observability for Agents: monitor drift, regression, failure modes, latency, hallucination rate, tool misuse, policy adherence.
• Agentic Test Strategy: design test suites for non-determinism, red-teaming, golden datasets, LLM-as-judge, and property-based testing.
• Champion Developer Experience: build internal tooling, feedback loops, testing workflows to speed safe AI delivery.
• Drive AI-First Engineering Culture: raise quality bars, patterns, education for responsible AI features.
• Cross-Team Collaboration: embed quality gates with Security, Platform, Product, and AI/ML teams.
• Mentorship: guide engineers on evaluation design, observability, and testing for AI systems.

Skills

Python

Java

Go

Backend engineering

LLM testing

CI/CD quality gates

Evaluation pipelines

AI tooling

Education

Bachelor's degree in Computer Science, Engineering, or equivalent

Tools

Claude Code

Cursor

Anthropic SDK

Job description

ID.me is seeking a Staff Engineer to define AI quality standards, build eval infrastructure, and ensure production reliability of agentic AI features. You will mentor engineers, shape testing strategies, and collaborate with product and security teams to embed quality into every stage of development.

Based in Mountain View, CA, this role emphasizes non-deterministic AI systems, retrieval-augmented generation, and scalable test architectures to reduce hallucinations and improve user trust.