Staff Software Engineer- AI Agent Evaluations

ID.me; San Francisco (CA); On-site; USD 218,000 - 271,000; Full time

Benefits offered by this job

Medical insurance

Dental insurance

Vision insurance

Health savings account

Flexible spending accounts

401(k) with company match

Parental leave

Unlimited paid time off

Disability insurance

Life and AD&D insurance

Learning and development benefit

Job summary

ID.me is seeking a Staff Engineer to define AI quality standards, build eval infrastructure, and ensure production reliability of agentic AI features. You will mentor engineers, shape testing strategies, and collaborate with product and security teams to embed quality into every stage of development.

Based in Mountain View, CA, this role emphasizes non-deterministic AI systems, retrieval-augmented generation, and scalable test architectures to reduce hallucinations and improve user trust.

Qualifications

• Bachelor's degree in Computer Science, Engineering, or equivalent experience.
• 8+ years building and operating production software systems.
• Experience evaluating or testing LLM-powered features or autonomous agents in production.
• Proficiency with AI-assisted development tools and building with AI daily.
• Strong backend engineering fundamentals in Python, Java, Go, or equivalent.
• Experience designing test infrastructure, CI/CD quality gates, or evaluation pipelines at scale.
• Experience improving developer experience and building internal tooling.

Responsibilities

• Define AI Quality Standards: framework for evaluating, validating, and monitoring AI agents across features and workflows.
• Build Eval Infrastructure: pipelines for LLM outputs, agent behavior, tool use, and multi-turn interactions.
• Production Observability for Agents: monitor drift, regression, failure modes, latency, hallucination rate, tool misuse, policy adherence.
• Agentic Test Strategy: design test suites for non-determinism, red-teaming, golden datasets, LLM-as-judge, and property-based testing.
• Champion Developer Experience: build internal tooling, feedback loops, testing workflows to speed safe AI delivery.
• Drive AI-First Engineering Culture: raise quality bars, patterns, education for responsible AI features.
• Cross-Team Collaboration: embed quality gates with Security, Platform, Product, and AI/ML teams.
• Mentorship: guide engineers on evaluation design, observability, and testing for AI systems.

Skills

Python

Java

Go

Backend engineering

LLM testing

CI/CD quality gates

Evaluation pipelines

AI tooling

Education

Bachelor's degree in Computer Science, Engineering, or equivalent

Tools

Claude Code

Cursor

Anthropic SDK

Job description

Company Overview ID.me is the next-generation digital identity wallet that simplifies how individuals securely prove their identity online. Consumers can verify their identity with ID.me once and seamlessly login across websites without having to create a new login and verify their identity again. Over 152 million users experience streamlined login and identity verification with ID.me at 20 federal agencies, 45 state government agencies, and 70+ healthcare organizations. More than 600+ consumer brands use ID.me to verify communities and user segments to honor service and build more authentic relationships. ID.me’s technology meets the federal standards for consumer authentication set by the Commerce Department and is approved as a NIST 800-63-3 IAL2 / AAL2 credential service provider by the Kantara Initiative. ID.me is committed to “No Identity Left Behind” to enable all people to have a secure digital identity. To learn more, visit https://network.id.me/. Company Overview ID.me is the next-generation digital identity wallet that simplifies how individuals securely prove their identity online. Consumers can verify their identity with ID.me once and seamlessly login across websites without having to create a new login and verify their identity again. Over 152 million users experience streamlined login and identity verification with ID.me at 20 federal agencies, 45 state government agencies, and 70+ healthcare organizations. More than 600+ consumer brands use ID.me to verify communities and user segments to honor service and build more authentic relationships. ID.me’s technology meets the federal standards for consumer authentication set by the Commerce Department and is approved as a NIST 800-63-3 IAL2 / AAL2 credential service provider by the Kantara Initiative. ID.me is committed to “No Identity Left Behind” to enable all people to have a secure digital identity. To learn more, visit https://network.id.me/. ID.me is a full-time, in-office culture. Unless a specific job description explicitly states otherwise, all roles are on-site five days per week at one of our offices in McLean, VA; Mountain View, CA; New York City, NY; or Tampa, FL. Certain roles — such as field-based sales or other remote‑by‑design positions — may have different work arrangements as noted in their individual postings. At ID.me, we embrace the thoughtful use of AI tools in our daily work and there are even occasions where we leverage AI in our hiring process. However, during the interview process, we want to understand your individual skills and experiences. Therefore, we have guidelines on how AI can be appropriately used during your application and interviews which can be found here. About The Role This Staff Engineer role sits at the intersection of engineering, applied AI, testing and developer experience. You will define and lead the discipline of testing AI agents, evaluating LLM behavior, and ensuring the reliability of agentic systems operating in production. It requires deep engineering rigor, original thinking about what “correctness” means for non-deterministic systems, and the ability to build eval infrastructure and developer tooling that the entire engineering org depends on.

Expert in building and maintaining Retrieval-Augmented Generation (RAG) pipelines, with a deep focus on strategic data chunking and data quality enforcement. Experience in establishing pre-retrieval data quality gates to optimize vector search accuracy, minimize retrieval‑induced noise, and significantly reduce LLM hallucination rates in production‑deployed agent systems.

You will establish quality standards for how ID.me ships AI‑powered features safely, mentor engineers across teams on AI testing best practices, and partner directly with product and platform teams to embed quality into every stage of agent development.

What You’ll Do

• Define AI Quality Standards: Own the framework for how ID.me evaluates, validates, and monitors AI agents — from prompt‑based features to fully autonomous multi‑step workflows.
• Build Eval Infrastructure: Design and maintain evaluation pipelines for LLM outputs, agent behavior, tool use, and multi‑turn interactions across development, staging, and production environments.
• Production Observability for Agents: Instrument agentic systems for behavioral drift, regression, and failure modes that traditional metrics miss — latency, correctness, hallucination rate, tool misuse, and policy adherence.
• Agentic Test Strategy: Lead the design of test suites that handle non‑determinism — red‑teaming agents, golden dataset construction, LLM‑as‑judge pipelines, and property‑based testing for AI outputs.
• Champion Developer Experience: Build the internal tooling, feedback loops, and testing workflows that make it fast and safe for engineers to develop and ship AI features with confidence. Reduce friction in the agent development inner loop — local testing, fast eval runs, and clear signal on regressions.
• Drive AI‑First Engineering Culture: Raise the quality bar across the engineering org by establishing patterns, tooling, and education for how teams write, test, and deploy AI features responsibly.
• Cross‑Team Collaboration: Partner with Security, Platform, Product, and AI/ML teams to embed quality gates into agent development workflows.
• Mentorship: Guide senior and mid‑level engineers through evaluation design, observability strategy, and testing approaches specific to AI systems.

Basic Qualifications

• Bachelor's degree in Computer Science, Engineering, or equivalent experience
• 8+ years building and operating production software systems
• Demonstrated experience evaluating or testing LLM‑powered features or autonomous agents in production
• Proficiency with AI‑assisted development tools (Claude Code, Cursor, or equivalent) — you build with AI every day
• Strong backend engineering fundamentals in Python, Java, Go, or equivalent
• Experience designing test infrastructure, CI/CD quality gates, or evaluation pipelines at scale
• Experience improving developer experience — building internal tooling, reducing toil, or accelerating engineering workflows
• Proven ability to lead cross‑team technical initiatives and influence engineering standards
• Strong written and verbal communication across engineering, product, and leadership
• Experience building eval frameworks for LLM agents (e.g., correctness graders, LLM‑as‑judge, human‑in‑the‑loop evals, benchmark dataset curation)
• Familiarity with agentic frameworks (Claude API / Anthropic SDK, BrainTrust, LangChain, LangGraph, CrewAI, or similar)
• Production monitoring experience for AI systems: behavioral drift detection, output sampling, shadow scoring
• Red‑teaming or adversarial testing experience for AI models or agents

Preferred Qualifications

• Background in identity verification, fraud detection, or regulated industries
• Familiarity with Anthropic's model evaluation methodology or similar published eval research
• Experience with observability tooling (Datadog, OpenTelemetry) applied to AI workloads
• Track record of building developer tooling or platforms that other teams adopt widely

The annual base salary listed does not include a company bonus, incentive for sales roles, equity and benefits which will be determined based on experience, skills, education, relevant training, geographic location and role. ID.me offers comprehensive medical, dental, vision, health savings account, flexible spending accounts (medical, limited purpose, dependent care, commuter benefit accounts), basic and voluntary life and AD&D insurance, 401(k) with company match, parental leave, ability to participate in unlimited paid time off subject to the terms and conditions of the PTO policy, including 8 company wide holidays, short and long‑term disability insurance, accident and critical illness insurance, referral bonus policy, employee assistance program, pet insurance, travel assistant program, wellbeing and childcare discounts, benefit advocates, and a learning and development benefit. Final offers may vary from the amount listed based on qualifications, professional experiences, skills, education, relevant training, geographic location, and other job related factors. Mountain View, CA Pay Range $217,565—$271,000 USD ID.me maintains a work environment free from discrimination, where employees are treated with dignity and respect. All ID.me employees share in the responsibility for fulfilling our commitment to equal employment opportunity. ID.me does not discriminate against any employee or applicant on the basis of age, ancestry, color, family or medical care leave, gender identity or expression, genetic information, marital status, medical condition, national origin, physical or mental disability, political affiliation, protected veteran status, race, religion, sex (including pregnancy), sexual orientation, or any other characteristic protected by applicable laws, regulations and ordinances. ID.me adheres to these principles in all aspects of employment, including recruitment, hiring, training, compensation, promotion, benefits, social and recreational programs, and discipline. In addition, ID.me's policy is to provide reasonable accommodation to qualified employees who have protected disabilities to the extent required by applicable laws, regulations and ordinances where a particular employee works. Upon request we will provide you with more information about such accommodations. Please review our Privacy Policy, including our CCPA policy, at id.me/privacy. If you provide ID.me with any personally identifiable information you confirm that you have read and agree to be bound by the terms and conditions set out in our Privacy Policy. ID.me participates in E-Verify.