Lead AI Evaluation Engineer — Agent Quality

FurtherAI Inc.; San Francisco (CA); On-site; USD 140,000 - 210,000; Full time

Benefits offered by this job

Fully covered health, dental, and visa

Competitive compensation, stock + 401k

Unlimited PTO

Outstanding in-office culture in San F

Lunch and dinner onsite

Team events

Pre-tax commuter benefits

Job summary

FurtherAI Inc. is seeking an exceptional AI Engineer to own how we evaluate and continuously improve the quality of AI agents across our insurance-focused platform.

You will build evaluation frameworks, golden datasets, and regression suites, collaborating with Forward Deployed Engineers and domain experts to capture real-world insurance tasks and edge cases. You will design multi-step agent evaluations, tool use, and end-to-end task completion while developing automated grading systems.

Qualifications

• Strong Python engineering skills and the ability to build reliable, production-grade AI/ML systems.
• Experience building evaluation frameworks, LLM-based applications, model benchmarking systems, or applied AI infrastructure.
• Strong understanding of modern LLMs, agent architectures, evaluation methodologies, and model performance tradeoffs.

Responsibilities

• Own evaluations across FurtherAI: build evaluation frameworks and infrastructure to measure accuracy, quality, and reliability of AI agents across workflows.
• Build high-quality datasets: develop golden datasets, benchmarks, and regression suites with domain experts to capture real-world insurance tasks.
• Evaluate complex agent behavior: design evaluations for multi-step agent trajectories, tool use, reasoning, and end-to-end task completion.
• Build automated grading systems: develop LLM-as-judge evaluations, deterministic graders, and human evaluation workflows.
• Make development evaluation-driven: integrate offline and online evaluations into release processes to measure improvements and catch regressions.
• Drive model evaluation and selection: benchmark frontier/open-weight models across accuracy, latency, cost, and reliability.
• Build continuous improvement loops: turn production failures and user feedback into new evaluation cases and improvements.

Skills

Python

Production AI/ML systems

LLM-based applications

Evaluation frameworks

Communication

Tools

Benchmarking systems

Open-weight models

Job description

FurtherAI Inc. is seeking an exceptional AI Engineer to own how we evaluate and continuously improve the quality of AI agents across our insurance-focused platform.

You will build evaluation frameworks, golden datasets, and regression suites, collaborating with Forward Deployed Engineers and domain experts to capture real-world insurance tasks and edge cases. You will design multi-step agent evaluations, tool use, and end-to-end task completion while developing automated grading systems.