AI Engineer, Evals & Agent Quality

Town.com, Inc.; San Francisco (CA); On-site; USD 250,000 - 300,000; Full time

Job summary

You will run validation checks to confirm improvements, build model routing and online evaluation tooling, and ensure every prompt change is measurable. This role collaborates with product engineers to instrument quality and drive fixes.

Qualifications

• Experience building or owning LLM eval systems.
• Strong, rigorous approach to measurement design.
• Hands-on familiarity with the eval landscape and tooling.
• Comfort reasoning about model routing and tradeoffs.

Responsibilities

• Design and implement a generalized eval system to measure assistant quality across all user touchpoints.
• Ensure evaluation coverage includes multi-step agent trajectories, not just single-turn outputs.
• Create and maintain golden datasets with a labeling loop to keep data current.
• Run continuous validation checks to confirm improvements and avoid regressions.
• Build model routing capabilities and online evaluation tooling to learn which models perform best.
• Make every prompt and system change measurable to enable rapid iteration without breaking behavior.
• Collaborate with product engineers to instrument quality and close the loop from signals to fixes.

Skills

LLM eval systems

Measurement design

Evaluation tooling

Dashboards & metrics

Tools

Eval frameworks

Job description

Town.com is building an AI assistant, and this role will own evaluation and agent quality end to end, from measurement to routing and regression prevention.

Responsibilities

• Design and implement a generalized eval system to measure assistant quality across all user touchpoints
• Ensure evaluation coverage includes multi-step agent trajectories, not just single-turn outputs
• Create and maintain golden datasets along with a labeling loop to keep data current
• Run continuous validation checks to confirm improvements and avoid regressions
• Build model routing capabilities and supporting online evaluation tooling to learn which models perform best
• Make every prompt and system change measurable so the team can iterate quickly without breaking working behavior
• Collaborate with product engineers to instrument quality and close the loop from quality signals to implementation fixes

Requirements

• Experience building or owning LLM eval systems, or delivering offline/online quality measurement at scale
• Strong, rigorous approach to measurement design
• Hands-on familiarity with the eval landscape, including off-the-shelf tooling and eval frameworks, plus the ability to choose approaches based on judgment
• Comfort reasoning about model routing and tradeoffs between models
• Track record of shipping fixes in addition to producing dashboards and metrics
• Senior or staff-level capability operating in a greenfield environment where the evaluation system does not yet exist

Location

• San Francisco, CA (onsite)
• Five days a week in person at the Financial District office

Compensation

• USD 250,000 - 300,000 per yearly