Senior AI Eval & Quality Engineer — Onsite SF

Town; San Francisco (CA); On-site; USD 180,000 - 240,000; Full time

Job summary

Town is building the most personalized AI assistant that knows you deeply and works across every tool you use. You will own the evals and quality systems that turn assistant performance into numbers the whole team can trust, spanning multi-step trajectories and model routing to balance cost, quality, and speed.

This foundational 0→1 build requires owning the eval framework, golden datasets and labeling loop, and online measurement, setting the bar for what “best” means at Town.

Qualifications

• Built or owned LLM eval systems, or offline/online quality measurement at scale.
• Think rigorously about measurement and what defines “better”.
• Familiar with eval frameworks/tools and their tradeoffs.
• Able to reason about routing models to balance cost, quality, and speed.
• Deliver fixes along with dashboards and metrics.

Responsibilities

• Build the evals and quality systems to quantify assistant performance.
• Create golden datasets and labeling loops to keep evals current.
• Develop model routing and online evaluation tooling to route to best models.
• Make prompts and system changes measurable to accelerate iteration.
• Collaborate with product engineers to instrument quality and close the loop from signal to fix.

Skills

LLM eval systems

Model routing

Online evaluation

Measurement

Greenfield development

Job description

Town is building the most personalized AI assistant that knows you deeply and works across every tool you use. You will own the evals and quality systems that turn assistant performance into numbers the whole team can trust, spanning multi-step trajectories and model routing to balance cost, quality, and speed.

This foundational 0→1 build requires owning the eval framework, golden datasets and labeling loop, and online measurement, setting the bar for what “best” means at Town.