Senior AI Eval & Agent-Quality Engineer

Town.com; San Francisco (CA); On-site; USD 180,000 - 250,000; Full time

Job summary

Town is building the most personalized, capable AI assistant for everyone and is looking for a senior/staff engineer to own the eval framework and quality systems. You will design and implement evaluation infrastructure, establish golden datasets, and create routing logic to balance model cost, quality, and speed.

This is foundational, 0-1 work with broad ownership. You will collaborate with product engineers to instrument quality signals, iterate on prompts and systems, and ship measurable

Qualifications

• Built or owned LLM eval systems.
• Rigorous measurement mindset for ML systems.
• Familiar with eval frameworks and tooling.
• Comfortable reasoning about model routing and tradeoffs.
• Experience shipping fixes, not just dashboards.
• Senior or staff engineer in a greenfield setting.

Responsibilities

• Build a generalized eval system that measures assistant quality across surfaces and multi-step trajectories.
• Stand up golden datasets and labeling loops to keep them current and track improvements.
• Build model routing and online evaluation tooling to optimize model selection and cost/quality/speed.
• Make every prompt and system change measurable to enable fast iteration without breaking existing work.
• Partner with engineers across the product to instrument quality and close the loop from signal to fix.

Skills

LLM eval systems

Quality measurement

Model routing

Applied ML background

Job description

Town is building the most personalized, capable AI assistant for everyone and is looking for a senior/staff engineer to own the eval framework and quality systems. You will design and implement evaluation infrastructure, establish golden datasets, and create routing logic to balance model cost, quality, and speed.

This is foundational, 0-1 work with broad ownership. You will collaborate with product engineers to instrument quality signals, iterate on prompts and systems, and ship measurable