Senior AI Evaluation Architect & Model Routing Engineer

Town.com, Inc.; San Francisco (CA); On-site; USD 250,000 - 300,000; Full time

Job summary

You will run validation checks to confirm improvements, build model routing and online evaluation tooling, and ensure every prompt change is measurable. This role collaborates with product engineers to instrument quality and drive fixes.

Qualifications

• Experience building or owning LLM eval systems.
• Strong, rigorous approach to measurement design.
• Hands-on familiarity with the eval landscape and tooling.
• Comfort reasoning about model routing and tradeoffs.

Responsibilities

• Design and implement a generalized eval system to measure assistant quality across all user touchpoints.
• Ensure evaluation coverage includes multi-step agent trajectories, not just single-turn outputs.
• Create and maintain golden datasets with a labeling loop to keep data current.
• Run continuous validation checks to confirm improvements and avoid regressions.
• Build model routing capabilities and online evaluation tooling to learn which models perform best.
• Make every prompt and system change measurable to enable rapid iteration without breaking behavior.
• Collaborate with product engineers to instrument quality and close the loop from signals to fixes.

Skills

LLM eval systems

Measurement design

Evaluation tooling

Dashboards & metrics

Tools

Eval frameworks

Job description

You will run validation checks to confirm improvements, build model routing and online evaluation tooling, and ensure every prompt change is measurable. This role collaborates with product engineers to instrument quality and drive fixes.