Senior Researcher, Evals

techire ai; San Francisco (CA); Hybrid; USD 200,000 - 350,000; Full time

Job summary

Techire AI is seeking a Senior Research, Evals to rethink how model performance is measured across real-world interactions. You will develop evaluation frameworks that go beyond static benchmarks and build systems that integrate evaluations into model development.

The role is hands-on, bridging model development and evaluation, with responsibilities to design studies, pipelines and metrics for reasoning, memory, and interaction quality.

Qualifications

• Experience building evaluation frameworks for generative models across text, audio or multimodal AI.
• Strong technical and analytical skills.
• Experience turning open-ended research questions into working evaluation systems.
• A good understanding of metrics, experimentation and statistical analysis.
• An interest in measuring subjective qualities such as naturalness, adaptability and interaction quality.

Responsibilities

• Develop new evaluations for reasoning, memory, interaction and model behaviour.
• Build evaluation pipelines with robust statistical analysis.
• Create quantitative metrics for qualities that can be difficult to measure objectively.
• Integrate evals directly into model training and research workflows.
• Design user studies and behavioural experiments to understand real-world model performance.

Job description

How do you actually evaluate an AI model when accuracy on a benchmark only tells part of the story?

A well-funded AI company developing next-generation foundation models is looking for a Senior Research, Evals to rethink how model performance is measured across real-world interactions.

The role

You’ll work on evaluation frameworks that go beyond static benchmarks, looking at how models reason, remember, adapt and interact over time.

This is a hands‑on research role sitting between model development and evaluation. You’ll help define what should be measured, work out how to measure it, then build the systems that bring those evaluations directly into model development.

What you’ll do

• Develop new evaluations for reasoning, memory, interaction and model behaviour
• Build evaluation pipelines with robust statistical analysis
• Create quantitative metrics for qualities that can be difficult to measure objectively
• Integrate evals directly into model training and research workflows
• Design user studies and behavioural experiments to understand real-world model performance

What you’ll bring

• Experience building evaluation frameworks for generative models across text, audio or multimodal AI
• Strong technical and analytical skills
• Experience turning open-ended research questions into working evaluation systems
• A good understanding of metrics, experimentation and statistical analysis
• An interest in measuring subjective qualities such as naturalness, adaptability and interaction quality

Experience with alignment or model behaviour research would be useful, but isn’t essential.

You’ll work closely with researchers building new foundation models, helping understand whether changes are creating meaningful improvements rather than simply moving benchmark scores.

Base salary is $200k-$350k DOE + generous equity.

Based in San Francisco, New York or London, working hybrid.