Senior AI Evaluation Scientist - Model Behavior & Metrics

techire ai; San Francisco (CA); Hybrid; USD 200,000 - 350,000; Full time

Job summary

Techire AI is seeking a Senior Research, Evals to rethink how model performance is measured across real-world interactions. You will develop evaluation frameworks that go beyond static benchmarks and build systems that integrate evaluations into model development.

The role is hands-on, bridging model development and evaluation, with responsibilities to design studies, pipelines and metrics for reasoning, memory, and interaction quality.

Qualifications

• Experience building evaluation frameworks for generative models across text, audio or multimodal AI.
• Strong technical and analytical skills.
• Experience turning open-ended research questions into working evaluation systems.
• A good understanding of metrics, experimentation and statistical analysis.
• An interest in measuring subjective qualities such as naturalness, adaptability and interaction quality.

Responsibilities

• Develop new evaluations for reasoning, memory, interaction and model behaviour.
• Build evaluation pipelines with robust statistical analysis.
• Create quantitative metrics for qualities that can be difficult to measure objectively.
• Integrate evals directly into model training and research workflows.
• Design user studies and behavioural experiments to understand real-world model performance.

Job description

Techire AI is seeking a Senior Research, Evals to rethink how model performance is measured across real-world interactions. You will develop evaluation frameworks that go beyond static benchmarks and build systems that integrate evaluations into model development.

The role is hands-on, bridging model development and evaluation, with responsibilities to design studies, pipelines and metrics for reasoning, memory, and interaction quality.