Senior AI Evaluation Scientist – Real-World Performance

techire ai; San Francisco (CA); Hybrid; USD 200,000 - 350,000; Full time

Job summary

techire ai in San Francisco, New York or London is seeking a Senior Research, Evals to rethink how model performance is measured across real‑world interactions.

This hands‑on role sits between model development and evaluation. You’ll define what should be measured, how to measure it, then build the systems that bring those evaluations directly into development. Base salary is $200k-$350k DOE + generous equity.

Qualifications

• Experience building evaluation frameworks for generative models across text, audio or multimodal AI.
• Strong technical and analytical skills.
• Experience turning open-ended research questions into working evaluation systems.
• A good understanding of metrics, experimentation and statistical analysis.
• An interest in measuring subjective qualities such as naturalness, adaptability and interaction quality.

Responsibilities

• Develop new evaluations for reasoning, memory, interaction and model behaviour
• Build evaluation pipelines with robust statistical analysis
• Create quantitative metrics for qualities that can be difficult to measure objectively
• Integrate evals directly into model training and research workflows
• Design user studies and behavioural experiments to understand real-world model performance

Skills

Evaluations for generative models

Statistical analysis

Eval systems design

Subjective metrics

Model behaviour research

Job description

techire ai in San Francisco, New York or London is seeking a Senior Research, Evals to rethink how model performance is measured across real‑world interactions.

This hands‑on role sits between model development and evaluation. You’ll define what should be measured, how to measure it, then build the systems that bring those evaluations directly into development. Base salary is $200k-$350k DOE + generous equity.