Senior LLM Evaluation Scientist

Cohere; Greater London; Remote; GBP 90,000 - 130,000; Full time

Benefits offered by this job

Lunch stipend

Health and dental benefits

RRSP matching / Pension

Parental Leave top-up

Enrichment benefits

6 weeks of paid vacation

Travel budget to other offices

Home office stipend

Office lunch program

Job summary

Cohere is seeking an experienced role focused on creating next-generation evaluation methods and infrastructure to measure LLM progress. You will work across teams to translate model feedback into trustworthy evaluations and build scalable tools to analyze performance.

The role emphasizes rigorous measurement of AI capabilities, prototype-driven work, and strong software engineering to support scalable evaluation pipelines. Remote-friendly with global offices.

Qualifications

• Rapidly build prototypes to demonstrate LLM boundaries and capabilities.
• Spend dozens of hours reviewing complex data and LLM outputs for data quality.
• Rigorously measure AI capabilities and align measurements with relevant capabilities.
• Strong software engineering skills to implement evaluation systems and tools.

Responsibilities

• Create ambitious new evaluation benchmarks for LLMs.
• Collaborate with cross-functional teams to translate model feedback into evaluations.
• Conduct research to advance state-of-the-art in LLM evaluation methods, including data synthesis.
• Build scalable and reusable tools for analyzing model performance.

Skills

Prototype development

Data review

Measurement rigor

Software engineering

Job description

Cohere is seeking an experienced role focused on creating next-generation evaluation methods and infrastructure to measure LLM progress. You will work across teams to translate model feedback into trustworthy evaluations and build scalable tools to analyze performance.

The role emphasizes rigorous measurement of AI capabilities, prototype-driven work, and strong software engineering to support scalable evaluation pipelines. Remote-friendly with global offices.