Senior LLM Evaluation Scientist

Cohere; City Of London; Remote; GBP 70,000 - 120,000; Full time

Benefits offered by this job

Lunch stipend

Health benefits

RRSP matching / Pension

Parental leave top-up

Enrichment benefits

Vacation 6 weeks

Travel to other offices

Home office stipend

Job summary

Cohere is seeking an Evaluation Researcher to design and implement next‑generation evaluation methods for LLMs. You will build benchmarks, collaborate with product and engineering teams, and advance research on LLM evaluation techniques and data synthesis pipelines.

The role involves creating scalable tools and methods to assess model progress, with a focus on ensuring data quality and reliable measurement of AI capabilities. Remote-friendly with office options in several cities.

Qualifications

• Ability to rapidly build prototypes that demonstrate LLM boundaries.
• Hours spent reviewing complex data and LLM outputs to ensure data quality.
• Rigorous measurement of AI capabilities and alignment with desired KPIs.
• Strong software engineering skills and code quality discipline.

Responsibilities

• Create ambitious new evaluation benchmarks for LLMs.
• Collaborate with cross-functional teams to translate feedback into evaluations.
• Conduct research to advance state-of-the-art evaluation methods for LLMs.
• Build scalable, reusable tools to analyze model performance.

Skills

Prototype development

Data quality review

Rigorous measurement

Software engineering

Job description

Cohere is seeking an Evaluation Researcher to design and implement next‑generation evaluation methods for LLMs. You will build benchmarks, collaborate with product and engineering teams, and advance research on LLM evaluation techniques and data synthesis pipelines.

The role involves creating scalable tools and methods to assess model progress, with a focus on ensuring data quality and reliable measurement of AI capabilities. Remote-friendly with office options in several cities.