Senior AI Model Evaluation Scientist (LLM Benchmarks)

Cohere; Seattle (WA); Remote; USD 180,000 - 385,000; Full time

Benefits offered by this job

Lunch stipend

Health and dental benefits

RRSP matching / 401K / Pension

Parental leave

Enrichment benefits

Education stipend

Vacation days

Home office stipend

Job summary

Cohere is seeking a role focused on evaluating next-generation AI models, developing evaluation benchmarks and infrastructure to measure LLM progress. You will build ambitious benchmarks, work with cross-functional teams to translate model feedback into trustworthy evaluations, and advance state-of-the-art methods.

This remote-friendly role offers health and dental benefits, a lunch stipend, parental leave and education stipends, plus opportunities to work across our global offices and tools

Qualifications

• Experience building evaluation benchmarks for ML models.
• Ability to review complex data and model outputs.
• Strong software engineering fundamentals.

Responsibilities

• Create ambitious evaluation benchmarks for models.
• Collaborate with cross-functional teams to translate feedback into evaluations.
• Research LLM evaluation methods and improve efficiency.
• Build scalable tools for model performance analysis.

Skills

Prototype development

Data review

Rigorous measurement

Software engineering

Job description

Cohere is seeking a role focused on evaluating next-generation AI models, developing evaluation benchmarks and infrastructure to measure LLM progress. You will build ambitious benchmarks, work with cross-functional teams to translate model feedback into trustworthy evaluations, and advance state-of-the-art methods.

This remote-friendly role offers health and dental benefits, a lunch stipend, parental leave and education stipends, plus opportunities to work across our global offices and tools