GenAI Evaluation Scientist - LLM Benchmarks & Failures

Scale; San Francisco, Seattle, New York (CA, WA, NY); On-site; USD 166,000 - 207,000; Full time

Benefits offered by this job

Health insurance

Dental coverage

Vision coverage

Retirement benefits

PTO

Job summary

Scale is seeking a Machine Learning Research Scientist, Evaluations to join the GenAI Research Organization in San Francisco. You will develop rigorous evaluations, diagnose failure modes in frontier LLMs and agents, and design benchmarks for text and multimodal modalities.

Collaboration with researchers and engineers will shape evaluation-driven AI development. The role emphasizes post-training techniques like SFT and RLHF, with opportunities to publish findings at top conferences and influence

Qualifications

• Ph.D. or Master's degree in Computer Science, Machine Learning, AI, or a related field.
• Deep understanding of deep learning, reinforcement learning, and large-scale model fine-tuning.
• Experience with post-training techniques such as RLHF, preference modeling, or instruction tuning, and with LLM evaluation or benchmark development.
• Excellent written and verbal communication skills.
• Published research in areas of machine learning at major conferences (NeurIPS, ICML, ICLR, ACL, EMNLP, CVPR, etc.) and/or journals.
• Previous experience in a customer facing role.

Responsibilities

• Analyze model behavior to identify, characterize, and diagnose failure modes in frontier LLMs and Agents.
• Design and build benchmarks and evaluation methods that measure LLM capabilities in both text and multimodal modalities.
• Apply post-training expertise (SFT, RLHF, reward modeling) to connect observed failures to the data and training interventions that address them.
• Publish research findings in top-tier AI conferences.

Skills

LLM evaluation

Benchmark design

Research publication

Communication skills

Education

PhD or Master's in CS/ML

Tools

PyTorch

TensorFlow

Job description

Scale is seeking a Machine Learning Research Scientist, Evaluations to join the GenAI Research Organization in San Francisco. You will develop rigorous evaluations, diagnose failure modes in frontier LLMs and agents, and design benchmarks for text and multimodal modalities.

Collaboration with researchers and engineers will shape evaluation-driven AI development. The role emphasizes post-training techniques like SFT and RLHF, with opportunities to publish findings at top conferences and influence