GenAI Evaluation Scientist: Benchmarks & Failures

Scale AI, Inc.; San Francisco (CA); On-site; USD 181,000 - 226,000; Full time

Benefits offered by this job

Health benefits

Retirement benefits

Learning stipend

Generous PTO

Commuter stipend

Job summary

Scale AI, Inc. seeks Research Scientists and Research Engineers with expertise in LLM post-training (SFT, RLHF, reward modeling) and evaluation.

The role focuses on building benchmarks and diagnosing model failure modes in text and multimodal modalities within the GenAI Research Organization. You will develop rigorous evaluations, collaborate with researchers and engineers, and translate failure analysis into input for next-generation generative AI models.

Qualifications

• PhD or master's in CS, ML, AI, or related field.
• Deep understanding of deep learning and reinforcement learning.
• Experience with RLHF, preference modeling, or instruction tuning, and LLM evaluation or benchmark development.
• Published research at major conferences (NeurIPS, ICML, ICLR, ACL, EMNLP, CVPR) and/or journals.
• Excellent written and verbal communication skills.
• Previous experience in a customer-facing role.

Responsibilities

• Analyze model behavior to identify, characterize, and diagnose failure modes in frontier LLMs and Agents.
• Design and build benchmarks and evaluation methods for text and multimodal modalities.
• Apply post-training techniques (SFT, RLHF, reward modeling) to connect failures to data and training interventions.
• Publish research findings in top-tier AI conferences.

Skills

Deep learning

Reinforcement learning

LLM evaluation

Education

PhD or MS in CS/ML/AI

Job description

Scale AI, Inc. seeks Research Scientists and Research Engineers with expertise in LLM post-training (SFT, RLHF, reward modeling) and evaluation.

The role focuses on building benchmarks and diagnosing model failure modes in text and multimodal modalities within the GenAI Research Organization. You will develop rigorous evaluations, collaborate with researchers and engineers, and translate failure analysis into input for next-generation generative AI models.