GenAI Evaluation Scientist — Benchmark & Diagnose LLMs

Scale AI, Inc.; New York (NY); Vor Ort; USD 181.000 - 226.000; Vollzeit

Benefits dieser Stelle

Base salary + equity

Health, dental, vision coverage

Retirement benefits

Learning & development stipend

Generous PTO

Commuter stipend

Zusammenfassung

Scale AI, Inc. is recruiting for Research Scientists and Research Engineers specializing in LLM post-training evaluation and benchmark development.

This role focuses on rigorously evaluating frontier models, diagnosing failure modes, and building robust benchmarks for text and multimodal modalities. You will collaborate with researchers and engineers to define best practices in evaluation-driven AI development and translate failure analyses into strategic input for the next generation of

Qualifikationen

• Ph.D. or Master's degree in Computer Science, Machine Learning, AI, or a related field.
• Deep understanding of deep learning, reinforcement learning, and large-scale model fine-tuning.
• Experience with post-training techniques such as RLHF, preference modeling, or instruction tuning, and with LLM evaluation or benchmark development.
• Excellent written and verbal communication skills.
• Published research in machine learning at major conferences and/or journals.
• Previous experience in a customer facing role.

Aufgaben

• Analyze model behavior to identify, characterize, and diagnose failure modes in frontier LLMs and Agents.
• Design and build benchmarks and evaluation methods that measure LLM capabilities in text and multimodal modalities.
• Apply post-training techniques to connect observed failures to data and training interventions.
• Publish research findings in top-tier AI conferences.

Kenntnisse

LLM evaluation

SFT RLHF

benchmark development

communication skills

published research

customer facing experience

Ausbildung

Ph.D. or Master's in CS/ML/AI

Jobbeschreibung

Scale AI, Inc. is recruiting for Research Scientists and Research Engineers specializing in LLM post-training evaluation and benchmark development.

This role focuses on rigorously evaluating frontier models, diagnosing failure modes, and building robust benchmarks for text and multimodal modalities. You will collaborate with researchers and engineers to define best practices in evaluation-driven AI development and translate failure analyses into strategic input for the next generation of