GenAI Evaluation Scientist: LLM Benchmarks & Failures

Scale AI; New York (NY); On-site; USD 140,000 - 210,000; Full time

Benefits offered by this job

Health coverage

PTO and flexible schedules

Learning & development stipend

Career growth opportunities

Job summary

Scale AI in New York is seeking Research Scientists and Research Engineers to advance evaluation for state-of-the-art LLMs and multimodal models. You will design benchmarks, diagnose failure modes, and publish findings from rigorous experiments.

You will collaborate with researchers and engineers across teams, translate failure analysis into actionable data and training guidance, and contribute to the next generation of scalable AI systems at Scale AI.

Qualifications

• Experience with post-training techniques such as RLHF, preference modeling, or instruction tuning.
• Published research at major conferences or journals.
• Excellent written and verbal communication skills.
• Ph.D. or Master’s in Computer Science, ML, AI, or related field.
• Experience in customer-facing roles is a plus.
• Deep understanding of deep learning, reinforcement learning, and large-scale model fine-tuning.

Responsibilities

• Design and build benchmarks and evaluation methods for LLMs and multimodal models.
• Analyze model behavior to diagnose failure modes and RCA in frontier models.
• Collaborate with researchers and engineers to define evaluation-driven AI development.
• Publish research findings at top-tier AI conferences.
• Translate failure analysis into training interventions and data improvements.

Skills

RLHF

Preference modeling

Instruction tuning

LLM evaluation

Benchmark development

Written and verbal communication

Education

Ph.D. or MS in CS/ML/AI

Job description

Scale AI in New York is seeking Research Scientists and Research Engineers to advance evaluation for state-of-the-art LLMs and multimodal models. You will design benchmarks, diagnose failure modes, and publish findings from rigorous experiments.

You will collaborate with researchers and engineers across teams, translate failure analysis into actionable data and training guidance, and contribute to the next generation of scalable AI systems at Scale AI.