GenAI Evaluations Lead: Benchmarking & Failure Diagnostics

Scale AI; United States; On-site; USD 181,000 - 226,000; Full time

Job summary

Scale AI is seeking Research Scientists and Research Engineers with expertise in LLM post-training and evaluation to join the GenAI Research Organization. The role focuses on building benchmarks, diagnosing failure modes, and evaluating models across text and multimodal modalities.

You will develop rigorous evaluation methods, collaborate with researchers and engineers, and translate failure analysis into actionable input for next-gen generative models.

Qualifications

• Develop rigorous evaluations and diagnostic methods for frontier models.
• Design benchmarks and evaluation methods for text and multimodal modalities.
• Apply post-training techniques (SFT, RLHF, reward modeling) to address model failures.

Responsibilities

• Analyze model behavior to identify failure modes in LLMs and Agents.
• Design and build benchmarks and evaluation methods for text and multimodal tasks.
• Apply post-training techniques to connect failures to data and interventions.
• Publish research findings in top-tier AI conferences.

Skills

LLM evaluation

Benchmark design

Post-training methods

Communication skills

Education

Ph.D. or Master's in CS/ML/AI

Job description

Scale AI is seeking Research Scientists and Research Engineers with expertise in LLM post-training and evaluation to join the GenAI Research Organization. The role focuses on building benchmarks, diagnosing failure modes, and evaluating models across text and multimodal modalities.

You will develop rigorous evaluation methods, collaborate with researchers and engineers, and translate failure analysis into actionable input for next-gen generative models.