GenAI Evaluation Scientist: LLM Benchmarking & Diagnostics

Scale AI, Inc.; New York, Northern (NY, KY); Hybrid; USD 181,000 - 226,000; Full time

Benefits offered by this job

Health, dental and vision coverage

Retirement benefits

Learning and development stipend

Generous PTO

Commuter stipend

Job summary

Scale AI, Inc. invites Research Scientists and Research Engineers to join its GenAI Research Organization in New York. You will work on evaluation for LLMs and agents, building benchmarks for text and multimodal data, and applying post-training methods to improve model performance.

You will collaborate with leading labs, publish findings, and influence next-generation AI systems. The role offers base salary, equity, extensive benefits, PTO, and a commuter stipend with a salary range provided in

Qualifications

• Ph.D. or Master’s degree in CS, ML, AI or related field.
• Deep understanding of deep learning, RL, and large-scale model fine-tuning.
• Experience with post-training techniques (RLHF, preference modeling, instruction tuning) and LLM evaluation or benchmarks.

Responsibilities

• Analyze model behavior to identify and diagnose failure modes in frontier LLMs and Agents (RCA).
• Design benchmarks and evaluation methods for text and multimodal capabilities.
• Apply post-training techniques (SFT, RLHF, reward modeling) to connect failures to data/training interventions.
• Publish research findings in top-tier AI conferences.

Skills

Deep learning

Reinforcement learning

LLM fine-tuning

Education

Ph.D. or Master’s in CS/ML/AI

Job description

Scale AI, Inc. invites Research Scientists and Research Engineers to join its GenAI Research Organization in New York. You will work on evaluation for LLMs and agents, building benchmarks for text and multimodal data, and applying post-training methods to improve model performance.

You will collaborate with leading labs, publish findings, and influence next-generation AI systems. The role offers base salary, equity, extensive benefits, PTO, and a commuter stipend with a salary range provided in