LLM Evaluation Scientist: Benchmarks & Failure Insights

AI Chopping Block; New York, Northern (NY, KY); Hybrid; USD 181,000 - 226,000; Full time

Benefits offered by this job

Health coverage

Dental coverage

Vision coverage

Retirement benefits

Learning stipend

Generous PTO

Commuter stipend

Job summary

Scale is seeking Research Scientists and Research Engineers in New York to advance evaluation-driven AI development for GenAI models. You will analyze model behavior, design benchmarks, and apply post-training techniques to improve performance across text and multimodal modalities.

Join a team that collaborates with world-class labs to translate failure analysis into actionable feedback and publish findings at top conferences. Compensation includes base salary, equity, and comprehensive benefits.

Qualifications

• PhD or Master’s degree in CS/ML or related field.
• Deep understanding of deep learning, RL, and fine-tuning.
• Experience with RLHF, preference modeling, or instruction tuning.

Responsibilities

• Analyze model behavior to diagnose failure modes in frontier LLMs and Agents.
• Design benchmarks and evaluation methods for text and multimodal modalities.
• Apply post-training techniques to link failures to data and training interventions.
• Publish research findings in top AI conferences.

Skills

LLM evaluation

SFT

RLHF

Reward modeling

Education

PhD or MS

Job description

Scale is seeking Research Scientists and Research Engineers in New York to advance evaluation-driven AI development for GenAI models. You will analyze model behavior, design benchmarks, and apply post-training techniques to improve performance across text and multimodal modalities.

Join a team that collaborates with world-class labs to translate failure analysis into actionable feedback and publish findings at top conferences. Compensation includes base salary, equity, and comprehensive benefits.