GenAI Evaluation Scientist: LLM Benchmarks & Diagnostics

Neura Market; San Francisco, Northern (CA, KY); Hybrid; USD 181,000 - 226,000; Full time

Benefits offered by this job

Health coverage

Retirement benefits

Learning stipend

Generous PTO

Commuter stipend

Job summary

Scale is seeking Research Scientists and Research Engineers in San Francisco to build rigorous evaluations and benchmarks for LLMs and multimodal models. You will diagnose failure modes, design evaluation methods, and apply post-training techniques to guide data interventions.

You will publish findings with top researchers and partner with leading labs to shape next-gen generative AI. The role emphasizes collaboration with researchers and engineers, cross-functional communication, and

Qualifications

• PhD or Master's in CS/ML/AI or related field.
• Deep understanding of deep learning, RL, and large-scale model fine-tuning.
• Experience with RLHF, reward modeling, instruction tuning, or LLM evaluation.
• Strong written and verbal communication skills.
• Published research at NeurIPS, ICML, ICLR, ACL, EMNLP, CVPR, or similar.
• Experience in customer-facing roles.

Responsibilities

• Analyze model behavior to identify failure modes in frontier LLMs and Agents.
• Design benchmarks and evaluation methods for text and multimodal outputs.
• Apply post-training techniques (SFT, RLHF, reward modeling) to connect failures to data interventions.
• Publish research findings at top AI conferences.

Skills

Deep learning

Reinforcement learning

LLM evaluation

Communication skills

Research publication experience

Customer facing experience

Education

PhD or Master's degree in CS/ML/AI

Job description

Scale is seeking Research Scientists and Research Engineers in San Francisco to build rigorous evaluations and benchmarks for LLMs and multimodal models. You will diagnose failure modes, design evaluation methods, and apply post-training techniques to guide data interventions.

You will publish findings with top researchers and partner with leading labs to shape next-gen generative AI. The role emphasizes collaboration with researchers and engineers, cross-functional communication, and