ML Evaluation Scientist - LLM Benchmarks

Scale AI; Seattle (WA); Presencial; USD 181.000 - 226.000; Jornada completa

Ventajas ofrecidas por este puesto de trabajo

Health coverage

Retirement benefits

L&D stipend

Generous PTO

Commuter stipend

Descripción de la vacante

Scale AI is hiring for roles focused on evaluating and benchmarking frontier LLMs and Agents within the GenAI Research Organization. You will develop robust evaluations and RCA-driven insights, collaborating with researchers to shape evaluation-driven AI development and translate failure analyses into strategic input for next-gen models.

The role emphasizes post-training techniques like SFT and RLHF, publication of findings at major AI conferences, and building benchmarks for text and multimodal

Formación

• Ph.D. or Master's degree in Computer Science, Machine Learning, AI, or a related field.
• Deep understanding of deep learning, reinforcement learning, and large-scale model fine-tuning.
• Experience with post-training techniques such as RLHF, preference modeling, or instruction tuning, and with LLM evaluation or benchmark development.
• Excellent written and verbal communication skills.
• Published research in areas of machine learning at major conferences (NeurIPS, ICML, ICLR, ACL, EMNLP, CVPR, etc.) and/or journals.
• Previous experience in a customer facing role.

Responsabilidades

• Analyze model behavior to identify failure modes in frontier LLMs and Agents.
• Design benchmarks and evaluation methods for text and multimodal modalities.
• Apply post-training techniques to connect observed failures to data and training interventions.
• Publish research findings in top-tier AI conferences.

Conocimientos

LLM evaluation

RLHF

SFT

Reward modeling

Diagnostics

Educación

PhD or MSc in CS/ML

Herramientas

Python

PyTorch

Benchmark tools

Descripción del empleo

Scale AI is hiring for roles focused on evaluating and benchmarking frontier LLMs and Agents within the GenAI Research Organization. You will develop robust evaluations and RCA-driven insights, collaborating with researchers to shape evaluation-driven AI development and translate failure analyses into strategic input for next-gen models.

The role emphasizes post-training techniques like SFT and RLHF, publication of findings at major AI conferences, and building benchmarks for text and multimodal