Frontier AI Data & Evaluation Scientist

idler; San Francisco (CA); On-site; USD 150,000 - 210,000; Full time

Benefits offered by this job

Equity

Free meals in office

Healthcare

401(k)

15 days PTO

Relocation assistance

Job summary

idler is a frontier data research lab building the evals and environments that frontier labs use to measure and train models. As a Research Scientist at idler, you’ll own measuring how models learn from our tasks, validate data quality with researchers, and create new datasets.

This role is based in San Francisco, with collaboration across teams on post-training recipes, scalable data ingestion, and data refinement.

Qualifications

• 1+ years of experience doing RL in production at a frontier lab.
• Track record of post-training an LLM end to end
• Desire to drive the research roadmap and implementation on a fast-moving team

Responsibilities

• Design novel post-training recipes and data quality measurement techniques.
• Design and run our in-house post-training stack to measure model lift on our tasks.
• Develop new data products based on datasets and experts available to us.
• Create scaleable systems for ingesting & evaluating data we are considering buying

Skills

RL in production

Post-training LLM

Data quality

Dataset design

Collaboration

Frontier lab experience

Curiosity

Tools

Typescript

React

NodeJS

Postgres

Redis

Vercel

Cursor

Job description

idler is a frontier data research lab building the evals and environments that frontier labs use to measure and train models. As a Research Scientist at idler, you’ll own measuring how models learn from our tasks, validate data quality with researchers, and create new datasets.

This role is based in San Francisco, with collaboration across teams on post-training recipes, scalable data ingestion, and data refinement.