Research Scientist

Idler; San Francisco, Northern (CA, KY); Hybrid; USD 150,000 - 230,000; Full time

Benefits offered by this job

Free meals in office

Healthcare

401(k)

15 days PTO per year

Relocation assistance

Equity

Job summary

Idler, a frontier data research lab in San Francisco, seeks a Research Scientist to measure and improve how models learn from our tasks. You will maximize the learning signal per unit time and validate data quality through production-scale experiments with frontier researchers.

Responsibilities include designing post-training recipes, running our in-house evaluation stack, developing data products, and building scalable data-ingest and evaluation pipelines.

Qualifications

• 1+ year of RL experience in production at a frontier lab.
• Track record of post-training an LLM end-to-end.
• Desire to drive the research roadmap on a fast-moving team.
• Deep curiosity about how machines learn from data and how to maximize learning signals.

Responsibilities

• Design novel post-training recipes and data quality measurement techniques with customers.
• Design and run in-house post-training stack to measure model lift on tasks.
• Develop data products based on datasets and experts available to us.
• Create scalable systems for ingesting and evaluating data.
• Identify opportunities to leverage self-reinforcing feedback loops.

Skills

Reinforcement learning

LLM post-training

Research roadmap

Data quality

Problem solving

Tools

Typescript

React

NodeJS

Postgres

Redis

Job description

About idler

idler is a frontier data research lab. We build the evals and environments that the world's leading frontier labs use to measure and train their models.

After raising a $9m seed round led by Paradigm, we spent the last year developing coding evals for top coding models you know and love. At the same time, we've expanded into other domains besides coding: RSI & Auto-Research, Law, Enterprise Business, Cybersecurity, and others. Now, we are facing more lab demand for our data than we can serve, and are rapidly scaling the team to grow the business.

Our approach to creating training data scales using technology, and all of our data products are built on a unified self-reinforcing platform that learns through experience.

You would be joining a close-knit team that has reached product market fit, and your work would directly help to multiply our revenue.

You can see some of our work here: https://idler.ai/collections

About the role

As a Research Scientist at idler, you'll own measuring and improving how models learn from our tasks. The job is to maximize learning signal we produce per unit time. You'll draw on your own experience and collaborate with researchers at the frontier to validate our data quality, identify where improvements are needed, and create new datasets. To succeed, you'll need to have extensive experience doing this work in production at a frontier lab.

Examples of what you’ll do

• Work with our customers — researchers at frontier labs — to design novel post-training recipes and data quality measurement techniques

• Design and run our in-house post-training stack to measure model lift on our tasks

• Develop new data products based on datasets and experts available to us

• Identify opportunities to take advantage of self-reinforcing exponential feedback loops

• Create agents to analyze thousands of environments and millions of trajectories

• Help curate and specify task distributions for new corpora

• Work with procurement to ensure external data we acquire is suitable for refinement

• Create scaleable systems for ingesting & evaluating data we are considering buying

• Develop new techniques for mining data for signal

What we’re looking for

• 1+ years of experience doing RL in production at a frontier lab

• Track record of post-training an LLM end to end

• Desire to drive the research roadmap and implementation on a fast-moving team

• Deep curiosity about how machines learn from data and how to extract the maximum learning signal from our tasks

Tech stack

Typescript, React, NodeJS, Postgres, Redis, Vercel, Cursor/Claude Code/Codex, Tinker, Modal, AWS, Daytona, GRPO

Details

• In-person in San Francisco

• Competitive salary + meaningful equity

• Free meals in office

• Healthcare, 401(k), 15 days of PTO per year

• Relocation assistance

• Small, ambitious team

This is an in-person role in San Francisco. We're a tight-knit founding team and we play to win. Join us if you like to win too.