Research Engineer - Data and Evaluation

Ladders

United States

Remote

USD 150,000 - 350,000

Full time

35 hours ago
Be an early applicant

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Benefits offered by this job

Remote work
SF in-person option
Visa sponsorship
Relocation assistance
Quarterly team off-sites
Hackathons
Equity incentives
Professional development

Job summary

Ladders is seeking a Research Engineer - Data and Evaluation to join a leader in IT research. The role focuses on designing and scaling data and evaluation systems to measure model performance in complex coding environments.

You will build executable task environments, robust scoring, and investigate failure modes, collaborating with RL researchers to improve data quality and results. Remote position in the United States with compensation ranging from $150,000 to $350,000 annually.

Qualifications

  • Strong Python skills for building data pipelines and evaluation systems.
  • Hands-on experience with language models or agents and their training data.
  • Ability to analyze data and model outputs to identify failure modes.
  • Proven track record of managing technical projects from experimentation to validation.
  • Experience building evaluations or agent environments with debugging and scoring code.

Responsibilities

  • Design and create challenging evaluations for coding and long-horizon agents.
  • Develop executable environments with realistic task setups and clear success criteria.
  • Create and test scoring logic and verifiers, addressing reward hacks and misleading progress measures.
  • Trace model failures through outputs and code to separate model limits from environment issues.
  • Build and scale RL data pipelines that generate and curate complex coding tasks.
  • Collaborate with RL researchers to assess the impact of task and training data changes.

Skills

Python
Data pipelines
Evaluation systems
Language models / agents
Experimentation to validation
Collaboration

Job description

For our client, we are seeking a Research Engineer - Data and Evaluation to join the team of a leader in the Information Technology space.

This role focuses on designing and scaling the data and evaluation systems that help researchers measure model performance in complex coding and agentic environments. You will build executable task setups, create robust scoring logic, and investigate failure modes to distinguish model limitations from environment issues. The work is highly collaborative and will partner closely with RL researchers and engineers to improve training data quality, validation methods, and downstream results.

Location: Remote

Compensation: $150,000 - $350,000 annually

Responsibilities
  • Design and create challenging evaluations for coding and long-horizon agents.
  • Develop executable environments with realistic task setups and clear success criteria.
  • Create and test scoring logic and verifiers, addressing reward hacks and misleading progress measures.
  • Trace model failures through outputs and code to differentiate between model limitations and environmental issues.
  • Build and scale RL data pipelines that generate and curate complex coding and agentic tasks.
  • Collaborate with RL researchers to assess the impact of task and training data modifications on performance.
Qualifications
  • Strong software engineering skills in Python, particularly in building data pipelines and evaluation systems.
  • Hands-on experience with language models or agents, including understanding their training data and evaluation impacts.
  • Ability to analyze complex data and model outputs to identify failure modes and create testable measurements.
  • Proven track record of managing technical projects from experimentation to validation, collaborating with researchers and engineers.
  • Experience building technical evaluations or agent environments, with skills in debugging model traces and scoring code.
Benefits
  • Flexible work arrangements, including remote or in-person options in San Francisco.
  • Visa sponsorship and relocation assistance for international candidates.
  • Quarterly team off-sites, hackathons, and opportunities for professional development.
  • Equity incentives aligned with the company's growth and impact.
  • Work alongside a talented, mission-driven team passionate about advancing AI technology.

Our client is an equal opportunity employer. We encourage you to apply even if you don't meet every qualification-your background could be exactly what this team needs.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Research Engineer - Data and Evaluation (Fully Remote)
Research Engineer - Data and Evaluation (Fully Remote)

Prime Intellect, Inc. • United States

Remote
USD 150,000 - 350,000
Quarterly off-sites
Hackathons
Conferences
+1
Research Engineer – RL Infrastructure & Agent Environments
Research Engineer – RL Infrastructure & Agent Environments

MaxIT Consulting - Max Corporate Group • San Francisco (CA)

On-site
USD 150,000 - 190,000
Evals Research Engineer – LLM Agents & RL Environments
Evals Research Engineer – LLM Agents & RL Environments

MaxIT Consulting - Max Corporate Group • San Francisco (CA)

On-site
USD 150,000 - 190,000
Remote Research Engineer - Data & Evaluation for RL
Remote Research Engineer - Data & Evaluation for RL

Ladders • United States

Remote
USD 150,000 - 350,000
Remote work
SF in-person option
Visa sponsorship
+5
Member of Technical Staff, Coding Agent Research & Evaluations
Member of Technical Staff, Coding Agent Research & Evaluations

Confidential • United States

Remote
USD 200,000 - 260,000
Equity compensation
Performance-based bonuses
Health-insurance premium reimbursement
+2
Software Engineer, RL Environments
Software Engineer, RL Environments

True Talent Solutions Partners • San Francisco (CA)

On-site
USD 260,000 - 290,000
AI Evals Engineer – Reward Design & Agent Environments
AI Evals Engineer – Reward Design & Agent Environments

MaxIT Consulting - Max Corporate Group • San Francisco (CA)

On-site
USD 120,000 - 180,000
Member of Technical Staff – RL Environments & Evals
Member of Technical Staff – RL Environments & Evals

MaxIT Consulting - Max Corporate Group • San Francisco (CA)

On-site
USD 170,000 - 250,000
AI Engineer, RL & Evals
AI Engineer, RL & Evals

Raydar • San Francisco (CA)

On-site
USD 175,000 - 275,000
Competitive equity
Member of Technical Staff, Research Engineering
Member of Technical Staff, Research Engineering

Confidential • United States

Remote
USD 200,000 - 300,000
Equity compensation
Performance-based bonuses
Health insurance reimbursement
+2