Founding RL & Evals Engineer - Shape Long-Horizon AI

Runtime Labs, Inc.; San Francisco, Northern (CA, KY); Hybrid; USD 150,000 - 200,000; Full time

Benefits offered by this job

Equity stake

In-person collaboration

Job summary

Runtime Labs, Inc. is seeking a Founding Engineer to own RL and Evals for Óra, shaping how model behavior is measured across planning, memory, tool use, and long-horizon interaction.

You will build datasets, harnesses, and regression suites to drive product decisions, model selection, and post-training options when warranted. You will partner with cross-functional teams to define metrics, sampling, contamination control, and reproducible comparisons, ensuring evaluation signals guide engineering

Qualifications

• Experience building evaluation systems for ML models or agents.
• Strong background in RL, evaluation metrics, and regression testing.
• Ability to design reproducible evaluation datasets and harnesses.

Responsibilities

• Own the eval roadmap for RL and long-horizon model behavior.
• Build datasets, annnotation schemas, and evaluation harnesses.
• Drive product decisions through evaluation signals and experiments.

Skills

Evaluation systems

Reinforcement learning

Dataset design

Job description

Runtime Labs, Inc. is seeking a Founding Engineer to own RL and Evals for Óra, shaping how model behavior is measured across planning, memory, tool use, and long-horizon interaction.

You will build datasets, harnesses, and regression suites to drive product decisions, model selection, and post-training options when warranted. You will partner with cross-functional teams to define metrics, sampling, contamination control, and reproducible comparisons, ensuring evaluation signals guide engineering