Research Engineer: RL Environments & Evaluation Harnesses

Nolla Health; New York (NY); On-site; USD 180,000 - 240,000; Full time

Benefits offered by this job

Health Insurance

HSA/FSA Eligible

Flexible, unlimited vacation

Meal stipends

Team retreats

Job summary

Nolla Health is building AI-powered healthcare evaluation environments and data pipelines to turn clinical knowledge into trustworthy benchmarks and training assets. We’re seeking a research engineer to design reproducible tasks, patient-state environments, and robust data processing.

You will collaborate with clinicians and engineers to define tasks, validate results, and ensure privacy and data governance while contributing to scalable evaluation tooling and product improvements in NYC.

Qualifications

• Strong programming skills and experience building reliable software for experiments, data processing, or agent systems.
• Hands-on experience building an agent evaluation harness, an RL environment, or closely related infrastructure.
• The ability to translate an open-ended problem into a runnable task with measurable outcomes.
• Careful judgment about data quality, reproducibility, grading reliability, and what a benchmark result actually supports.
• Comfort debugging across datasets, environment state, agent behavior, and scoring code.
• The ability to work with clinicians and turn domain expertise into precise specifications and tests.
• Evidence of independent execution through shipped systems, research implementations, open‑source work, or projects you’ve owned.

Responsibilities

• Build harnesses that run agents through reproducible tasks, capture their actions, and make failures easy to investigate.
• Develop multi-step and multi-visit environments with patient state, clinical tools, and realistic consequences for an agent’s decisions.
• Work with clinicians to turn clinical workflows into well-defined tasks, rubrics, and success criteria.
• Build data pipelines for preparing, validating, versioning, and tracing evaluation and training datasets.
• Develop scoring and reward mechanisms, combining programmatic checks, model-based grading, and clinician review where appropriate.
• Test graders and environments for ambiguity, shortcuts, data leakage, and reward hacking.
• Run experiments, analyze trajectories, and identify whether changes improve performance across meaningful slices of cases.
• Work with our applied AI engineer to turn evaluation findings into product improvements.
• Build with privacy, access controls, and permitted data use in mind.

Skills

Programming

Agent evaluation

Task specification

Data quality

Debugging

Clinician collaboration

Independent execution

Tools

RL Environments

Simulation tools

Data pipelines

Job description

Nolla Health is building AI-powered healthcare evaluation environments and data pipelines to turn clinical knowledge into trustworthy benchmarks and training assets. We’re seeking a research engineer to design reproducible tasks, patient-state environments, and robust data processing.

You will collaborate with clinicians and engineers to define tasks, validate results, and ensure privacy and data governance while contributing to scalable evaluation tooling and product improvements in NYC.