ML Research Engineer: Reproducible Eval & Experiments

Nodi; New York (NY); On-site; USD 235,000 - 295,000; Full time

Job summary

Nodi is seeking a research-forward ML engineering role in New York. You will build the evaluation and experimentation systems that drive continuous improvement of the agent's decisions, working with the Principal Scientist to design controlled experiments and robust benchmarks.

Your work includes reproducible replay, versioned evaluation datasets, metrics, and performance analysis, turning hypotheses into production-ready improvements with a strong software engineering discipline.

Qualifications

• 4-8 years building production ML systems, research infrastructure, or data-intensive backends; strong enough to design, build, and ship end-to-end
• Excellent Python and strong software engineering discipline: testing, typing, packaging, code review, CI
• Direct experience with ML evaluation and experimentation: offline evaluation harnesses, dataset and split design, metric design, and sound reasoning about noisy or underpowered results
• Data pipeline skills: schema design, versioning, orchestration (Airflow, Prefect, Dagster, Ray, or equivalent), object storage
• A reproducibility instinct: pinned environments, seeded runs, versioned artifacts, results that hold up when someone else reruns them six months later
• The ability to take ambiguous system behavior, reduce it to a controlled experiment, and write up what the result does and does not support
• Ability to turn ideas into working prototypes quickly, then harden successful ones into reliable production systems
• Initiative and ownership in an environment where the roadmap is still being written

Responsibilities

• Research and experimentation: formulate hypotheses about agent behavior, design controlled experiments, and implement and evaluate improvements to planning, tool use, and decision-making
• Reproducible replay: reconstruct historical agent runs from recorded context for consistent comparisons
• Versioned evaluation datasets: curated task sets, splits, and ground truth that evolve without invalidating past results
• Evaluation harness and metrics: decision quality, task success, recovery from error, cost, and latency
• Trajectory analysis: instrument agent runs, characterize failure modes, and convert recurring failures into regression coverage
• Control policies: prototype, test, and ship the logic that decides when the agent revises an approach
• Experiment infrastructure: data pipelines, artifact and dataset versioning, experiment tracking, and tooling for fast clean comparisons
• Close partnership with scientists and platform engineers to connect agent behavior to Workcell outcomes

Skills

Python

Production ML systems

Data pipelines

Testing and CI

Experimentation design

Reproducibility

Tools

Airflow

Prefect

Dagster

Ray

MLflow

Weights & Biases

Langfuse

Job description

Nodi is seeking a research-forward ML engineering role in New York. You will build the evaluation and experimentation systems that drive continuous improvement of the agent's decisions, working with the Principal Scientist to design controlled experiments and robust benchmarks.

Your work includes reproducible replay, versioned evaluation datasets, metrics, and performance analysis, turning hypotheses into production-ready improvements with a strong software engineering discipline.