ML Research Engineer - Experimental Systems & Evaluation

Nodi; New York (NY); Hybrid; USD 235,000 - 295,000; Full time

Job summary

Nodi is seeking an engineering-focused ML scientist to design, run, and ship evaluations that improve agent decision-making and production systems. You will collaborate with a Principal Scientist to create reproducible experiments, build evaluation datasets, and push improvements from prototype to production in a data-driven Workcell environment.

Your role emphasizes end-to-end ML workflows, from hypothesis to deployment, with an emphasis on reproducibility, instrumentation, and scalable

Qualifications

• 4-8 years building production ML systems or data-intensive backends
• Excellent Python and software engineering discipline (testing, typing, CI)
• Direct experience with ML evaluation and experimentation (offline harnesses, metrics)
• Data pipeline design, versioning, orchestration tools and object storage
• Strong reproducibility instincts: pinned environments and repeatable runs
• Ability to turn ambiguous system behavior into controlled experiments
• Prototyping ideas quickly and shipping robust production improvements
• Ownership and initiative in a roadmap still being written

Responsibilities

• Formulate hypotheses about agent behavior with the Principal Scientist
• Design controlled experiments to evaluate planning, tool use, and decision-making
• Build and maintain reproducible replay of historical agent runs
• Develop versioned evaluation datasets and evaluation harnesses
• Instrument experiments and track artifacts and datasets across runs
• Collaborate with scientists and platform engineers to connect agent behavior to Workcell outcomes

Skills

Production ML systems

Python proficiency

ML evaluation & experimentation

Data pipelines

Reproducibility & versioning

Experiment design

Prototyping to production

Ownership & initiative

Tools

Airflow

Prefect

Dagster

Ray

Langfuse

MLflow

Weights & Biases

DVC

Job description

Nodi is seeking an engineering-focused ML scientist to design, run, and ship evaluations that improve agent decision-making and production systems. You will collaborate with a Principal Scientist to create reproducible experiments, build evaluation datasets, and push improvements from prototype to production in a data-driven Workcell environment.

Your role emphasizes end-to-end ML workflows, from hypothesis to deployment, with an emphasis on reproducibility, instrumentation, and scalable