ML Research Engineer Brooklyn, NY On-site $235,000 – $295,000 · per year · USD

Nodi; New York (NY); Hybrid; USD 235,000 - 295,000; Full time

Job summary

Nodi is seeking an engineering-focused ML scientist to design, run, and ship evaluations that improve agent decision-making and production systems. You will collaborate with a Principal Scientist to create reproducible experiments, build evaluation datasets, and push improvements from prototype to production in a data-driven Workcell environment.

Your role emphasizes end-to-end ML workflows, from hypothesis to deployment, with an emphasis on reproducibility, instrumentation, and scalable

Qualifications

• 4-8 years building production ML systems or data-intensive backends
• Excellent Python and software engineering discipline (testing, typing, CI)
• Direct experience with ML evaluation and experimentation (offline harnesses, metrics)
• Data pipeline design, versioning, orchestration tools and object storage
• Strong reproducibility instincts: pinned environments and repeatable runs
• Ability to turn ambiguous system behavior into controlled experiments
• Prototyping ideas quickly and shipping robust production improvements
• Ownership and initiative in a roadmap still being written

Responsibilities

• Formulate hypotheses about agent behavior with the Principal Scientist
• Design controlled experiments to evaluate planning, tool use, and decision-making
• Build and maintain reproducible replay of historical agent runs
• Develop versioned evaluation datasets and evaluation harnesses
• Instrument experiments and track artifacts and datasets across runs
• Collaborate with scientists and platform engineers to connect agent behavior to Workcell outcomes

Skills

Production ML systems

Python proficiency

ML evaluation & experimentation

Data pipelines

Reproducibility & versioning

Experiment design

Prototyping to production

Ownership & initiative

Tools

Airflow

Prefect

Dagster

Ray

Langfuse

MLflow

Weights & Biases

DVC

Job description

Compensation $235,000 – $295,000 per year USD

Interested in this role?

The Role

Our scientific agent decides what experiment to run next. This role builds the machinery that lets us understand, evaluate, and continuously improve the agent's decisions.

You will work closely with the Principal Scientist to turn hypotheses about agent behavior into reproducible experiments and production improvements. Your initial focus will be evaluation datasets, replay, trajectory analysis, and the policies governing how the agent plans, responds to results, and recovers from failure.

You will own the evaluation and experimentation systems, contribute research ideas, and carry promising changes from prototype through production in partnership with platform engineers.

The profile is engineering-first, ML-strong, and research-capable. You should be as comfortable designing a data schema and CI pipeline as turning an ML or scientific idea into a controlled experiment and shipping the resulting improvement.

What You'll Work On

• Research and experimentation: work with the Principal Scientist to formulate hypotheses about agent behavior, design controlled experiments, and implement and evaluate improvements to planning, tool use, and decision-making.
• Reproducible replay: reconstruct historical agent runs from recorded context so models, prompts, and policies can be compared under consistent conditions, with clear limits on what those comparisons establish.
• Versioned evaluation datasets: curated task sets, splits, and ground truth that evolve without invalidating past results
• Evaluation harness and metrics: decision quality, task success, recovery from error, cost, and latency, with honest treatment of noise and small sample sizes
• Trajectory analysis: instrument agent runs, characterize failure modes, and convert recurring failures into regression coverage
• Control policies: prototype, test, and ship the logic that decides when the agent revises an approach, pivots, escalates to a scientist, or terminates
• Experiment infrastructure: data pipelines, artifact and dataset versioning, experiment tracking, and the tooling that lets the team run a clean comparison in minutes rather than days
• Close partnership with our scientists and platform engineers to connect agent behavior to what actually happens in the Workcell

What We're Looking For

Required:

• 4-8 years building production ML systems, research infrastructure, or data-intensive backends; strong enough to design, build, and ship end-to-end
• Excellent Python and strong software engineering discipline: testing, typing, packaging, code review, CI
• Direct experience with ML evaluation and experimentation: offline evaluation harnesses, dataset and split design, metric design, and sound reasoning about noisy or underpowered results
• Data pipeline skills: schema design, versioning, orchestration (Airflow, Prefect, Dagster, Ray, or equivalent), object storage
• A reproducibility instinct: pinned environments, seeded runs, versioned artifacts, results that hold up when someone else reruns them six months later
• The ability to take ambiguous system behavior, reduce it to a controlled experiment, and write up what the result does and does not support
• Ability to turn ideas into working prototypes quickly, then harden successful ones into reliable production systems
• Initiative and ownership in an environment where the roadmap is still being written

Nice to have:

• Experience with LLM agents: tool calling, planning loops, context management, trajectory logging, and agent evaluation
• Background in sequential decision making: bandits, Bayesian optimization, reinforcement learning, or off-policy evaluation
• Experiment tracking, agent observability, and artifact versioning tools (Langfuse, MLflow, Weights and Biases, DVC)
• Scientific computing, or experience with materials, chemistry, or instrument data
• Experience operating large-scale inference systems, including GPU scheduling and distributed execution
• Observability experience (OpenTelemetry, Prometheus, Grafana, Datadog)
• Prior work in lab automation, robotics, or a self-driving lab

What Success Looks Like

In your first six to twelve months:

• Representative historical agent runs can be replayed against recorded context and compared with candidate models, prompts, or policies, with clear limits on those comparisons.
• A versioned evaluation suite gates changes to the agent, and the team trusts it enough to act on the result
• Recurring failure modes are documented, categorized, and covered by regression tests.
• Scientific and agent hypotheses can be turned into controlled experiments quickly, with clear conclusions about what the evidence supports and what to try next.
• At least one meaningful agent or control improvement is shipped to production, with measured gains in decision quality or scientific throughput.

Radical AI is an equal opportunity employer. We do not discriminate on the basis of race, color, ancestry, national origin, religion, sex, age, sexual orientation, gender identity and expression, marital status, disability, or veteran status.