AI Experimentation Engineer for Agent Decision Systems

Radical AI; New York (NY); On-site; USD 140,000 - 210,000; Full time

Job summary

Radical AI is seeking an engineering-focused researcher to design evaluation datasets, replay historical runs, and ship improvements to our agent's planning and control loops. You will own evaluation and experimentation systems and collaborate with platform engineers.

The profile is engineering-first, ML-strong, and research-capable. You should design data schemas and CI pipelines, and turn ML ideas into controlled experiments that ship improvements with production-quality results.

Qualifications

• 4–8 years building production ML systems, research infrastructure, or data-intensive backends.
• Excellent Python and strong software engineering discipline: testing, typing, packaging, code review, CI
• Direct experience with ML evaluation and experimentation: offline evaluation harnesses, dataset and split design, metric design, and sound reasoning about noisy or underpowered results
• Data pipeline skills: schema design, versioning, orchestration (Airflow, Prefect, Dagster, Ray, or equivalent), object storage
• A reproducibility instinct: pinned environments, seeded runs, versioned artifacts, results that hold up when someone else reruns them six months later
• The ability to take ambiguous system behavior, reduce it to a controlled experiment, and write up what the result does and does not support
• Ability to turn ideas into working prototypes quickly, then harden successful ones into reliable production systems
• Initiative and ownership in an environment where the roadmap is still being written

Responsibilities

• Formulate hypotheses about agent behavior and design controlled experiments, implementing and evaluating improvements to planning, tool use, and decision-making.
• Reproduce historical agent runs from recorded context for fair model comparisons under consistent conditions.
• Develop versioned evaluation datasets, curated task sets, splits, and ground truth that evolve without invalidating past results.
• Build evaluation harness and metrics: decision quality, task success, recovery from error, cost and latency.
• Instrument agent runs, characterize failure modes, and convert recurring failures into regression coverage.
• Prototype, test, and ship control policies for when the agent revises an approach or escalates to a scientist.
• Create experiment infrastructure: data pipelines, artifact and dataset versioning, experiment tracking to enable quick clean comparisons.
• Collaborate with scientists and platform engineers to connect agent behavior to Workcell outcomes.

Skills

Python programming

Production ML systems

CI/testing

Experiment design

Data pipelines

Versioned experiments

LLM agents concepts

Tools

Airflow

Prefect

Dagster

Ray

Langfuse

MLflow

Weigths & Biases

DVC

OpenTelemetry

Prometheus

Grafana

Datadog

Job description

Radical AI is seeking an engineering-focused researcher to design evaluation datasets, replay historical runs, and ship improvements to our agent's planning and control loops. You will own evaluation and experimentation systems and collaborate with platform engineers.

The profile is engineering-first, ML-strong, and research-capable. You should design data schemas and CI pipelines, and turn ML ideas into controlled experiments that ship improvements with production-quality results.