ML Evaluation Platform Engineer

Runway; United States; Remote; USD 180,000 - 240,000; Full time

Benefits offered by this job

Medical, dental, and vision coverage

Unlimited PTO

401k matching 3%

Remote stipend $250/mo

Community donations program

Tech setup stipend

Parental leave 16 weeks

L&D stipend $600/yr

Wellness stipend $600/yr

Job summary

Runway is seeking an ML infrastructure engineer to own model evaluation end to end. You will design and build the systems that generate samples, score with automated metrics and human annotations, and present results to researchers in minutes.

You’ll define what better means and drive the roadmap for evals across the ML Platform. You will work with research teams on video, image, audio, agents, and robotics, ensuring reproducibility and trust in ship decisions.

Qualifications

• 5+ years of experience building ML infrastructure or data platforms in production environments with evaluation/benchmarking systems.
• Strong Python and PyTorch experience; hands-on with large GPU workloads on Kubernetes.
• Ability to gather requirements, set direction and drive a roadmap across teams.
• Experience designing data pipelines and storage for large media/model outputs with versioning and reproducibility.
• Familiarity with experimental statistics and multiple-comparison considerations.
• Hands-on work with LLM- or VLM-as-judge pipelines and evaluating generative models.
• Experience with online experimentation platforms and connecting offline metrics to product outcomes.

Responsibilities

• Own the evaluation platform end to end: tooling and systems to generate, annotate, review and adapt at frontier scale.
• Design and build systems that generate samples at scale and score them with metrics and human annotations.
• Define CLIs, APIs, GUIs and storage layers to make evaluation seamless and fast.
• Collaborate with research teams across video, image, audio, agents, and robotics to measure needs.
• Set standards for evaluation, reproducibility, metric definitions and trustworthy results.
• Support broad adoption of the platform across training, production serving and more.

Skills

Systems thinking

Lead broad technical area

Data pipelines

Self-starter

ML infra production

Python & PyTorch

Internal tools

Experimental statistics

Humility/open-mindedness

Model lifecycle

LLM/VLM evaluation

Evaluation suites

Online experimentation

Agents/robotics evaluation

Tools

Kubernetes

Python

PyTorch

Job description

Runway is seeking an ML infrastructure engineer to own model evaluation end to end. You will design and build the systems that generate samples, score with automated metrics and human annotations, and present results to researchers in minutes.

You’ll define what better means and drive the roadmap for evals across the ML Platform. You will work with research teams on video, image, audio, agents, and robotics, ensuring reproducibility and trust in ship decisions.