Member of Technical Staff (ML Platform)

Runway; United States; Remote; USD 180,000 - 240,000; Full time

Benefits offered by this job

Medical, dental, and vision coverage

Unlimited PTO

401k matching 3%

Remote stipend $250/mo

Community donations program

Tech setup stipend

Parental leave 16 weeks

L&D stipend $600/yr

Wellness stipend $600/yr

Job summary

Runway is seeking an ML infrastructure engineer to own model evaluation end to end. You will design and build the systems that generate samples, score with automated metrics and human annotations, and present results to researchers in minutes.

You’ll define what better means and drive the roadmap for evals across the ML Platform. You will work with research teams on video, image, audio, agents, and robotics, ensuring reproducibility and trust in ship decisions.

Qualifications

• 5+ years of experience building ML infrastructure or data platforms in production environments with evaluation/benchmarking systems.
• Strong Python and PyTorch experience; hands-on with large GPU workloads on Kubernetes.
• Ability to gather requirements, set direction and drive a roadmap across teams.
• Experience designing data pipelines and storage for large media/model outputs with versioning and reproducibility.
• Familiarity with experimental statistics and multiple-comparison considerations.
• Hands-on work with LLM- or VLM-as-judge pipelines and evaluating generative models.
• Experience with online experimentation platforms and connecting offline metrics to product outcomes.

Responsibilities

• Own the evaluation platform end to end: tooling and systems to generate, annotate, review and adapt at frontier scale.
• Design and build systems that generate samples at scale and score them with metrics and human annotations.
• Define CLIs, APIs, GUIs and storage layers to make evaluation seamless and fast.
• Collaborate with research teams across video, image, audio, agents, and robotics to measure needs.
• Set standards for evaluation, reproducibility, metric definitions and trustworthy results.
• Support broad adoption of the platform across training, production serving and more.

Skills

Systems thinking

Lead broad technical area

Data pipelines

Self-starter

ML infra production

Python & PyTorch

Internal tools

Experimental statistics

Humility/open-mindedness

Model lifecycle

LLM/VLM evaluation

Evaluation suites

Online experimentation

Agents/robotics evaluation

Tools

Kubernetes

Python

PyTorch

Job description

• We’re looking for an ML infrastructure engineer to own model evaluation at Runway, end to end. Every decision we make about a model – which checkpoint to keep training, what to ship to millions of users, which datamix and architecture shows the most promise – rests on evals. Today that work is spread across the organization. You’ll turn it into one platform
• Your job will be to design and build the systems that generate samples at scale, score them with automated metrics and human annotations, track results across checkpoints and releases, and put the answers in front of researchers in minutes rather than days. You’ll define what “better” means operationally – how we measure it, how confident we are, and how a result becomes a ship/no-ship decision
• You’ll be embedded within research teams as a member of ML Platform, and you’ll set the technical direction for evals across the company. This is an extremely high-leverage role: the quality of our models is bounded by how well we can measure them
• Our inference and training code is written in Python (PyTorch) and is cloud-native. We leverage multiple clouds and use a combination of Kubernetes native and home-grown tooling for efficient job orchestration
• We’re big users of agentic development and operations. We have a robust internal platform and provide broad access to the latest models and harnesses
• You’ll have access to best-in-class models, agents, GPUs, storage and cloud services you need to drive a world class evaluation pipeline that helps us deliver frontier models
• Own the evaluation platform end to end: the tooling and systems required to generate, annotate, review and adapt at frontier scale
• Define the CLIs, APIs, GUIs and storage layers required to make this process seamless, fast, sophisticated and collaborative
• Work directly with research teams on video, image, audio, agents, and robotics to understand what they need to measure and build it, then generalize the result into the platform
• Help set the standards for how we evaluate models at Runway: reproducibility, metric definitions, reporting formats, and when a result is trustworthy enough to act on
• Support broad adoption of the platform and its integration throughout Runway’s research efforts across training, production model serving and more
• Contribute broadly as a member of the ML Platform team to tools and systems that help Runway train and serve frontier models

Benefits

• 100% medical, dental, and vision coverage: We cover 100% of medical, dental and vision for our US employees AND their dependents. Based on the plans selected, we have complimentary mental health benefits and a free One Medical membership
• Unlimited days of PTO: We have unlimited PTO with a minimum of 15 days a year. We also have a paid sabbatical program after employees have been with Runway for 3 years and again after 5 years. And of course, we can’t forget about Summer Fridays
• 401k Matching, 3% matching: Plan for the future! We provide a 3% 401K match for our US based employees
• Remote $250 stipend: All of our employees outside of commuting distance to the office, can expense up to $250 per month on co-working spaces, lunch or groceries, and internet connection
• Donations supporting our community: Beyond our employees, we have a commitment to our communities. Every quarter, Runway employees choose an organization for the company to support through a donation
• Custom setup, tech setup stipend: In addition to the choice of laptop, all employees get a stipend to customize their tech set up as they want
• Parental leave for all parents: We offer 16 weeks of fully paid parental leave plus a gradual return to work plan. This coverage is for all parents, regardless of how a child is brought into the family
• L&D $600 stipend: We’re investing in our employees’ long term growth by giving every employee $600 per year to spend on their personal or professional development. That means conferences, courses, and books, oh my!
• Wellness $600 stipend: We set ambitious goals and work hard to achieve them, so we believe it’s important to invest in employees’ mental and physical health. The $600 annual wellness stipend can be used for anything from therapy sessions to gym memberships

Requirements

• Strong systems thinking and pragmatic approach to production reliability
• Ability to lead a broad technical area: gather requirements from many teams, set direction, make tradeoffs, and drive a roadmap without waiting to be told
• Experience designing data pipelines and storage for large volumes of media or model outputs, with attention to versioning and reproducibility
• Self-starter who can work embedded with research teams and move fast
• 5+ years of experience building ML infrastructure or data platforms in production environments, with at least some of that time spent on evaluation, experimentation, or benchmarking systems
• Strong Python and PyTorch, and hands-on experience running large batch GPU workloads on Kubernetes
• Comfort building internal tools end to end, from the command line to the browser
• Familiarity with experimental statistics: paired comparisons, confidence intervals, multiple-comparison pitfalls, inter-rater agreement
• Humility and open mindedness; at Runway we love to learn from one another
• Familiarity with the full model development lifecycle: data, training, evaluation, serving
• Hands-on work with LLM- or VLM-as-judge pipelines
• Track record of building evaluation suites for generative models
• Experience with online experimentation platforms and connecting offline metrics to product outcomes
• Prior work evaluating agents or robotics policies