ML Inference Engineer – Terminal-Bench Task Designer (vLLM / SGLang / TensorRT-LLM)

Toptal; United States; On-site; USD 138,000 - 276,000; Part time

Job summary

Toptal is seeking ML inference and serving experts to design and deliver expert-level Terminal-Bench tasks. You will turn real, paid ML inference work into hands-on challenges with reproducible environments and working reference solutions.

Tasks are bundled with Docker and Git, and include automated correctness checks. You will ensure tasks reflect real system workloads, including multi-GPU serving, and provide clear documentation of data, rules, and environments so reviewers can validate

Qualifications

• 3+ years doing ML inference or serving hands-on (or a PhD with active research in the subfield)
• Can name the systems, tools, data and rules used in recent work (last 3 years)
• Experience with real-scale model serving (multi-GPU or production)
• Hands-on with Python, CLI, and file-based data formats (CSV/JSON)
• Able to explain how ML results fail and how to detect issues

Responsibilities

• Design expert-level Terminal-Bench tasks from real ML inference work
• Define environments, models, and systems tasks run on in reproducible, solvable ways
• Write reference solutions that run end-to-end in 60–75 minutes
• Build automated checks to prove results are correct and robust
• Package tasks with Docker and Git and document data, tools, and rules
• Iterate tasks based on reviewer feedback to maintain expert-level quality

Skills

ML inference

Python

Shell/CLI

Docker

Git

Model serving systems

Automated tests

Multi-GPU serving

Tools

PyTorch

JAX

vLLM

SGLang

TensorRT-LLM

Job description

Overview

We are building a benchmark that tests how well AI agents handle real expert work in a terminal. We are hiring ML inference and serving experts to turn real, paid tasks from their jobs into expert-level Terminal-Bench challenges. Each task includes the environment, a working solution, and an automated check that proves the result is correct. ***THIS IS A TASK BASED COMPENSATION MODEL. PLEASE CHECK THE APPLICATION QUESTIONS FOR MORE DETAILS.***

Responsibilities

• Design expert-level Terminal-Bench tasks that turn real, paid ML inference work from your own experience into challenges an AI agent must solve end to end.
• Define the files, systems and environments each task runs on, such as model serving with streaming, so the challenge is realistic, reproducible and command-line solvable.
• Write a working reference solution for every task and verify it runs end to end, sized for about 60 to 75 minutes of expert effort.
• Build automated checks that objectively prove a result is correct, including cases where ML outputs go wrong silently, such as numerical drift or metric bugs.
• Package tasks with Docker and Git so they are reproducible, and document the systems, tools, data and rules involved so reviewers can validate difficulty.
• Iterate on tasks based on reviewer feedback and agent test runs, adjusting difficulty and fixing ambiguities so each task stays expert-level, fair and objectively gradable.

Skills/Experience

• 3+ years doing ML inference or serving work hands-on (not only studying, selling or managing it), or a PhD plus active research in the subfield
• Did this work in the last 3 years and can name the systems, tools, data and rules used
• Trained, served or evaluated models at real scale (multi-GPU or production serving), not only notebooks or API calls
• Hands-on with PyTorch or JAX internals and serving stacks such as vLLM, SGLang or TensorRT-LLM
• Can explain how ML results go wrong without errors (numerical drift, metric bugs) and how to catch them
• Technical fluency in Python and the command line, and with files, CSV/JSON, APIs or databases
• Availability to start by early November 2026 and complete tasks of about 60 to 75 minutes each

Nice-to-have

• AI lab or LLM infrastructure experience
• Contributions to PyTorch, vLLM or SGLang
• GPU profiling experience
• Docker and Git proficiency
• Experience writing automated tests

Process

• Task vetting instead of an interview: complete a vetting task so the client can assess your task design
• Paid per task; each task takes about 60 to 75 minutes
• Work is delivered task by task, with reviewer feedback on submissions
• Expected start in early November 2026