Senior ML Inference Engineer for Terminal Bench Challenges

Toptal; United States; Remote; USD 110,000 - 165,000; Full time

Job summary

Toptal is seeking ML inference and serving experts to design Terminal-Bench tasks based on real paid ML inference work. You will create end-to-end tasks with environments, reference solutions, and automated correctness checks.

Each task takes about 60 to 75 minutes and is paid per task. Expected start by early November 2026, with submissions delivered task by task and reviewer feedback guiding the work.

Qualifications

• 3+ years doing ML inference or serving work hands-on (not only studying, selling or managing it), or a PhD plus active research in the subfield.
• Did this work in the last 3 years and can name the systems, tools, data and rules used
• Trained, served or evaluated models at real scale (multi-GPU or production serving), not only notebooks or API calls
• Hands‑on with PyTorch or JAX internals and serving stacks such as vLLM, SGLang or TensorRT‑LLM
• Can explain how ML results go wrong without errors (numerical drift, metric bugs) and how to catch them
• Technical fluency in Python and the command line, and with files, CSV/JSON, APIs or databases
• Availability to start by early November 2026 and complete tasks of about 60 to 75 minutes each

Responsibilities

• Design expert-level Terminal-Bench tasks that turn real, paid ML inference work from your own experience into challenges an AI agent must solve end to end.
• Define the files, systems and environments each task runs on, such as model serving with streaming, so the challenge is realistic, reproducible and command-line solvable.
• Write a working reference solution for every task and verify it runs end to end, sized for about 60 to 75 minutes of expert effort.
• Build automated checks that objectively prove a result is correct, including cases where ML outputs go wrong silently, such as numerical drift or metric bugs.
• Package tasks with Docker and Git so they are reproducible, and document the systems, tools and data involved so reviewers can validate difficulty.
• Iterate on tasks based on reviewer feedback and reviewer results, adjusting difficulty and fixing ambiguities so each task stays expert-level, fair and gradable.

Skills

ML inference

Model serving

Python

CLI / Bash

Debugging ML results

Multi-GPU / production serving

Tools

Docker

Git

vLLM

SGLang

TensorRT-LLM

PyTorch

JAX

Job description

Toptal is seeking ML inference and serving experts to design Terminal-Bench tasks based on real paid ML inference work. You will create end-to-end tasks with environments, reference solutions, and automated correctness checks.

Each task takes about 60 to 75 minutes and is paid per task. Expected start by early November 2026, with submissions delivered task by task and reviewer feedback guiding the work.