ML Inference Engineer & Terminal Benchmark Architect

Toptal; United States; Remote; USD 165,000 - 248,000; Full time

Job summary

Toptal is seeking an ML inference and serving expert to design and deliver terminal-based benchmarks that mimic real paid ML work. You will define tasks, environments, and reference solutions that are end-to-end runnable by review teams, with time estimates of 60 to 75 minutes per task.

Responsibilities include building automated checks, packaging tasks with Docker and Git, and documenting data, tools, and rules so reviewers can validate difficulty and fairness.

Qualifications

• 3+ years doing ML inference or serving work hands-on (not only studying, selling or managing it).
• Trained, served or evaluated models at real scale (multi-GPU or production serving).
• Hands-on with PyTorch or JAX internals and serving stacks such as vLLM, SGLang or TensorRT-LLM.

Responsibilities

• Design expert-level Terminal-Bench tasks from real ML inference work.
• Define environments and streaming serving details for realistic challenges.
• Write a working reference solution and verify end-to-end execution (60–75 minutes).
• Build automated checks to prove results are correct.
• Package tasks with Docker and Git, and document systems and rules.

Skills

ML inference/serving

PyTorch/JAX internals

Python

Command line

Documentation writing

Education

PhD in ML (optional)

Tools

Docker

Git

Job description

Toptal is seeking an ML inference and serving expert to design and deliver terminal-based benchmarks that mimic real paid ML work. You will define tasks, environments, and reference solutions that are end-to-end runnable by review teams, with time estimates of 60 to 75 minutes per task.

Responsibilities include building automated checks, packaging tasks with Docker and Git, and documenting data, tools, and rules so reviewers can validate difficulty and fairness.