ML Inference Engineer â Terminal-Bench Task Designer (vLLM / SGLang / TensorRT-LLM)

Toptal

Netherlands

Remote

EUR 83,000 - 138,000

Part time

12 days ago

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Job summary

Toptal is building a benchmark to test AI agents on real expert Terminal-Bench tasks. You will design and package expert-level ML inference tasks with end-to-end environments, reference solutions, and automated checks, delivering tasks that take about 60 to 75 minutes of expert effort.

Work is paid per task, with reviewers providing feedback and the opportunity to iterate on difficulty and clarity. Start date is early November 2026; remote work is possible within the Netherlands.

Qualifications

  • 3+ years hands-on ML inference or serving work.
  • Experience with multi-GPU or production serving.
  • Proficient in Python and the command line.
  • Experience with PyTorch or JAX internals and serving stacks.
  • Ability to explain numerical drift and metric bugs.
  • Familiarity with Docker and Git.
  • Available to start by early November 2026.

Responsibilities

  • Design expert-level Terminal-Bench tasks that translate real ML inference work into challenges for AI agents.
  • Define required environments, models, data, and tooling for reproducible tasks.
  • Write reference solutions and verify end-to-end execution in 60–75 minutes.
  • Build automated checks to prove correctness and handle drift or metric issues.
  • Package tasks with Docker and Git and document systems and rules for reviewers.
  • Iterate on tasks based on reviewer feedback and maintain expert-level fairness.

Skills

ML inference
Model serving
PyTorch
JAX
Python
Command line
Data formats

Tools

Docker
Git

Job description

Overview

We are building a benchmark that tests how well AI agents handle real expert work in a terminal. We are hiring ML inference and serving experts to turn real, paid tasks from their jobs into expert-level Terminal-Bench challenges. Each task includes the environment, a working solution, and an automated check that proves the result is correct. ***THIS IS A TASK BASED COMPENSATION MODEL. PLEASE CHECK THE APPLICATION QUESTIONS FOR MORE DETAILS.***

Responsibilities
  • Design expert-level Terminal-Bench tasks that turn real, paid ML inference work from your own experience into challenges an AI agent must solve end to end.
  • Define the files, systems and environments each task runs on, such as model serving with streaming, so the challenge is realistic, reproducible and command-line solvable.
  • Write a working reference solution for every task and verify it runs end to end, sized for about 60 to 75 minutes of expert effort.
  • Build automated checks that objectively prove a result is correct, including cases where ML outputs go wrong silently, such as numerical drift or metric bugs.
  • Package tasks with Docker and Git so they are reproducible, and document the systems, tools, data and rules involved so reviewers can validate difficulty.
  • Iterate on tasks based on reviewer feedback and agent test runs, adjusting difficulty and fixing ambiguities so each task stays expert-level, fair and objectively gradable.
Skills/Experience
  • 3+ years doing ML inference or serving work hands‑on (not only studying, selling or managing it), or a PhD plus active research in the subfield
  • Did this work in the last 3 years and can name the systems, tools, data and rules used
  • Trained, served or evaluated models at real scale (multi-GPU or production serving), not only notebooks or API calls
  • Hands‑on with PyTorch or JAX internals and serving stacks such as vLLM, SGLang or TensorRT‑LLM
  • Can explain how ML results go wrong without errors (numerical drift, metric bugs) and how to catch them
  • Technical fluency in Python and the command line, and with files, CSV/JSON, APIs or databases
  • Availability to start by early November 2026 and complete tasks of about 60 to 75 minutes each
Nice-to-have
  • AI lab or LLM infrastructure experience
  • Contributions to PyTorch, vLLM or SGLang
  • GPU profiling experience
  • Docker and Git proficiency
  • Experience writing automated tests
Process
  • Task vetting instead of an interview: complete a vetting task so the client can assess your task design
  • Paid per task; each task takes about 60 to 75 minutes
  • Work is delivered task by task, with reviewer feedback on submissions
  • Expected start in early November 2026

Please mention the word ASTONISHED and tag RMzUuMTU2LjI0OC4xMDQ= when applying to show you read the job post completely (#RMzUuMTU2LjI0OC4xMDQ=). This is a beta feature to avoid spam applicants. Companies can search these words to find applicants that read this and see they're human.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

AI Benchmark Engineer | Native Language Specialist - Dutch - Remote
AI Benchmark Engineer | Native Language Specialist - Dutch - Remote

LILT AI • Amsterdam

Remote
EUR 83,000 - 165,000
Flexible schedule
Competitive pay
Remote work possible
Senior Staff Engineer (Machine Learning) - 45391
Senior Staff Engineer (Machine Learning) - 45391

Turing • Netherlands

Remote
EUR 80,000 - 120,000
Work in a fully remote environment
Opportunity to work on cutting-edge AI projects
Flexible commitments with minimum hours
Operations Expert with Python for AI Supply Chain Task Design
Operations Expert with Python for AI Supply Chain Task Design

Toptal • Netherlands

Remote
EUR 34,000 - 83,000
Senior Software Engineer (LLM Ops & Evals)
Senior Software Engineer (LLM Ops & Evals)

Datasnipper • Netherlands

On-site
EUR 90,000 - 130,000
Staff Machine Learning Engineer, LLM Inference Optimization
Staff Machine Learning Engineer, LLM Inference Optimization

Nebius Group • Amsterdam

On-site
EUR 120,000 - 190,000
Competitive compensation
Career growth and learning opportunity
Flexibility and ownership
+2
Software Engineer (Full-Stack, AI & LLM Integration)
Software Engineer (Full-Stack, AI & LLM Integration)

Ethical Hat Security Pvt. Ltd. • Netherlands

Remote
EUR 70,000 - 95,000
Principal ML Researcher
Principal ML Researcher

Toloka • Netherlands

Hybrid
EUR 150,000 - 220,000
Base salary + bonus
ESOP
Remote or hybrid model
+3
ML Engineer (MSc. or PhD)
ML Engineer (MSc. or PhD)

Micro1 • Netherlands

Remote
EUR 75,000 - 150,000
Fully remote
Flexible schedule
ML Researcher
ML Researcher

Desert Ant Labs • Amsterdam

Remote
EUR 70,000 - 120,000
Junior Machine Learning Engineer
Junior Machine Learning Engineer

Micro1 • Netherlands

Remote
EUR 75,000 - 150,000