Senior ML Inference Systems Engineer

Acceler8 Talent; San Francisco (CA); On-site; USD 180,000 - 240,000; Full time

Job summary

Acceler8 Talent is seeking a Member of Technical Staff to join its AI infrastructure team in San Francisco. You will design and build production-grade ML inference and model serving systems, optimizing latency, throughput, and resource utilization across large-scale AI workloads.

You will work with inference runtimes such as vLLM or TensorRT-LLM, tackle memory management and scheduling challenges, and collaborate with compiler, kernel, and distributed systems engineers to push the performance of

Qualifications

• Strong software engineering fundamentals.
• Experience building ML inference or model serving systems in production.
• Deep understanding of system performance and memory behaviour under production workloads.
• Experience with inference runtimes such as vLLM, TensorRT-LLM, or custom serving frameworks.
• Familiarity with batching, scheduling, concurrency, and KV cache management.
• Experience profiling and optimising latency- and throughput-critical systems.
• Strong Python and C++ development experience.

Responsibilities

• Design and build production-grade ML inference and model serving systems
• Optimise latency, throughput, and resource utilisation across large-scale AI workloads
• Develop execution strategies around batching, scheduling, concurrency, and runtime optimisation
• Improve KV cache management, memory efficiency, and model execution behaviour
• Enable new model architectures and inference techniques to run efficiently in production
• Partner closely with compiler, kernel, networking, and distributed systems engineers to drive end-to-end performance
• Help shape the architecture of a next-generation AI inference platform powering production workloads at scale

Skills

Python

C++

ML inference

Model serving

Latency optimisation

Performance profiling

Concurrency

Batching

Scheduling

KV cache

Job description

Acceler8 Talent is seeking a Member of Technical Staff to join its AI infrastructure team in San Francisco. You will design and build production-grade ML inference and model serving systems, optimizing latency, throughput, and resource utilization across large-scale AI workloads.

You will work with inference runtimes such as vLLM or TensorRT-LLM, tackle memory management and scheduling challenges, and collaborate with compiler, kernel, and distributed systems engineers to push the performance of