Production-Scale ML Inference Engineer for LLMs

Acceler8 Talent; San Francisco (CA); Hybrid; USD 150,000 - 230,000; Full time

Job summary

Acceler8 Talent in San Francisco is seeking an AI Inference Engineer to help redesign the infrastructure behind its production AI platform.

You will architect high-performance systems for serving LLMs, optimize latency and throughput, and scale GPU workloads across Kubernetes infrastructure, focusing on batching, quantisation, and memory efficiency to deliver faster, more cost-effective AI services.

Qualifications

• Strong background in ML systems, distributed computing, HPC or performance engineering.
• Proficiency in Python and/or C++, with solid PyTorch and GPU computing knowledge.
• Experience with CUDA, NCCL or Triton is valuable; hands-on work with inference frameworks like vLLM or TensorRT-LLM is a plus.

Responsibilities

• Architect high-performance systems for serving LLMs at production scale.
• Optimize inference for latency, throughput and cost efficiency.
• Build distributed execution across multi-GPU and multi-node environments.
• Improve GPU resource scheduling, allocation and utilisation.
• Profile the inference stack to identify bottlenecks in compute, memory and networking.
• Scale GPU workloads across Kubernetes-based infrastructure.
• Explore batching, quantisation, KV caching and parallelism to improve serving performance.

Skills

ML systems

Distributed computing

Python

C++

PyTorch

GPU computing

Tools

CUDA

NCCL

Triton

TensorRT-LLM

SGLang

vLLM

Job description

Acceler8 Talent in San Francisco is seeking an AI Inference Engineer to help redesign the infrastructure behind its production AI platform.

You will architect high-performance systems for serving LLMs, optimize latency and throughput, and scale GPU workloads across Kubernetes infrastructure, focusing on batching, quantisation, and memory efficiency to deliver faster, more cost-effective AI services.