LLM Inference Systems Engineer

Acceler8 Talent; San Francisco (CA); On-site; USD 150,000 - 230,000; Full time

Job summary

Acceler8 Talent in San Francisco seeks an ML Inference Engineer to redesign the infrastructure behind its production AI platform. You will architect high-performance systems for serving LLMs, optimize inference for latency and throughput, and scale GPU workloads across a Kubernetes-based stack.

You will work close to the hardware and software stack, balancing trade-offs for impactful improvements at scale.

Qualifications

• Strong background in ML systems, distributed computing, HPC or performance engineering.
• Experience working close to infrastructure that runs modern ML models.
• Proficiency in Python and/or C++.
• Solid understanding of PyTorch and GPU computing.
• Experience with CUDA, NCCL or Triton.
• Hands-on work with inference frameworks such as vLLM, TensorRT-LLM or SGLang.

Responsibilities

• Architect high-performance systems for serving LLMs at production scale.
• Optimize inference for latency, throughput and cost efficiency.
• Build distributed execution across multi-GPU and multi-node environments.
• Improve GPU resource scheduling, allocation and utilization.
• Profile the inference stack to identify bottlenecks across compute, memory and networking.
• Scale GPU workloads across Kubernetes-based infrastructure.
• Optimize techniques including batching, quantisation, KV caching and parallelism.

Skills

ML systems

Distributed computing

HPC

Performance engineering

Python

C++

PyTorch

GPU computing

CUDA

NCCL

Triton

Tools

CUDA Toolkit

NCCL

Triton

Job description

Acceler8 Talent in San Francisco seeks an ML Inference Engineer to redesign the infrastructure behind its production AI platform. You will architect high-performance systems for serving LLMs, optimize inference for latency and throughput, and scale GPU workloads across a Kubernetes-based stack.

You will work close to the hardware and software stack, balancing trade-offs for impactful improvements at scale.