AI Inference Engineer

Acceler8 Talent; San Francisco (CA); Hybrid; USD 150,000 - 230,000; Full time

Job summary

Acceler8 Talent in San Francisco is seeking an AI Inference Engineer to help redesign the infrastructure behind its production AI platform.

You will architect high-performance systems for serving LLMs, optimize latency and throughput, and scale GPU workloads across Kubernetes infrastructure, focusing on batching, quantisation, and memory efficiency to deliver faster, more cost-effective AI services.

Qualifications

• Strong background in ML systems, distributed computing, HPC or performance engineering.
• Proficiency in Python and/or C++, with solid PyTorch and GPU computing knowledge.
• Experience with CUDA, NCCL or Triton is valuable; hands-on work with inference frameworks like vLLM or TensorRT-LLM is a plus.

Responsibilities

• Architect high-performance systems for serving LLMs at production scale.
• Optimize inference for latency, throughput and cost efficiency.
• Build distributed execution across multi-GPU and multi-node environments.
• Improve GPU resource scheduling, allocation and utilisation.
• Profile the inference stack to identify bottlenecks in compute, memory and networking.
• Scale GPU workloads across Kubernetes-based infrastructure.
• Explore batching, quantisation, KV caching and parallelism to improve serving performance.

Skills

ML systems

Distributed computing

Python

C++

PyTorch

GPU computing

Tools

CUDA

NCCL

Triton

TensorRT-LLM

SGLang

vLLM

Job description

AI Inference Engineer

A Stanford-spun AI company in San Francisco is hiring anML Inference Engineerto help redesign the infrastructure behind its production AI platform.

What you'll be solving

• Architect high-performance systems for servingLLMs at production scale
• Optimise inference forlatency, throughput and cost efficiency
• Build distributed execution acrossmulti-GPU and multi-node environments
• Improve how GPU resources arescheduled, allocated and utilised
• Profile the inference stack to identify bottlenecks acrosscompute, memory and networking
• Scale GPU workloads acrossKubernetes-based infrastructure
• Optimise techniques includingbatching, quantisation, KV caching and parallelism
• Make systems-level trade-offs where small improvements can have a major impact at scale

What you’ll bring

• You’ll have a strong background inML systems, distributed computing, HPC or performance engineering, with experience working close to the infrastructure that runs modern ML models.
• StrongPython and/or C++skills are important, alongside a solid understanding ofPyTorch and GPU computing.
• Experience with technologies such asCUDA, NCCL or Tritonwould be highly relevant, as would hands‑on work with inference frameworks includingvLLM, TensorRT-LLM or SGLang.
• You should be comfortable reasoning about performance at a systems level — understanding howGPU utilisation, memory bandwidth, communication overhead and model architectureinteract to determine serving performance.

This is an opportunity to own meaningful parts of aninference stack being built from the ground up, rather than maintaining an established platform.