Inference infrastructure Engineer

Acceler8 Talent; Cambridge, Northern (MA, KY); Hybrid; USD 150,000 - 210,000; Full time

Job summary

Acceler8 Talent seeks a Member of Technical Staff, ML Systems for an early-stage AI infrastructure company focused on creating a memory layer for large-scale inference. The role centers on building low-level ML systems and optimizing performance across GPUs and accelerators.

You will work on long-context inference, KV-cache management, and reducing the cost of keeping model state close to compute, while contributing to hardware/software co-design and system-level integration with inference

Qualifications

• Experience in low-level ML systems.
• Experience with GPUs/accelerators and performance optimization.
• Familiarity with inference infrastructure tools.

Responsibilities

• Build and optimize LLM inference systems around distributed and expandable memory.
• Work across KV cache, data movement, GPU kernels and serving performance.
• Help shape the underlying hardware architecture using real workload behaviour.

Skills

Low-level ML systems

GPU performance

Hardware/software co-design

Distributed memory systems

Inference workload analysis

Tools

CUDA

vLLM

SGLang

TensorRT-LLM

LMCache

Job description

I'm hiring a Member of Technical Staff, ML Systems to join an early-stage AI infrastructure company building a new memory layer for large-scale inference.

The team is developing custom silicon and systems software that give GPUs and other AI accelerators access to a much larger high-bandwidth memory tier, with a particular focus on long-context inference, KV-cache management and reducing the cost of keeping large amounts of model state close to compute.

This is a low-level ML systems role working across inference engines, GPU performance and hardware/software co-design rather than application-layer AI.

• Build and optimise LLM inference systems around distributed and expandable memory
• Work across KV cache, data movement, GPU kernels and serving performance
• Help shape the underlying hardware architecture using real workload behaviour

The team is particularly interested in engineers with experience around vLLM, SGLang, TensorRT-LLM, LMCache, CUDA or similar inference infrastructure.