ML Inference Systems Engineer — Throughput & Latency

Cast AI; United States; Remote; USD 180,000 - 230,000; Full time

Benefits offered by this job

Equity options

Remote-first global environment

Learning budget for conferences and up

Annual hackathon and team events

Equipment budget and extra days off

Job summary

Cast AI seeks a senior ML systems engineer to optimize its inference stack and distributed GPU deployments. You will drive throughput gains, reduce latency, and own tuning across loaders, schedulers, and KV caches in a global, Kubernetes-based environment.

You will collaborate with a remote-first team, shaping technical direction while balancing performance, cost, and scalability across models and workloads.

Qualifications

• 5+ years building real ML systems, especially inference or training infrastructure.
• Production-grade Python for services, not scripts.
• Experience with vLLM, SGLang, or TensorRT-LLM in real workloads.
• Measured quantization impacts beyond compression ratios.
• Multi-GPU and multi-node setups, including tensor and expert parallelism.

Responsibilities

• Push throughput with continuous batching, prefill and speculative decoding strategies.
• Tune and optimize KV caches, hierarchical and paged attention, and eviction policies.
• Scale inference across nodes with distributed topologies and network-aware placement.
• Reduce latency and cold starts; optimize loaders and JIT caches.
• Define benchmark strategy and drive reproducible experiments.

Skills

Production-grade Python

Performance profiling

Distributed systems

Inference optimization

Online A/B testing

Tools

SGLang

vLLM

TensorRT-LLM

PyTorch

fastsafetensors

CUDA

Kubernetes

Job description

Cast AI seeks a senior ML systems engineer to optimize its inference stack and distributed GPU deployments. You will drive throughput gains, reduce latency, and own tuning across loaders, schedulers, and KV caches in a global, Kubernetes-based environment.

You will collaborate with a remote-first team, shaping technical direction while balancing performance, cost, and scalability across models and workloads.