Senior ML Inference Engineer — Remote

Cast AI; United States; Remote; USD 180,000 - 240,000; Full time

Benefits offered by this job

Equity options

Remote-first

Learning budget

Annual hackathon

Equipment budget

Job summary

Cast AI in Miami-headquartered, globally distributed team seeks an engineer to own the inference stack optimization. Move throughput, latency, cold starts, and cost per token in the LLM serving pipeline.

Your work will tune loaders, schedulers, and KV caches across multi-node GPU deployments, using SGLang, vLLM, TensorRT-LLM, and PyTorch to deliver real-world speedups while controlling budgets. You will own benchmarking, data-driven decisions, and collaborating with a world-wide team in a

Qualifications

• You've built real ML systems for 5+ years, with depth in inference or training infra.
• Python is production-grade: services, not scripts.
• Experience with vLLM, SGLang, or TensorRT-LLM in production.
• You understand quantization effects and deployment realities.
• Multi-GPU/multi-node with tensor/experts parallelism.
• Instrumentation before optimization with measurable wins.

Responsibilities

• Push throughput via continuous batching and optimized prefill.
• Cut latency by profiling compute, memory, and I/O bottlenecks.
• Improve KV cache with hierarchical/paged attention and caching strategies.
• Reduce cold starts with weight loaders and JIT caches.
• Quantize without regressing quality across weights and activations.
• Scale distributed inference across nodes with efficient deployment.
• Set technical direction with reproducible experiments.

Skills

Real ML systems

Production-grade Python

Inference infra

Quantization awareness

Multi-GPU/Distributed

Benchmarking & instrumentation

Tools

SGLang

vLLM

TensorRT-LLM

PyTorch

CUDA

Kubernetes

OpenTelemetry

Prometheus/Grafana

Job description

Cast AI in Miami-headquartered, globally distributed team seeks an engineer to own the inference stack optimization. Move throughput, latency, cold starts, and cost per token in the LLM serving pipeline.

Your work will tune loaders, schedulers, and KV caches across multi-node GPU deployments, using SGLang, vLLM, TensorRT-LLM, and PyTorch to deliver real-world speedups while controlling budgets. You will own benchmarking, data-driven decisions, and collaborating with a world-wide team in a