Senior ML Engineer — Inference & Optimization (Remote)

Cast AI; Miami (FL); Remote; GBP 120,000 - 180,000; Full time

Benefits offered by this job

Equity options

Remote-friendly

Learning budget

Annual hackathon

Equipment budget

Extra days off

Team events

Job summary

Cast AI is hiring for a remote-first role focused on optimizing inference stacks for large models. You will drive throughput, reduce latency, and tackle cold starts across multi-GPU, multi-node deployments using vLLM, SGLang, and TensorRT-LLM, with deep attention to quantization and system-level bottlenecks.

You will own the technical direction, benchmark-driven decisions, and scalable architecture, collaborating with a global team to push Cast AI's GPU-based inference platform forward.

Qualifications

• 5+ years building real ML systems with inference or training infra.
• Python production-grade services, not scripts.
• Hands-on with vLLM, SGLang, or TensorRT-LLM and understanding GPU performance.
• Quantization effects and regression testing.
• Experience with multi-GPU/multi-node tensor and expert parallelism.

Responsibilities

• Push throughput via continuous batching, prefill, and scheduling optimizations across the engine fleet.
• Reduce latency by profiling compute, memory, scheduling, storage, and network bottlenecks.
• Improve KV cache performance with hierarchical and paged attention and eviction strategies.
• Eliminate cold starts with optimized weight loaders and JIT/compile caches.
• Quantize without regressing model quality (INT8/INT4, FP8/FP4).
• Scale dispersion across nodes with distributed inference topologies and storage-friendly checkpointing.

Skills

ML systems

Python production

vLLM experience

SGLang/TensorRT-LLM

multi-GPU / multi-node

profiling & benchmarking

observability

Tools

vLLM

SGLang

TensorRT-LLM

PyTorch

Kubernetes

CUDA

OpenTelemetry

Prometheus

Grafana

ClickHouse

GitLab CI

Job description

Cast AI is hiring for a remote-first role focused on optimizing inference stacks for large models. You will drive throughput, reduce latency, and tackle cold starts across multi-GPU, multi-node deployments using vLLM, SGLang, and TensorRT-LLM, with deep attention to quantization and system-level bottlenecks.

You will own the technical direction, benchmark-driven decisions, and scalable architecture, collaborating with a global team to push Cast AI's GPU-based inference platform forward.