Senior ML Systems Engineer Inference Optimization (Remote)

CAST AI Group; Miami (FL); Remote; USD 180,000 - 280,000; Full time

Benefits offered by this job

Competitive salary

Remote-first global environment

Equity options

Learning budget

Annual hackathon

Team-building budget

Equipment budget

Extra days off

Job summary

CAST AI, a Miami-headquartered AI platform, seeks an experienced ML systems engineer to optimize the inference stack across multi-GPU clusters. You’ll tune loaders, schedulers, and KV caches to improve throughput, latency, and cost per token.

You’ll work with SGLang, vLLM, TensorRT-LLM, PyTorch, and CUDA tooling, handling large checkpoints and distributed setups. The role offers a remote-first global environment and equity options.

Qualifications

• Built real ML systems for inference or training infra (5+ years).
• Python is production-grade: services, not scripts.
• Experienced with vLLM, SGLang, or TensorRT-LLM in real scenarios.
• Measured quantization effects, not just ratios.
• Comfort with multi-GPU and multi-node tensor/communication patterns.
• Know where data lives (page caches, O_DIRECT, shared storage).
• Instrument before optimizing; distinguish real wins from artifacts.

Responsibilities

• Push throughput via continuous batching and scheduler tuning across the engine fleet.
• Cut latency by profiling bottlenecks (compute, memory, storage, network).
• Improve KV cache efficiency with hierarchical/paged attention and eviction policies.
• Eliminate cold starts with weight loaders, JIT caches, and shared-filesystem read paths.
• Quantize with minimal impact to quality (INT8/INT4, FP8/FP4 across weights and activations).
• Scale distributed inference across nodes with topology-aware placement and prefill strategies.
• Define technical direction; publish reproducible experiments and findings.

Skills

5+ years ML infra

Production-grade Python

vLLM/SGLang/TensorRT-LLM experience

Quantization impact analysis

Multi-GPU & multi-node

Instrumentation & benchmarking

System-level optimization

Tools

Python

SGLang

vLLM

TensorRT-LLM

PyTorch

Job description

CAST AI, a Miami-headquartered AI platform, seeks an experienced ML systems engineer to optimize the inference stack across multi-GPU clusters. You’ll tune loaders, schedulers, and KV caches to improve throughput, latency, and cost per token.

You’ll work with SGLang, vLLM, TensorRT-LLM, PyTorch, and CUDA tooling, handling large checkpoints and distributed setups. The role offers a remote-first global environment and equity options.