Applied ML Engineer - Real-time LLM Router (Remote)

Cast AI; United States; Remote; USD 180,000 - 240,000; Full time

Benefits offered by this job

Flexible remote environment

Equity options

Learning budget

Annual hackathons

Equipment budget

Job summary

Cast AI, based in Miami with a globally distributed team, is seeking an experienced ML engineer to own the auto-router for LLM workloads. You will design training objectives, implement evaluation gates, and drive a feedback loop from live traffic to continuously improve routing decisions.

You will run production-grade experiments, calibrate models, and collaborate with platform engineers to instrument data paths, ensuring latency-aware, cost-effective routing while maintaining high quality.

Qualifications

• 5+ years in applied ML or ML research with real-time decision models
• Strong Python and SQL at scale with large datasets
• Experience shipping models in production with measurable impact
• Knowledge of LLM benchmarks and routing approaches
• Understanding of serving considerations (TTFT, batching, caching) and instrumentation

Responsibilities

• Own the routing model end to end: features, objective, training, calibration, and release cadence.
• Build evaluation gates: production-parity benchmarks, router-vs-router comparisons, holdouts by task family, error bars on pass rates.
• Turn fleet data into training signal with failure categorization to avoid poisoned labels reaching the policy.
• Model objective to reflect user experience: task success, token cost, and latency under concurrency.
• Run online loops: shadow, A/B, full rollout, drift monitoring, retrain.
• Investigate production offline-eval disagreements before next ship.
• Push frontier with calibrated decisions, abstention, and speed-aware routing.
• Collaborate with platform engineers on instrumentation and data-path support.

Skills

Python

SQL

ML engineering

LLM routing

A/B testing

Observability

Optimization

PyTorch

Tools

ClickHouse

GitLab CI

AWS/GCP

Job description

Cast AI, based in Miami with a globally distributed team, is seeking an experienced ML engineer to own the auto-router for LLM workloads. You will design training objectives, implement evaluation gates, and drive a feedback loop from live traffic to continuously improve routing decisions.

You will run production-grade experiments, calibrate models, and collaborate with platform engineers to instrument data paths, ensuring latency-aware, cost-effective routing while maintaining high quality.