Senior AI Inference Gateway Engineer (Remote)

Salad; United States, Northern America, Americas; Hybride; USD 180 000 - 220 000; Plein temps

Avantages offerts par ce poste

Unlimited PTO

Health insurance coverage (75% Premium

Dental and vision coverage

401(k) plan

Stock options

Company-provided computer

Fully remote

Résumé du poste

Salad is seeking a Staff Software Engineer for the AI Inference Gateway to own the system end-to-end, from request routing to scaling across gateway servers. You will optimize fleet efficiency, operate inference nodes (vLLM, llama.cpp, etc.), and maintain observability with OpenTelemetry.

Expect on-call responsibilities and close collaboration with product and marketing teams to shape production deployments.

Qualifications

• Proven production Rust experience on high-throughput networked services.
• Hands-on experience running LLM inference servers (vLLM, llama.cpp, TGI, TensorRT-LLM, or similar).
• Ability to implement metrics, tracing, and on-call readiness.
• Willing to be on-call and treat pages as design bugs to fix.
• Strong written and verbal communication; capable of guiding others.

Responsabilités

• Own the AI Gateway system end to end: request routing, streaming, batch/async job handling, and horizontal scaling across multiple gateway servers.
• Build and tune the fleet-efficiency algorithms — autoscaling nodes, scoring performance, and evicting underperformers so the network converges on peak throughput per dollar.
• Operate and scale the underlying inference nodes running vLLM, llama.cpp, and similar servers.
• Design and maintain OpenTelemetry-based observability across the gateway and the fleet; pay-per-token and subscription billing already rides on this telemetry, so you'll keep that integration accurate.
• Spend roughly 25% of your time benchmarking new models, quantizations, and GPU hardware, and work with the product and marketing teams to make the call on what goes into production.
• Collaborate on cross-system design and integrations with the rest of Salad engineering; explain trade-offs clearly to both technical and non-technical teammates.

Connaissances

Rust

LLM inference servers

Metrics & tracing

On-call discipline

Clear communication

Outils

Pingora

Tokio

CUDA

ROCm

Description du poste

Salad is seeking a Staff Software Engineer for the AI Inference Gateway to own the system end-to-end, from request routing to scaling across gateway servers. You will optimize fleet efficiency, operate inference nodes (vLLM, llama.cpp, etc.), and maintain observability with OpenTelemetry.

Expect on-call responsibilities and close collaboration with product and marketing teams to shape production deployments.