Staff Engineer - Large-Scale Model Inference & Systems

Xai; Palo Alto (CA); On-site; USD 180,000 - 440,000; Full time

Job summary

SpaceXAI is building a high-performance inference platform serving Grok at global scale. You will design and optimize distributed model-serving systems, owning from global KV caches to low-level GPU kernel work and code generation.

This role drives latency, throughput, and reliability for billions of users, with opportunities to push innovations in batching, quantization, and speculative decoding on next-gen hardware.

Qualifications

• Deep low-level systems programming.
• Experience with large-scale, high-concurrent production serving.
• Experience with GPU inference engines (vLLM, SGLang, Triton, TensorRT-LLM, etc.).
• Strong background in system optimizations: batching, caching, load balancing, parallelism.
• Low-level inference optimizations: GPU kernels, code generation.
• Algorithmic inference optimizations: quantization, speculative decoding, distillation, low-precision numerics.
• Experience designing and implementing CI/CD infrastructure for inference.

Responsibilities

• Architect and implement scalable distributed infrastructure for model serving (load balancing, auto-scaling, batch scheduling, global KV cache).
• Optimize latency and throughput of model inference under real production workloads.
• Build reliable, high-concurrency serving systems that serve billions of users with 100% uptime, 0% error rate, and excellent tail latency.
• Benchmark, fine-tune, and accelerate inference engines (including low-level GPU kernel work and code generation).
• Develop custom tools to trace, replay, and fix issues across the full stack — from orchestration down to GPU kernels.
• Create robust CI/CD infrastructure for seamless endpoint deployment, image publishing, and inference engine updates.
• Accelerate research on scaling test-time compute, RL rollout, and model-hardware co-design for next-generation systems.

Skills

Low-level systems

C/C++ or Rust

Large-scale serving

GPU inference

Performance optimization

Batching & caching

Tools

GPU kernels

Code generation

Auto-scaling

Distributed infra

Job description

SpaceXAI is building a high-performance inference platform serving Grok at global scale. You will design and optimize distributed model-serving systems, owning from global KV caches to low-level GPU kernel work and code generation.

This role drives latency, throughput, and reliability for billions of users, with opportunities to push innovations in batching, quantization, and speculative decoding on next-gen hardware.