Inference Infra Engineer: Scale Low-Latency AI Serving

Elorian; Palo Alto (CA); On-site; USD 200,000 - 400,000; Full time

Benefits offered by this job

Health, dental, and vision benefits

Unlimited PTO

Parental leave

Relocation support

Job summary

Elorian is seeking an infrastructure engineer in Palo Alto to design, optimize, and scale systems for large multimodal models. You will focus on fast, cost-efficient inference and reliable deployments across production environments.

You will implement multi-GPU and multi-node model parallelism, autoscaling, and observability standards, collaborating with researchers to advance novel architectures and achieve high-performance inference.

Qualifications

• 3+ years building low-latency inference serving systems for large models
• Strong knowledge of inference optimization techniques (quantization, batching, speculative decoding, KV cache)
• Hands-on experience with serving frameworks such as vLLM, TensorRT-LLM, Triton, or SGLang
• Experience with multi-GPU/multi-node model parallelism for serving (tensor or pipeline)
• Strong systems programming skills; C++/CUDA a plus alongside Python
• Experience with autoscaling and load balancing for production ML services
• A track record of GPU cost optimization at scale

Responsibilities

• Build low-latency, high-throughput inference serving systems for large multimodal models
• Design and implement latency, throughput, and efficiency improvements (quantization, batching, speculative decoding, KV cache management)
• Optimize codebase and GPU fleet for full hardware utilization
• Implement multi-GPU/multi-node model parallelism for serving (tensor/pipeline)
• Build autoscaling and load balancing for production ML services
• Establish reliability, observability, and reproducibility standards across the inference stack
• Collaborate with researchers to enable high-performance inference for novel architectures

Skills

Low-latency inference

Quantization

Multi-GPU/Node parallelism

C++/CUDA

Autoscaling & load balancing

GPU cost optimization

Tools

vLLM

TensorRT-LLM

Triton

SGLang

Job description

Elorian is seeking an infrastructure engineer in Palo Alto to design, optimize, and scale systems for large multimodal models. You will focus on fast, cost-efficient inference and reliable deployments across production environments.

You will implement multi-GPU and multi-node model parallelism, autoscaling, and observability standards, collaborating with researchers to advance novel architectures and achieve high-performance inference.