LLM Inference Engineer: Scale & Optimize Production

Comunidade Metodista; Palo Alto (CA); On-site; USD 150,000 - 210,000; Full time

Job summary

Hippocratic AI is seeking an experienced LLM Inference Engineer to optimize its LLM serving infrastructure in Palo Alto, CA. You will design multi-node architectures, implement multi-LoRA serving, and apply quantization techniques to reduce footprint while maintaining quality.

The role emphasizes practical production deployment, benchmarking, and latency-focused optimizations across GPUs, with a strong emphasis on Python, C++, and CUDA.

Qualifications

• 2+ years of experience optimizing LLM inference systems at scale.
• Proven expertise with distributed serving architectures for large language models.
• Hands-on experience implementing quantization techniques for transformer models.
• Strong understanding of modern inference optimization methods, including latency optimization strategies.
• Proficiency in Python and C++.
• Experience with CUDA programming and GPU optimization.

Responsibilities

• Design and implement multi-node serving architectures for distributed LLM inference.
• Optimize multi-LoRA serving systems.
• Apply advanced quantization techniques (FP4/FP6) to reduce model footprint while preserving quality.
• Implement speculative decoding and other latency optimization strategies.
• Develop disaggregated serving solutions with optimized caching strategies for prefill and decoding phases.
• Continuously benchmark and improve system performance across various deployment scenarios and GPU types.

Skills

Python

C++

CUDA

Distributed systems

Quantization

Latency optimization

Tools

vLLM

TensorRT-LLM

SGLang

Custom CUDA kernels

Job description

Hippocratic AI is seeking an experienced LLM Inference Engineer to optimize its LLM serving infrastructure in Palo Alto, CA. You will design multi-node architectures, implement multi-LoRA serving, and apply quantization techniques to reduce footprint while maintaining quality.

The role emphasizes practical production deployment, benchmarking, and latency-focused optimizations across GPUs, with a strong emphasis on Python, C++, and CUDA.