LLM Inference Engineer (Mid/Senior/Staff) Onsite

We Love AI Jobs; Menlo Park, Northern (CA, KY); Hybrid; USD 180,000 - 280,000; Full time

Benefits offered by this job

Health insurance

Equity

Job summary

Hippocratic AI in Menlo Park, CA is seeking an LLM Inference Engineer (Mid/Senior/Staff) to design and optimize multi-node serving infrastructures for healthcare language models. You will tackle distributed inference, quantization, speculative decoding, and caching to achieve sub-100ms responses across diverse GPU types.

Responsibilities include shipping major serving improvements within 90 days, deploying advanced architectures, and driving core optimization techniques that enhance latency,

Qualifications

• Must have experience optimizing LLM inference systems at scale.
• Experience with distributed serving architectures for large language models.
• Hands-on experience implementing quantization techniques for transformer models.
• Strong understanding of modern inference optimization methods, including speculative decoding techniques with draft models.
• Proficiency in Python and C++.
• Experience with CUDA programming and GPU optimization.

Responsibilities

• Design and implement multi-node serving architectures for distributed LLM inference.
• Optimize multi-LoRA serving systems.
• Apply advanced quantization techniques (FP4/FP6) to reduce model footprint while preserving quality.
• Implement speculative decoding and other latency optimization strategies.
• Develop disaggregated serving solutions with optimized caching strategies for prefill and decoding phases.
• Continuously benchmark and improve system performance across various deployment scenarios and GPU types.

Skills

Python

C++

CUDA

Distributed LLM serving

Quantization

Speculative decoding

vLLMSGLang

TensorRT-LLM

Tools

TensorRT-LLM

vLLMSGLang

CUDA toolkit

Job description

Hippocratic AI in Menlo Park, CA is seeking an LLM Inference Engineer (Mid/Senior/Staff) to design and optimize multi-node serving infrastructures for healthcare language models. You will tackle distributed inference, quantization, speculative decoding, and caching to achieve sub-100ms responses across diverse GPU types.

Responsibilities include shipping major serving improvements within 90 days, deploying advanced architectures, and driving core optimization techniques that enhance latency,