LLM Inference Engineer (Mid, Senior, Staff)

Hippocratic AI Inc.; Menlo Park (CA); On-site; USD 180,000 - 280,000; Full time

Job summary

Hippocratic AI Inc. is seeking an LLM Inference Engineer to own the serving infrastructure enabling fast, reliable healthcare AI for patient conversations. You will optimize end-to-end latency (<100 ms) and cost, shaping the roadmap for scalable inference across production deployments.

By day 90 you will ship measurable improvements to the inference stack, validate gains, and outline a performance optimization plan. At 12 months you will deploy advanced serving architectures and contribute techniques that become core infra capabilities.

Qualifications

• Experience with CUDA programming and GPU optimization.
• Proficiency in Python and C++ for production systems.
• Hands-on experience implementing quantization techniques for transformer models.

Responsibilities

• Design and implement multi-node serving architectures for distributed LLM inference.
• Optimize multi-LoRA serving systems.
• Apply advanced quantization techniques to reduce model footprint while preserving quality.
• Implement speculative decoding and other latency optimization strategies.
• Develop disaggregated serving solutions with optimized caching for prefill and decoding phases.
• Continuously benchmark and improve system performance across deployment scenarios and GPU types.

Skills

CUDA programming

GPU optimization

Python

C++

Quantization

Speculative decoding

Distributed serving

Open-source frameworks

Tools

vLLM

SGLang

TensorRT-LLM

Job description

• As HAI’s LLM Inference Engineer, you will own the serving infrastructure that determines whether our breakthrough healthcare AI reaches patients efficiently and reliably
• You’ll optimize the systems that translate raw model capability into sub-100ms responses—making the difference between conversational experiences that feel natural and those that feel broken
• This role exists because inference optimization at scale is where research meets reality: your work directly determines latency, cost, and availability for millions of patient conversations across healthcare systems
• Own your first major outcome: By day 90, you will have shipped a measurable improvement to our inference serving stack (reduced latency, improved throughput, or optimized cost per inference), validated the gains across our production deployment scenarios, and established the performance optimization roadmap that will guide infrastructure investment
• Drive lasting impact: At 12 months, you will have designed and deployed advanced serving architectures (disaggregated inference, optimized caching, speculative decoding) that meaningfully improve patient experience and operational efficiency, contributed novel optimization techniques that become part of our core infrastructure, and made our serving stack a durable competitive advantage in healthcare AI deployment
• You’ll work alongside systems engineers, ML researchers, and infrastructure experts who are obsessed with making AI systems fast, reliable, and cost-effective. This is a team that values deep technical rigor, continuous benchmarking, and solving hard systems problems that have real impact on patient experience and business unit economics
• Design and implement multi-node serving architectures for distributed LLM inference
• Optimize multi-LoRA serving systems
• Apply advanced quantization techniques (FP4/FP6) to reduce model footprint while preserving quality
• Implement speculative decoding and other latency optimization strategies
• Develop disaggregated serving solutions with optimized caching strategies for prefill and decoding phases
• Continuously benchmark and improve system performance across various deployment scenarios and GPU types

• Experience with CUDA programming and GPU optimization
• Proficiency in Python and C++
• Hands-on experience implementing quantization techniques for transformer models
• Experience optimizing LLM inference systems at scale
• Speculative decoding techniques with draft models
• Proven expertise with distributed serving architectures for large language models
• Eagle speculative decoding approaches
• Strong understanding of modern inference optimization methods, including:
• Track record of deploying inference systems in production environments
• Contributions to open-source inference frameworks such as vLLM, SGLang, or TensorRT-LLM
• Deep understanding of performance optimization systems
• Experience with custom CUDA kernels