Fast, Scalable LLM Inference Engineer — Healthcare AI

Hippocratic AI Inc.; Menlo Park (CA); On-site; USD 180,000 - 280,000; Full time

Job summary

Hippocratic AI Inc. is seeking an LLM Inference Engineer to own the serving infrastructure enabling fast, reliable healthcare AI for patient conversations. You will optimize end-to-end latency (<100 ms) and cost, shaping the roadmap for scalable inference across production deployments.

By day 90 you will ship measurable improvements to the inference stack, validate gains, and outline a performance optimization plan. At 12 months you will deploy advanced serving architectures and contribute techniques that become core infra capabilities.

Qualifications

• Experience with CUDA programming and GPU optimization.
• Proficiency in Python and C++ for production systems.
• Hands-on experience implementing quantization techniques for transformer models.

Responsibilities

• Design and implement multi-node serving architectures for distributed LLM inference.
• Optimize multi-LoRA serving systems.
• Apply advanced quantization techniques to reduce model footprint while preserving quality.
• Implement speculative decoding and other latency optimization strategies.
• Develop disaggregated serving solutions with optimized caching for prefill and decoding phases.
• Continuously benchmark and improve system performance across deployment scenarios and GPU types.

Skills

CUDA programming

GPU optimization

Python

C++

Quantization

Speculative decoding

Distributed serving

Open-source frameworks

Tools

vLLM

SGLang

TensorRT-LLM

Job description

Hippocratic AI Inc. is seeking an LLM Inference Engineer to own the serving infrastructure enabling fast, reliable healthcare AI for patient conversations. You will optimize end-to-end latency (<100 ms) and cost, shaping the roadmap for scalable inference across production deployments.

By day 90 you will ship measurable improvements to the inference stack, validate gains, and outline a performance optimization plan. At 12 months you will deploy advanced serving architectures and contribute techniques that become core infra capabilities.