Senior LLM Inference Architect (On-Site Menlo Park)

Hippocratic AI Inc.; Menlo Park (CA); Presencial; USD 180.000 - 280.000; Jornada completa

Ventajas ofrecidas por este puesto de trabajo

Equity

Health insurance

Competitive compensation

Descripción de la vacante

Hippocratic AI Inc. is seeking an experienced LLM Inference Engineer to own and optimize its inference serving stack. You will drive sub-100ms responses, cost efficiency, and reliable availability for millions of patient conversations across healthcare systems.

You will design multi-node serving architectures and advance techniques like disaggregated serving and speculative decoding. You will collaborate with systems engineers, ML researchers, and infrastructure experts to push the boundaries of

Formación

• Experience optimizing LLM inference systems at scale.
• Proven expertise with distributed serving architectures for large language models.
• Hands-on experience implementing quantization techniques for transformer models.
• Strong understanding of modern inference optimization methods, including speculative decoding.

Responsabilidades

• Design and implement multi-node serving architectures for distributed LLM inference.
• Optimize multi-LoRA serving systems.
• Apply advanced quantization techniques (FP4/FP6) to reduce model footprint while preserving quality.
• Implement speculative decoding and other latency optimization strategies.
• Develop disaggregated serving solutions with optimized caching strategies for prefill and decoding phases.
• Continuously benchmark and improve system performance across deployment scenarios and GPU types.

Conocimientos

LLM inference optimization

Distributed serving architectures

Quantization techniques

Speculative decoding

Python

C++

Herramientas

CUDA programming

GPU optimization

Descripción del empleo

Hippocratic AI Inc. is seeking an experienced LLM Inference Engineer to own and optimize its inference serving stack. You will drive sub-100ms responses, cost efficiency, and reliable availability for millions of patient conversations across healthcare systems.

You will design multi-node serving architectures and advance techniques like disaggregated serving and speculative decoding. You will collaborate with systems engineers, ML researchers, and infrastructure experts to push the boundaries of