LLM Inference Engineer (Mid, Sr, Staff)

We Love AI Jobs; Menlo Park, Northern (CA, KY); Hybrid; USD 180,000 - 280,000; Full time

Benefits offered by this job

Health insurance

Equity

Job summary

Hippocratic AI in Menlo Park, CA is seeking an LLM Inference Engineer (Mid/Senior/Staff) to design and optimize multi-node serving infrastructures for healthcare language models. You will tackle distributed inference, quantization, speculative decoding, and caching to achieve sub-100ms responses across diverse GPU types.

Responsibilities include shipping major serving improvements within 90 days, deploying advanced architectures, and driving core optimization techniques that enhance latency,

Qualifications

• Must have experience optimizing LLM inference systems at scale.
• Experience with distributed serving architectures for large language models.
• Hands-on experience implementing quantization techniques for transformer models.
• Strong understanding of modern inference optimization methods, including speculative decoding techniques with draft models.
• Proficiency in Python and C++.
• Experience with CUDA programming and GPU optimization.

Responsibilities

• Design and implement multi-node serving architectures for distributed LLM inference.
• Optimize multi-LoRA serving systems.
• Apply advanced quantization techniques (FP4/FP6) to reduce model footprint while preserving quality.
• Implement speculative decoding and other latency optimization strategies.
• Develop disaggregated serving solutions with optimized caching strategies for prefill and decoding phases.
• Continuously benchmark and improve system performance across various deployment scenarios and GPU types.

Skills

Python

C++

CUDA

Distributed LLM serving

Quantization

Speculative decoding

vLLMSGLang

TensorRT-LLM

Tools

TensorRT-LLM

vLLMSGLang

CUDA toolkit

Job description

# LLM Inference Engineer (Mid, Sr, Staff)Hippocratic AIMenlo Park, CAFull-timeAI tools:vLLMSGLangTensorRT-LLMCUDAUpload your resume once and Scout applies for youApply with ScoutShare this job:LinkedInXSave imageThe LLM Inference Engineer builds and optimizes serving infrastructure for Hippocratic AI’s healthcare language models. The work spans distributed serving, quantization, speculative decoding, and benchmarking, with the goal of improving latency, throughput, cost, and reliability in production scenarios.* Permanent* On-site* Mid, Senior, or Staff### Skills and expertisePythonC++CUDADistributed LLM servingQuantizationSpeculative decodingvLLMSGLang### Key responsibilities* Design and implement multi-node serving architectures for distributed LLM inference.* Optimize multi-LoRA serving, quantization, speculative decoding, and caching strategies.* Benchmark serving performance across deployment scenarios and GPU types.## How this job comparesCompared with 329 open mid-level machine learning jobs on October 10, 2026.Pay: This job does not publish its pay. The 119 open mid-level machine learning jobs that publish US pay start at $180K a year on average.Similar open jobs: We Love AI Jobs lists 329 open mid-level machine learning jobs at 145 companies.Remote work: This job is on site. 66 of the 329 mid-level machine learning jobs (20%) are remote.AI in this role: High. AI is the core of the work: building, training or deploying AI systems. The role directly builds and optimizes LLM inference systems, including distributed serving, quantization, and decoding techniques. Hands-on model-serving and GPU optimization work is the core of the position.## Full descriptionRole MissionAs HAI's LLM Inference Engineer, you will own the serving infrastructure that determines whether our breakthrough healthcare AI reaches patients efficiently and reliably. You'll optimize the systems that translate raw model capability into sub-100ms responses—making the difference between conversational experiences that feel natural and those that feel broken. This role exists because inference optimization at scale is where research meets reality: your work directly determines latency, cost, and availability for millions of patient conversations across healthcare systems.What You Will AccomplishOwn your first major outcome: By day 90, you will have shipped a measurable improvement to our inference serving stack (reduced latency, improved throughput, or optimized cost per inference), validated the gains across our production deployment scenarios, and established the performance optimization roadmap that will guide infrastructure investment.Drive lasting impact: At 12 months, you will have designed and deployed advanced serving architectures (disaggregated inference, optimized caching, speculative decoding) that meaningfully improve patient experience and operational efficiency, contributed novel optimization techniques that become part of our core infrastructure, and made our serving stack a durable competitive advantage in healthcare AI deployment.The TeamYou'll work alongside systems engineers, ML researchers, and infrastructure experts who are obsessed with making AI systems fast, reliable, and cost-effective. This is a team that values deep technical rigor, continuous benchmarking, and solving hard systems problems that have real impact on patient experience and business unit economics.What You'll Do• Design and implement multi-node serving architectures for distributed LLM inference• Optimize multi-LoRA serving systems• Apply advanced quantization techniques (FP4/FP6) to reduce model footprint while preserving quality• Implement speculative decoding and other latency optimization strategies• Develop disaggregated serving solutions with optimized caching strategies for prefill and decoding phases• Continuously benchmark and improve system performance across various deployment scenarios and GPU typesLocation RequirementWe believe the best ideas happen together. This role is based in our Menlo Park, California office, expected to be in office five days a week.CompensationCompensation is based on experience, expertise, and level of responsibility. We offer competitive packages that reflect the seniority and scope of the role, along with equity, health insurance, and other benefits.What You BringMust-Have:• Experience optimizing LLM inference systems at scale• Proven expertise with distributed serving architectures for large language models• Hands-on experience implementing quantization techniques for transformer models• Strong understanding of modern inference optimization methods, including:• Speculative decoding techniques with draft models• Eagle speculative decoding approaches• Proficiency in Python and C++• Experience with CUDA programming and GPU optimizationNice-to-Have:• Contributions to open-source inference frameworks such as vLLM, SGLang, or TensorRT-LLM• Experience with custom CUDA kernels• Track record of deploying inference systems in production environments