Kubernetes ML Inference Engineer: Model Serving

Abridge; San Francisco (CA); On-site; USD 221,000 - 260,000; Full time

Benefits offered by this job

14 paid holidays

Flexible PTO

Health plans

HSA contributions

Parental leave

401(k) matching

Personal device allowance

FSA and commuter benefits

Sabbatical after 5 years

Equity

Job summary

Abridge is seeking an ML Infrastructure Engineer, Model Inference in San Francisco to build and optimize the core inference infrastructure powering our AI‑driven healthcare solutions. You will collaborate across Infrastructure and Research to deploy, optimize, and orchestrate AI models at scale.

Ideal candidates have 2+ years of production ML infrastructure experience, strong Kubernetes know‑how, and a track record of engineering scalable APIs and distributed systems for real-time workloads.

Qualifications

• 2+ years building and deploying machine learning models in production.
• Deep understanding of container orchestration and distributed systems architecture.
• Experience developing APIs and managing distributed systems for both batch and real-time workloads.

Responsibilities

• Design, deploy and maintain scalable Kubernetes clusters for AI model inference and training.
• Develop, optimize, and maintain ML model serving infrastructure, ensuring high-performance and low-latency.
• Collaborate with ML and product teams to scale backend infrastructure for AI-driven products, focusing on model deployment, throughput optimization, and compute efficiency.
• Optimize compute-heavy workflows and enhance GPU utilization for ML workloads.
• Build a robust model API orchestration system.
• Collaborate with leadership to define and implement strategies for scaling infrastructure as the company grows, ensuring long-term efficiency and performance.

Skills

Kubernetes administration

APIs development

Distributed systems

Model inference

Compute optimization

Collaboration with ML/product teams

Tools

NVIDIA Triton Server

VLLM

TRT-LLM

Terraform

Ansible

GitOps

Job description

Abridge is seeking an ML Infrastructure Engineer, Model Inference in San Francisco to build and optimize the core inference infrastructure powering our AI‑driven healthcare solutions. You will collaborate across Infrastructure and Research to deploy, optimize, and orchestrate AI models at scale.

Ideal candidates have 2+ years of production ML infrastructure experience, strong Kubernetes know‑how, and a track record of engineering scalable APIs and distributed systems for real-time workloads.