ML Infra Engineer | Scalable Inference & Kubernetes

Fdetimes; San Francisco (CA); On-site; USD 180,000 - 250,000; Full time

Benefits offered by this job

14 paid holidays

Health insurance

HSA contributions

Parental leave

401(k) matching

Personal device allowance

FSA & commuter benefits

Sabbatical leave

Job summary

Abridge is hiring an ML Infrastructure Engineer to build and optimize the core inference infrastructure powering our AI models. You’ll collaborate with Infrastructure and Research to deploy, scale, and orchestrate model workflows across GPUs and clusters.

You’ll help design scalable Kubernetes architectures, implement low-latency model serving, and drive efficiency across distributed systems. Join a fast-growing team shaping healthcare AI solutions.

Qualifications

• 5+ years of experience building and deploying ML models in production environments.
• Deep understanding of container orchestration and distributed systems architecture.
• Expertise in Kubernetes administration, including CRDs, operators, and cluster management.
• Experience developing APIs and managing distributed systems for batch and real-time workloads.
• Excellent communication skills interfacing between research and product engineering.

Responsibilities

• Design, deploy and maintain scalable Kubernetes clusters for AI model inference and training.
• Develop, optimize, and maintain ML model serving infrastructure for high performance and low latency.
• Collaborate with ML and product teams to scale backend infrastructure for AI-driven products, focusing on deployment, throughput, and compute efficiency.
• Optimize compute-heavy workflows and enhance GPU utilization for ML workloads.
• Build a robust model API orchestration system.
• Collaborate with leadership to scale infrastructure as the company grows for long-term efficiency.

Skills

ML in production

Container orchestration

Kubernetes

APIs & distributed systems

Communication

Tools

NVIDIA Triton

VLLM

TRT-LLM

PyTorch

TensorFlow

Terraform

Ansible

Job description

Abridge is hiring an ML Infrastructure Engineer to build and optimize the core inference infrastructure powering our AI models. You’ll collaborate with Infrastructure and Research to deploy, scale, and orchestrate model workflows across GPUs and clusters.

You’ll help design scalable Kubernetes architectures, implement low-latency model serving, and drive efficiency across distributed systems. Join a fast-growing team shaping healthcare AI solutions.