ML Infrastructure Engineer — Scalable AI Inference

Abridge; San Francisco (CA); On-site; USD 170,000 - 210,000; Full time

Benefits offered by this job

Generous Time Off

Comprehensive Health Plans

Generous 401(k) Matching

Mental Health Support

Parental Leave

Lifestyle Wallet

Sabbatical Leave

Job summary

Abridge is seeking an ML Infrastructure Engineer to build and optimize the core inference infrastructure powering our AI models. You will work with Infrastructure and Research teams to deploy, optimize and orchestrate AI models at scale.

You'll design scalable Kubernetes clusters, develop low-latency ML serving, and collaborate with ML and product teams to improve throughput and compute efficiency. Strong emphasis on GPU utilization and CI/CD practices.

Qualifications

• 5+ years of experience in building and deploying machine learning models in production environments.
• Deep understanding of container orchestration and distributed systems architecture.
• Expertise in Kubernetes administration, including CRDs, operators, and cluster management
• Experience developing APIs and managing distributed systems for both batch and real-time workloads
• Excellent communication skills, interfacing between research and product engineering

Responsibilities

• Design, deploy and maintain scalable Kubernetes clusters for AI model inference and training
• Develop, optimize, and maintain ML model serving infrastructure, ensuring high-performance and low-latency
• Collaborate with ML and product teams to scale backend infrastructure for AI-driven products, focusing on model deployment, throughput optimization, and compute efficiency
• Optimize compute-heavy workflows and enhance GPU utilization for ML workloads
• Build a robust model API orchestration system
• Collaborate with leadership to define and implement strategies for scaling infrastructure as the company grows, ensuring long-term efficiency and performance

Skills

Kubernetes administration

Distributed systems

API development

Communication

CUDA optimization

Container orchestration

Tools

NVIDIA Triton Server

VLLM

TRT-LLM

PyTorch

TensorFlow

CUDA

Terraform

Ansible

Job description

Abridge is seeking an ML Infrastructure Engineer to build and optimize the core inference infrastructure powering our AI models. You will work with Infrastructure and Research teams to deploy, optimize and orchestrate AI models at scale.

You'll design scalable Kubernetes clusters, develop low-latency ML serving, and collaborate with ML and product teams to improve throughput and compute efficiency. Strong emphasis on GPU utilization and CI/CD practices.