ML Serving Engineer – Real-Time Voice AI Platform

Unusual Ventures; San Francisco (CA); On-site; USD 180,000 - 240,000; Full time

Job summary

Rime is hiring a Software Engineer to own the serving infrastructure that connects inference engines to production. You will work at the intersection of ML systems and cloud infrastructure, building, hardening, and scaling real-time voice model serving.

You'll own the TTS serving stack, optimize multi-node serving, and ensure cross-hardware compatibility from HPC GPUs to cloud deployments. You will shape architecture, CI/CD, and reliability while collaborating with ML teams.

Qualifications

• Hands-on experience with real-time multinode ML serving infrastructure.
• Experience with distributed or disaggregated model serving.
• Strong cloud infrastructure fundamentals and IaC tooling.

Responsibilities

• Architect and implement the TTS serving infrastructure across GPUs and API surface.
• Optimize models from single-node to disaggregated fleet serving.
• Ensure compatibility across NVIDIA hardware for on-prem and cloud deployments.
• Maintain CI/CD workflows for the model serving pipeline.
• Manage site reliability: on-call, monitoring, and observability.
• Plan and manage GPU resource provisioning and costs.

Skills

Real-time ML serving

NVIDIA Dynamo/Triton

vLLM

SGLang

Distributed model serving

Cloud infrastructure

Linux

Docker

Kubernetes

Terraform

Tools

Docker

Kubernetes

Terraform

Packer

Job description

Rime is hiring a Software Engineer to own the serving infrastructure that connects inference engines to production. You will work at the intersection of ML systems and cloud infrastructure, building, hardening, and scaling real-time voice model serving.

You'll own the TTS serving stack, optimize multi-node serving, and ensure cross-hardware compatibility from HPC GPUs to cloud deployments. You will shape architecture, CI/CD, and reliability while collaborating with ML teams.