ML Engineer: Ultra-Low-Latency Inference & Speech Serving

IntelliPro Group; San Francisco (CA); Hybrid; USD 170,000 - 320,000; Full time

Benefits offered by this job

Competitive salary

Healthcare (medical, dental, vision)

401(k) with company match

Unlimited PTO and paid holidays

New parent leave

Hybrid office

Equipment provided

Job summary

IntelliPro Group is seeking a Machine Learning Engineer focused on Inference & Serving (Speech LLM) in San Francisco, CA, hybrid. You will build ultra-low-latency inference systems for large language and speech models, balancing latency and throughput across a GPU cluster.

You will collaborate with core ML training and backend infra teams to push performance, efficiency, and scalability, including real-time audio streaming and advanced decoding techniques. A strong systems mindset is essential.

Qualifications

• Hands-on experience building high-throughput, ultra-low-latency inference engines for LLMs or foundational speech models.
• Understanding latency, throughput and Time-To-First-Token tradeoffs in real-time streaming.
• Experience with multi-GPU multi-node inference pipelines and autoscaling with Kubernetes.

Responsibilities

• Design and deploy high-throughput, ultra-low-latency inference systems for LLMs and speech models.
• Collaborate with ML training and backend infrastructure teams.
• Optimize GPU utilization and memory hierarchy to remove bottlenecks.
• Implement advanced inference techniques to reduce latency and improve throughput.

Skills

Low-latency inference

Distributed GPUs

Real-time streaming

WebSocket/WebRTC

Tools

NVIDIA Triton

vLLM

TensorRT-LLM

Kubernetes

Job description

IntelliPro Group is seeking a Machine Learning Engineer focused on Inference & Serving (Speech LLM) in San Francisco, CA, hybrid. You will build ultra-low-latency inference systems for large language and speech models, balancing latency and throughput across a GPU cluster.

You will collaborate with core ML training and backend infra teams to push performance, efficiency, and scalability, including real-time audio streaming and advanced decoding techniques. A strong systems mindset is essential.