Senior ML Platform Engineer - Scalable AI Inference

Talenthop; United States; On-site; USD 100,000 - 160,000; Full time

Job summary

TalentHop is partnering to fill a senior ML Platform Engineer role in the United States. You will design and run high-performance model serving platforms for diverse workloads, balancing latency, throughput, and cost across production systems.

You will optimize deployments, implement autoscaling, manage GPU resources, integrate with API gateways, and lead reliability improvements. Candidates should have 10+ years in distributed infra and ML platforms.

Qualifications

• Bachelor's or Master's degree in Computer Science or related field.
• 10+ years of experience in distributed systems, infrastructure, or ML platform engineering.
• Proficiency in Python and a systems programming language (Go, Rust, or C++).
• Experience operating high-throughput, low-latency production services.
• Hands-on experience with large language model inference frameworks.
• Strong understanding of GPU architecture and accelerator utilization.
• Familiarity with Kubernetes, autoscaling, and cloud platforms.
• Experience with observability tools including metrics, tracing, and logging.
• Expertise in performance engineering and capacity planning.
• Strong communication and incident response abilities.

Responsibilities

• Design and operate model serving platforms for various workloads including language models, vision models, and recommendations.
• Optimize inference performance through batching, speculative decoding, and request multiplexing.
• Implement multi-tenant routing, rate limiting, and QoS policies.
• Develop autoscaling and capacity management systems balancing latency, throughput, and cost.
• Tune GPU utilization and memory management for large model serving.
• Integrate model serving with API gateways, identity systems, and observability tools.
• Implement caching, prompt deduplication, and response reuse strategies.
• Manage end-to-end observability covering latency, queue dynamics, GPU utilization, and error tracking.
• Develop deployment workflows with canary releases and automated rollback.
• Lead incident response and reliability improvements for AI services.
• Collaborate with ML and product teams on new model releases and features.
• Implement security controls like request signing and abuse detection.
• Document operational procedures, performance metrics, and tuning guidance.
• Stay current with AI serving research and apply advancements to production.

Skills

Distributed systems

High throughput

Low latency

Performance engineering

Python

Go

Rust

C++

Incident response

Education

Bachelor's or Master's degree in Computer Science or related field

Tools

Kubernetes

Cloud platforms

Observability tools

Large language model inference frameworks

Job description

TalentHop is partnering to fill a senior ML Platform Engineer role in the United States. You will design and run high-performance model serving platforms for diverse workloads, balancing latency, throughput, and cost across production systems.

You will optimize deployments, implement autoscaling, manage GPU resources, integrate with API gateways, and lead reliability improvements. Candidates should have 10+ years in distributed infra and ML platforms.