ML Platform Engineer

Talenthop; United States; On-site; USD 100,000 - 160,000; Full time

Job summary

TalentHop is partnering to fill a senior ML Platform Engineer role in the United States. You will design and run high-performance model serving platforms for diverse workloads, balancing latency, throughput, and cost across production systems.

You will optimize deployments, implement autoscaling, manage GPU resources, integrate with API gateways, and lead reliability improvements. Candidates should have 10+ years in distributed infra and ML platforms.

Qualifications

• Bachelor's or Master's degree in Computer Science or related field.
• 10+ years of experience in distributed systems, infrastructure, or ML platform engineering.
• Proficiency in Python and a systems programming language (Go, Rust, or C++).
• Experience operating high-throughput, low-latency production services.
• Hands-on experience with large language model inference frameworks.
• Strong understanding of GPU architecture and accelerator utilization.
• Familiarity with Kubernetes, autoscaling, and cloud platforms.
• Experience with observability tools including metrics, tracing, and logging.
• Expertise in performance engineering and capacity planning.
• Strong communication and incident response abilities.

Responsibilities

• Design and operate model serving platforms for various workloads including language models, vision models, and recommendations.
• Optimize inference performance through batching, speculative decoding, and request multiplexing.
• Implement multi-tenant routing, rate limiting, and QoS policies.
• Develop autoscaling and capacity management systems balancing latency, throughput, and cost.
• Tune GPU utilization and memory management for large model serving.
• Integrate model serving with API gateways, identity systems, and observability tools.
• Implement caching, prompt deduplication, and response reuse strategies.
• Manage end-to-end observability covering latency, queue dynamics, GPU utilization, and error tracking.
• Develop deployment workflows with canary releases and automated rollback.
• Lead incident response and reliability improvements for AI services.
• Collaborate with ML and product teams on new model releases and features.
• Implement security controls like request signing and abuse detection.
• Document operational procedures, performance metrics, and tuning guidance.
• Stay current with AI serving research and apply advancements to production.

Skills

Distributed systems

High throughput

Low latency

Performance engineering

Python

Go

Rust

C++

Incident response

Education

Bachelor's or Master's degree in Computer Science or related field

Tools

Kubernetes

Cloud platforms

Observability tools

Large language model inference frameworks

Job description

The organization operates in the technology consulting and software development industry, specializing in delivering cloud, artificial intelligence, data, and enterprise solutions across the United States. It addresses the challenge of integrating advanced AI and software technologies into business environments, supporting clients with scalable and reliable solutions.

About the Opportunity:

The ML PLATFORM ENGINEER is responsible for designing, building, and operating high-performance inference platforms for large machine learning models in production. This role focuses on optimizing AI deployment systems to improve efficiency, reliability, and scalability. The position contributes significantly by ensuring the organization can deliver robust AI services that balance latency, throughput, cost, and quality, thereby supporting diverse model workloads and advancing AI infrastructure capabilities.

Responsibilities:

• Design and operate model serving platforms for various workloads including language models, vision models, and recommendation systems.
• Optimize inference performance through techniques such as batching, speculative decoding, and request multiplexing.
• Implement multi-tenant routing, rate limiting, and quality-of-service policies.
• Develop autoscaling and capacity management systems balancing latency, throughput, and cost.
• Tune GPU utilization, memory management, and caching strategies for large model serving.
• Integrate model serving with API gateways, identity systems, and observability tools.
• Implement caching, prompt deduplication, and response reuse strategies.
• Manage end-to-end observability covering latency, queue dynamics, GPU utilization, and error tracking.
• Develop deployment workflows including canary releases and automated rollback.
• Lead incident response and reliability improvements for AI services.
• Collaborate with ML and product teams on new model releases and features.
• Implement security controls like request signing, content filtering, and abuse detection.
• Document operational procedures, performance metrics, and tuning guidance.
• Stay current with AI serving research and apply advancements to production.

Requirements:

• Bachelor's or Master's degree in Computer Science or related field.
• Over 10 years of experience in distributed systems, infrastructure, or ML platform engineering.
• Proficiency in Python and a systems programming language such as Go, Rust, or C++.
• Experience operating high-throughput, low-latency production services.
• Hands-on experience with large language model inference frameworks.
• Strong understanding of GPU architecture and accelerator utilization.
• Familiarity with Kubernetes, autoscaling, and cloud platforms.
• Experience with observability tools including metrics, tracing, and logging.
• Expertise in performance engineering and capacity planning.
• Strong communication and incident response abilities.

Pay Range and Compensation Package:

• Salary range: $100,000 to $160,000 annually.

Equal Opportunity Statement:

Our client is an equal opportunity employer. They celebrate diversity and are committed to creating an inclusive environment for all employees. All qualified applicants will receive consideration for employment without regard to race, color, religion, gender, gender identity or expression, sexual orientation, or national origin.

Note:

TalentHop is a recruitment partner of this role. Please note that all employment decisions, including candidate assessment, interviews, hiring, compensation, and employment terms, are made exclusively by the hiring employer.