ML Infrastructure Engineer

Sovyren; United States; Remote; USD 150,000 - 230,000; Full time

Job summary

Sovyren is seeking an engineer to build and operate the systems that serve machine learning models at scale. The role is remote across the Americas or EMEA and full-time.

You will operate and scale GPU inference clusters on AWS, tune speculative decoding and batching, and build evaluation infrastructure to improve model serving performance. Strong CUDA skills and experience with vLLM/TensorRT-LLM, plus containerized workloads at scale with Kubernetes or ECS, are required to enable low-latency

Qualifications

• At least four years of experience in ML infrastructure or distributed systems.
• Practical experience with vLLM, TensorRT-LLM, or a comparable inference framework.
• Strong CUDA skills, including GPU profiling and memory optimization.
• Experience running containerized workloads at scale with Kubernetes, ECS, or similar.
• Familiarity with transformer architecture and attention optimization techniques.

Responsibilities

• Operate and scale GPU inference clusters on AWS, including p4d and p5 instances.
• Tune speculative decoding, continuous batching, and KV-cache policies to improve serving performance.
• Reduce time to first byte for tool-use requests across multiple model-serving paths.
• Build and maintain evaluation and benchmarking infrastructure.
• Create observability pipelines for latency percentiles, token throughput, and error rates.

Skills

ML infrastructure

Distributed systems

CUDA profiling

GPU memory optimization

Kubernetes / container runtimes

TensorRT-LLM / vLLM

Transformer architectures

Tools

AWS

Kubernetes

ECS

Docker

Linux

Job description

Role overview

Build and operate the systems that serve machine learning models at scale. The work spans GPU cluster operations, inference performance, low-latency request handling, and the tools used to evaluate model quality and speed. The position is remote across the Americas or EMEA and full-time.

Responsibilities

• Operate and scale GPU inference clusters on AWS, including p4d and p5 instances.


• Tune speculative decoding, continuous batching, and KV-cache policies to improve serving performance.


• Reduce time to first byte for tool-use requests across multiple model-serving paths.


• Build and maintain evaluation and benchmarking infrastructure.


• Create observability pipelines for latency percentiles, token throughput, and error rates.

Requirements

• At least four years of experience in ML infrastructure or distributed systems.


• Practical experience with vLLM, TensorRT-LLM, or a comparable inference framework.


• Strong CUDA skills, including GPU profiling and memory optimization.


• Experience running containerized workloads at scale with Kubernetes, ECS, or similar.


• Familiarity with transformer architecture and attention optimization techniques.