Remote Python Inference Engineer for Production ML Latency

HellaGoodAudio; United States; Hybrid; USD 140,000 - 230,000; Full time

Benefits offered by this job

Competitive compensation

Flexible working hours and hybrid or 远

Work from anywhere up to 45 days/year

Private medical insurance for you and/

Extra paid vacation and sick leave

Language courses

Modern offices with snacks and drinks

Team sports and social activities

Job summary

Gcore is seeking an experienced inference engineer to build and optimize the inference layer of the Gcore Inference platform. You will integrate frameworks like vLLM, SGLang, NVIDIA Dynamo, and TensorRT-LLM to bring new language and multimodal models into production.

You will focus on latency, throughput, memory usage, and GPU utilization, collaborating with platform, infra, product, and customer teams, and contributing to open-source projects where appropriate.

Qualifications

• 5+ years of experience writing reliable, well-tested production code.
• Strong Python skills and experience designing production systems.
• Hands-on experience with PyTorch and deploying machine learning models.
• Experience with Linux, Docker, and Kubernetes.
• Experience in distributed systems, GPU computing, ML runtimes, model optimization, or cluster scheduling.

Responsibilities

• Build and improve the inference layer of the Gcore Inference platform.
• Integrate and operate inference frameworks such as vLLM, SGLang, NVIDIA Dynamo, and TensorRT-LLM.
• Bring new language and multimodal models into production.
• Improve inference latency, throughput, memory use, GPU utilization, and cost efficiency.
• Debug performance and reliability issues across model code, inference frameworks, GPU execution, networking, and Kubernetes.
• Collaborate with platform, infrastructure, product, and customer-facing teams to turn inference improvements into reliable product features.
• Contribute improvements to open-source inference projects when appropriate.

Skills

Python

PyTorch

Linux

Docker

Kubernetes

Distributed systems

GPU computing

Debugging

Communication

Willingness to learn

Tools

vLLM

SGLang

NVIDIA Dynamo

TensorRT-LLM

CUDA

Triton

Job description

Gcore is seeking an experienced inference engineer to build and optimize the inference layer of the Gcore Inference platform. You will integrate frameworks like vLLM, SGLang, NVIDIA Dynamo, and TensorRT-LLM to bring new language and multimodal models into production.

You will focus on latency, throughput, memory usage, and GPU utilization, collaborating with platform, infra, product, and customer teams, and contributing to open-source projects where appropriate.