Senior Machine Learning Engineer

SmartChoice International GCC; United States; Remote; USD 150,000 - 220,000; Full time

Job summary

SmartChoice International GCC is seeking a Senior Machine Learning Engineer to optimize how AI models run in production, focusing on latency, throughput, and GPU efficiency. You will take models from research to reliable production deployment, working across inference engines and distributed serving infrastructure.

The role is hands-on and requires ownership of performance and reliability for high-volume, real-time workloads, with 5+ years in ML inference or closely related production

Qualifications

• PhD or equivalent practical systems engineering experience.
• Strong background in ML inference systems and real-time performance.

Responsibilities

• Optimize inference and model-serving pipelines for low latency and high throughput.
• Profile and resolve performance bottlenecks across GPU compute and memory usage.
• Build and scale distributed inference across multiple GPUs/nodes.
• Deploy and operate ML models in production with containerization.

Skills

C++

CUDA

Rust

Optimised Python

Education

PhD in CS/Physics/Math

Tools

Kubernetes

Ray

Load balancing

Multi-GPU inference

Job description

Location: California, or remote within the US, Canada, UK, Germany, Switzerland or Serbia

Employment: Full-time

About the Company

Our client is an AI research and infrastructure company building real-time voice models and inference technology for consumer applications across healthcare, education, fitness, entertainment and more. Backed by over $125 million in funding, the business develops speech models, serves large language models and builds the infrastructure required to deliver responsive AI experiences at scale.

The Opportunity

We are looking for a Senior Machine Learning Engineer to optimise how AI models run in production, with a focus on reducing latency, increasing throughput and improving GPU efficiency. You will take models from research through to reliable production deployment, working across inference engines, GPU optimisation and distributed serving infrastructure. This is a hands‑on engineering role with ownership of performance and reliability for high-volume, real‑time workloads.

Key Responsibilities

• Optimise inference and model‑serving pipelines using frameworks such as vLLM or TensorRT-LLM.


• Profile and resolve performance bottlenecks across GPU compute, memory usage and serving infrastructure.


• Apply acceleration techniques such as quantisation, speculative decoding, continuous batching, paged attention, caching and distillation.


• Build and scale distributed inference systems across multiple GPUs and nodes.


• Work with research teams to containerise, deploy and operate models in production.


• Improve latency, throughput, reliability and infrastructure efficiency while maintaining model quality.


• Develop serving infrastructure capable of supporting high concurrency and real‑time AI interactions.


• Take ownership of technical decisions, challenge architectural assumptions and deliver measurable production improvements.


• Typically 5+ years of experience in ML inference, systems engineering or closely related production infrastructure roles.


• Senior‑level engineering experience, with evidence of owning complex technical work.


• Hands‑on experience optimising inference using vLLM or TensorRT-LLM.


• Experience deploying and serving machine learning models in production.


• Strong programming skills in one or more of C++, CUDA, Rust or highly optimised Python.


• Practical experience profiling and optimising workloads on NVIDIA GPUs.


• Experience applying model acceleration or serving optimisation techniques.


• Understanding of distributed systems, with experience in areas such as Kubernetes, Ray, load balancing or multi‑GPU/multi‑node inference.


• Ability to work independently, navigate ambiguity and take systems through to stable production operation.


• Real‑time voice, speech or large language model inference.


• Open‑source contributions to inference engines or substantial systems programming projects.


• Technical publications or detailed engineering write-ups demonstrating practical expertise.


• A PhD in Computer Science, Physics, Mathematics or equivalent practical systems engineering experience.


• Experience working in a fast‑paced AI or infrastructure business.