Inference Runtime Engineer — On-Device & Cloud ML

Lm-Studio; New York (NY); Hybrid; USD 140,000 - 210,000; Full time

Benefits offered by this job

Competitive salary and equity grants

Excellent medical, vision, dental care

Catered team lunch / expensed dinners

Flexible PTO

Flexible WFH

Sun-drenched office in SoHo in NYC

Job summary

LM Studio in New York is seeking an Inference Runtime Software Engineer to advance our on-device and cloud inference stack. You will integrate new inference engines, optimize execution across CPU and GPU, and contribute to open-source projects.

You will also bring up new models and modalities, improve latency and memory use, and ensure reliability across diverse runtimes and hardware. Join a highly skilled team shaping human‑AI interactions.

Qualifications

• Significant experience building production ML systems, inference runtimes, or performance-sensitive infrastructure.
• Strong programming ability in Python and C++.
• Deep understanding of transformer architectures and the mechanics of model inference.
• Experience profiling CPU or GPU workloads and reasoning about compute, memory, synchronization, and data movement.
• Experience with PyTorch and inference systems such as llama.cpp, MLX, ExecuTorch, vLLM, SGLang, or TensorRT-LLM.
• Strong debugging instincts across model code, runtime internals, operating systems, and CPU or GPU execution.
• Takes personal responsibility for the correctness and performance of their work.

Responsibilities

• Maintain and push forward our inference stack on-device and in the cloud
• Bring up new model architectures and multimodal models
• Improve latency, throughput, memory use, and reliability across CPU, CUDA, Metal, Vulkan, and ROCm runtimes
• Build runtime capabilities for model loading, batching, scheduling, caching, and distributed execution
• Benchmark and diagnose correctness and performance problems across the inference stack
• Contribute upstream to open-source projects such as llama.cpp and MLX

Skills

Python

C++

Transformer models

Inference runtimes

Profiling

PyTorch

Tools

llama.cpp

MLX

ExecuTorch

vLLM

SGLang

TensorRT-LLM

Job description

LM Studio in New York is seeking an Inference Runtime Software Engineer to advance our on-device and cloud inference stack. You will integrate new inference engines, optimize execution across CPU and GPU, and contribute to open-source projects.

You will also bring up new models and modalities, improve latency and memory use, and ensure reliability across diverse runtimes and hardware. Join a highly skilled team shaping human‑AI interactions.