Distributed Production Inference Engineer

Thinking Machines Lab Inc.; San Francisco (CA); On-site; USD 350,000 - 500,000; Full time

Benefits offered by this job

Health, dental, and vision benefits

Unlimited PTO

Paid parental leave

Relocation support

Job summary

Thinking Machines Lab Inc. is hiring a Software Engineer to design, build, and scale distributed production inference systems for Inkling, Inkling-Small, and Tinker.

You will own the end-to-end serving stack, from request routing to multi-node coordination and GPU utilization at scale. The role emphasizes production-first systems work, latency optimization, and deep observability, collaborating with research and infrastructure teams to translate evolving model architectures into reliable

Qualifications

• 3+ years building and operating distributed systems in production.
• Strong systems programming skills in Python, C++, Rust, or similar.
• Experience with production infrastructure at scale: reliability, observability, and performance under real-world load.
• Solid understanding of networking, concurrency, and distributed systems fundamentals.

Responsibilities

• Design, build, and operate distributed infrastructure for large-scale model serving, including request routing, load balancing, batching, and multi-node coordination
• Optimize inference latency and throughput in production, including work on KV cache management, continuous batching, speculative decoding, and quantization
• Build and maintain high-concurrency serving systems with strong uptime, low tail latency, and deep observability
• Benchmark, tune, and extend inference engines to support new model architectures as they move from research into production
• Partner with research and infrastructure teams to translate emerging model designs into production-ready serving systems
• Build tooling for tracing, debugging, and resolving issues across the serving stack, from orchestration down to GPU kernels
• Participate in on‑call rotation to support production inference systems

Skills

Distributed systems

Python

C++

Rust

Networking

Observability

Concurrency

Tools

Kubernetes

CUDA

TensorRT

vLLM

Job description

Thinking Machines Lab Inc. is hiring a Software Engineer to design, build, and scale distributed production inference systems for Inkling, Inkling-Small, and Tinker.

You will own the end-to-end serving stack, from request routing to multi-node coordination and GPU utilization at scale. The role emphasizes production-first systems work, latency optimization, and deep observability, collaborating with research and infrastructure teams to translate evolving model architectures into reliable