Distributed Inference Systems Engineer

Speedrun Talent Network; San Francisco (CA); On-site; USD 350,000 - 500,000; Full time

Benefits offered by this job

Health, dental, vision benefits

Unlimited PTO

Parental leave

Relocation support

Job summary

Thinking Machines is hiring a Software Engineer to build and scale the distributed production inference systems powering Inkling, Inkling-Small, and Tinker. You’ll own the systems that turn trained models into fast, reliable services—from request routing and batching to multi-node serving and GPU utilization at scale.

This is a systems-heavy, production-first role. You’ll work closely with research and infrastructure teams to translate rapidly evolving model architectures into serving systems

Qualifications

• 3+ years building and operating distributed systems in production.
• Strong systems programming skills in Python, C++, Rust, or similar languages.
• Experience with production infrastructure at scale: reliability, observability, and performance under real-world load.
• Solid understanding of networking, concurrency, and distributed systems fundamentals.

Responsibilities

• Design, build, and operate distributed infrastructure for large-scale model serving, including request routing, load balancing, batching, and multi-node coordination
• Optimize inference latency and throughput in production, including work on KV cache management, continuous batching, speculative decoding, and quantization
• Build and maintain high-concurrency serving systems with strong uptime, low tail latency, and deep observability
• Benchmark, tune, and extend inference engines to support new model architectures as they move from research into production
• Partner with research and infrastructure teams to translate emerging model designs into production-ready serving systems
• Build tooling for tracing, debugging, and resolving issues across the serving stack, from orchestration down to GPU kernels
• Participate in on‑call rotation to support production inference systems

Skills

Distributed systems

Systems programming

Production infrastructure

Networking & concurrency

Job description

Thinking Machines is hiring a Software Engineer to build and scale the distributed production inference systems powering Inkling, Inkling-Small, and Tinker. You’ll own the systems that turn trained models into fast, reliable services—from request routing and batching to multi-node serving and GPU utilization at scale.

This is a systems-heavy, production-first role. You’ll work closely with research and infrastructure teams to translate rapidly evolving model architectures into serving systems