Production Inference Systems Engineer

Thinking Machines Lab Inc.; San Francisco (CA); On-site; USD 350,000 - 500,000; Full time

Benefits offered by this job

Health benefits

Dental benefits

Vision benefits

Unlimited PTO

Parental leave

Relocation assistance

Job summary

Thinking Machines Lab Inc. is hiring a Software Engineer to build scalable distributed production inference systems for Inkling, Inkling-Small, and Tinker.

You will own systems that turn trained models into fast, reliable services, handling routing, batching, multi-node serving, and GPU utilization at scale. This is a systems-heavy, production-first role collaborating with research and infrastructure teams to meet latency and throughput targets, with on-call rotation and robust observability

Qualifications

• 3 years of experience building and operating distributed systems in production.
• Strong systems programming skills in Python, C, Rust, or similar languages.
• Experience with production infrastructure at scale: reliability, observability, and performance under real-world load.

Responsibilities

• Design, build, and operate distributed infrastructure for large-scale model serving, including request routing, load balancing, batching, and multi-node coordination.
• Optimize inference latency and throughput in production, including KV cache management and quantization.
• Build and maintain high-concurrency serving systems with uptime and observability.
• Benchmark and extend inference engines to support new model architectures as they move to production.
• Collaborate with research and infrastructure teams to translate model designs into production-ready serving systems.
• Develop tooling for tracing, debugging, and resolving issues across the serving stack.
• Participate in on-call rotations to support production inference systems.

Skills

Distributed systems

Python

C/Rust

Tools

Kubernetes

TensorRT

Job description

Thinking Machines Lab Inc. is hiring a Software Engineer to build scalable distributed production inference systems for Inkling, Inkling-Small, and Tinker.

You will own systems that turn trained models into fast, reliable services, handling routing, batching, multi-node serving, and GPU utilization at scale. This is a systems-heavy, production-first role collaborating with research and infrastructure teams to meet latency and throughput targets, with on-call rotation and robust observability