Senior Inference Platform Engineer - Data Center

Hamilton Barnes Associates Limited; San Francisco (CA); On-site; USD 270,000 - 330,000; Full time

Benefits offered by this job

Equity

Job summary

A technology recruitment agency seeks a professional to lead the architecture of a distributed inference platform within a stealth-mode hyperscale data center startup. Applicants must have over 5 years of experience with large-scale distributed systems and strong proficiency in languages such as Python, Go, or Rust, along with a solid understanding of GPU software stacks. This unique opportunity offers an annual salary of $300,000 and includes equity benefits.

Qualifications

• 5+ years’ experience building large-scale, fault-tolerant distributed systems.
• Strong understanding of GPU software stacks (CUDA, Triton, NCCL) and Kubernetes.
• Practical experience with model-serving frameworks such as vLLM, SGLang, TensorRT-LLM.

Responsibilities

• Take ownership of the inference platform architecture.
• Design, build, and optimise distributed inference systems.
• Integrate, tune, and operate inference engines across multiple model types.
• Develop APIs and orchestration layers for multi-tenant and dedicated deployments.

Skills

Building large-scale distributed systems

Proficiency in Python, Go, Rust

Understanding of GPU software stacks

Experience with model-serving frameworks

Knowledge of performance optimisation techniques

Familiarity with Infrastructure-as-Code tools

Job description

Join a stealth-mode hyperscale data center startup building an AI and cloud platform, powered by thousands of H100s, H200s, and B200s, ready to go for experimentation, full-scale model training, or inference.

Our client operates high-performance GPU clusters powering some of the most advanced AI workloads worldwide. They’re now building a serverless inference platform, beginning with cost-efficient batch inference and expanding into low-latency, real-time inference and custom model hosting. This is a unique chance to join at an early stage and help define the architecture, scalability, and technical direction of that platform.

If you are interested in this opportunity, get in tuch! You don't want to miss this opportunity!

Key Responsibilities

• Take ownership of the inference platform architecture, from batch to low-latency workloads.
• Design, build, and optimise distributed inference systems to maximise GPU utilisation and minimise cold starts.
• Integrate, tune, and operate inference engines such as vLLM, SGLang, and TensorRT-LLM across multiple model types.
• Develop APIs, orchestration layers, and autoscaling logic to support both multi-tenant and dedicated deployments.
• Collaborate with cross-functional teams to translate business and customer needs into robust technical solutions.
• Stay up to date with the latest models, serving frameworks, and optimisation techniques, applying best practices in performance and efficiency.
• Implement monitoring, alerting, and observability workflows for production systems.

Requirements

• 5+ years’ experience building large-scale, fault-tolerant distributed systems (ML inference, HPC, or similar).
• Proficiency in Python, Go, Rust, or a comparable language.
• Strong understanding of GPU software stacks (CUDA, Triton, NCCL) and Kubernetes orchestration.
• Practical experience with model-serving frameworks such as vLLM, SGLang, TensorRT-LLM, or custom PyTorch deployments.
• Knowledge of performance optimisation techniques, including batching, speculative decoding, quantisation, and caching.
• Familiarity with Infrastructure-as-Code tools (Terraform, Helm) and low-level OS performance tuning.

Nice to Have

• Experience with event-driven or serverless architectures.
• Exposure to hybrid cloud or multi-cluster environments.
• Contributions to open-source ML or inference systems projects.
• Proven track record of cost optimisation in high-performance compute environments.

Benefits

• Equity

Salary

• $300,000 gross per year