Staff Software Engineer, Inference API

Engg; Toronto; On-site; CAD 140,000 - 200,000; Full time

Job summary

Cerebras Systems is building a new generation of disaggregated AI inference systems. We are hiring a Software Engineer to evolve the ML API layer for a heterogeneous serving system across models and accelerator backends.

You will work across inference APIs, model integration, vLLM-based runtime, and Cerebras runtime components to deliver a consistent experience for developers and customers. This role is hands-on and focuses on production-quality APIs.

Qualifications

• 5+ years of software engineering experience in production software or distributed systems.
• Strong programming in Python and performance-sensitive services in C++/Go or similar.
• Understanding of modern LLM inference concepts: tokenization, sampling, streaming, KV-cache, model configuration.
• Experience integrating software across service, framework, runtime, and infrastructure boundaries.
• Experience building stable APIs with validation, error handling, observability, compatibility, and versioning.

Responsibilities

• Build production ML inference APIs.
• Design, implement, and maintain APIs for chat completions, text generation, streaming, model configuration, tool calling, structured outputs, and multimodal inputs.
• Deliver a unified serving experience across GPU prefill, Cerebras decode, and other backends.
• Enable new models and capabilities by integrating foundation models, tokenizers, prompts, sampling, and model-specific features.
• Own API compatibility and evolution with clear versioning, deprecation, validation, and backward compatibility.
• Integrate with model-serving runtimes (vLLM, PyTorch, Hugging Face, ROCm, Cerebras runtime).
• Support disaggregated inference: coordination of request routing, state transfer, errors, retries, lifecycle.
• Improve performance: streaming, latency, throughput, batching, tokenization, scheduling.
• Ensure correctness: validation for tokenization, sampling, logits, outputs, precision changes.
• Strengthen reliability and observability: logging, tracing, metrics, dashboards, health checks.
• Develop testing and qualification infrastructure: conformance tests, workload replay, model validation, performance benchmarks.
• Improve developer experience: configuration, SDKs, docs, examples, debugging tools.
• Collaborate across stack: translate model/customer requirements into scalable serving capabilities.

Skills

Python

C++/Go

Distributed systems

API design

Linux & Containers

Observability

Communication

Education

Bachelor's degree in CS/CE/EE

Tools

vLLM

SGLang

TensorRT-LLM

Triton Inference Server

Hugging Face Text Generation Inference

Job description

Cerebras Systems builds the world's largest AI chip, 56 times larger than GPUs. This architecture allows Cerebras to deliver industry-leading training and inference speeds; over 10 times faster than GPU-based hyperscale cloud inference services. This order of magnitude increase in speed is transforming the user experience of AI applications, unlocking real-time iteration and increasing intelligence via additional agentic computation. Cerebras works with the leading model labs, global enterprises, and cutting-edge AI-native startups. OpenAI recently announced a multi‑year partnership https://openai.com/index/cerebras-partnership/ with Cerebras, to deploy 750 megawatts of scale, transforming key workloads with ultra high-speed inference.

About the Role Cerebras is building a new generation of disaggregated AI inference systems that combine GPU-accelerated prefill with ultra-fast decode on the Cerebras Wafer-Scale Engine. We are hiring a Software Engineer to build and evolve the ML API layer that makes this heterogeneous serving system accessible, reliable, and easy to use. You will work across our inference APIs, model integration layer, request‑routing services, vLLM‑based GPU runtime, and Cerebras inference platform to deliver a consistent experience across models and accelerator backends. This role sits at the intersection of machine learning systems, API design, model serving, and distributed systems. You will enable new model architectures and inference capabilities, define stable user‑facing behavior, and ensure that features such as streaming, sampling, tool use, structured outputs, multimodal inputs, and model configuration behave correctly and consistently in production. You will work closely with model enablement, compiler, runtime, cloud infrastructure, product, and customer‑facing teams. This is a hands‑on software engineering role for someone who enjoys turning rapidly evolving ML capabilities into durable, production‑quality APIs.

Responsibilities

• Build production ML inference APIs.
• Design, implement, and maintain APIs for chat completions, text generation, streaming, model configuration, tool calling, structured outputs, multimodal inputs, and other emerging inference capabilities.
• Deliver a unified serving experience. Create consistent request and response semantics across GPU prefill, Cerebras decode, and other heterogeneous inference backends.
• Enable new models and capabilities. Integrate emerging foundation models, tokenizers, prompt formats, sampling methods, attention variants, multimodal inputs, and model‑specific features into the serving platform.
• Own API compatibility and evolution. Maintain compatibility with widely adopted inference interfaces while designing Cerebras‑specific extensions. Establish clear versioning, deprecation, validation, and backward‑compatibility practices.
• Integrate with model‑serving runtimes. Extend and integrate custom inference services with vLLM, PyTorch, Hugging Face libraries, the AMD ROCm stack, and Cerebras runtime components.
• Support disaggregated inference. Build the control and data paths required to coordinate GPU prefill with Cerebras decode, including request routing, state transfer, error handling, retries, and lifecycle management.
• Improve serving performance. Optimize streaming behavior, time to first token, request latency, throughput, batching, serialization, tokenization, scheduling, and communication between serving components.
• Ensure functional and numerical correctness. Build validation systems for tokenization, sampling, logits, generated outputs, precision changes, model upgrades, determinism, and compatibility across serving backends.
• Strengthen reliability and observability. Define end‑to‑end service indicators and build structured logging, tracing, metrics, dashboards, health checks, and diagnostic tooling for production inference traffic.
• Develop testing and qualification infrastructure. Create conformance tests, workload‑replay tools, model‑validation suites, performance benchmarks, integration tests, and release gates.
• Improve developer experience. Build intuitive configuration, SDKs, documentation, examples, debugging tools, and self‑service workflows for internal developers, customers, and partners.
• Collaborate across the stack. Partner with compiler, runtime, kernel, cloud, product, and solutions teams to translate model and customer requirements into scalable serving capabilities.

Minimum Qualifications

• 5+ years of software engineering experience, including substantial individual‑contributor ownership of production software or distributed systems.
• Strong programming ability in Python and experience developing performance‑sensitive or highly concurrent services in C++, Go, or a similar systems language.
• Hands‑on experience with a model‑serving framework such as vLLM, SGLang, TensorRT‑LLM, Triton Inference Server, Hugging Face Text Generation Inference, or an equivalent internally developed platform.
• Understanding of modern LLM inference concepts, including tokenization, prompt formatting, sampling, streaming generation, continuous batching, KV‑cache management, and model configuration.
• Experience integrating software across service, framework, runtime, and infrastructure boundaries.
• Experience building stable APIs with clear validation, error handling, observability, compatibility, and versioning practices.
• Experience with Linux, containers, Kubernetes or comparable orchestration systems, CI/CD, and operating latency‑sensitive services in production.
• Ability to diagnose correctness, reliability, and performance issues across multiple components of a distributed serving system.
• Strong communication and cross‑functional execution skills, with the ability to turn ambiguous model or product requirements into production‑quality software.
• Bachelor’s degree in computer science, Computer Engineering, Electrical Engineering, or a related discipline, or equivalent practical