Senior Software Engineer, ML API - AI Inference

Cerebras

Toronto

On-site

CAD 130,000 - 190,000

Full time

14 days+

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Benefits offered by this job

Remote-friendly? (not stated)

Job summary

Cerebras Systems is hiring a Senior Software Engineer to build and evolve the ML API layer for a disaggregated AI inference system. You will own features across inference APIs, model integration, and request routing, working with model enablement, runtime, cloud infrastructure, and customer teams.

This hands-on, IC role offers opportunities to mentor and lead small projects. You will collaborate across GPU prefill, Cerebras decode, and other backends, ensuring streaming, tool use, structured

Qualifications

  • 5+ years of software engineering experience with production services or distributed systems
  • Strong programming ability in Python; proficiency in Go, C++, or Rust
  • Experience designing features spanning multiple components and APIs
  • Experience building and evolving production APIs with validation and testing
  • Experience operating services on Linux with containers and orchestration
  • Strong communication and ability to lead technical delivery on small projects
  • Bachelor's degree in Computer Science or related field

Responsibilities

  • Own end-to-end features for chat completions, text generation, and inference machinery
  • Design, implement, test, and operate API features across multiple backends
  • Write design docs and coordinate implementation with engineers and partner teams
  • Ensure compatibility with common inference interfaces and maintain API versioning
  • Integrate models, tokenizers, templates, and configuration into the serving platform
  • Improve reliability, testing, and performance of production inference software

Skills

Python
Go
C++
Rust
Distributed systems
REST
gRPC

Education

Bachelor's degree in Computer Science

Tools

Kubernetes
CI/CD
Docker
PyTorch
Hugging Face
LLGuidance
vLLM

Job description

Cerebras Systems builds the world's largest AI chip, 56 times larger than GPUs. This architecture allows Cerebras to deliver industry-leading training and inference speeds; over 10 times faster than GPU-based hyperscale cloud inference services. This order of magnitude increase in speed is transforming the user experience of AI applications, unlocking real-time iteration and increasing intelligence via additional agentic computation. Cerebras works with the leading model labs, global enterprises, and cutting-edge AI-native startups. OpenAI recently announced a multi-year partnership https://openai.com/index/cerebras-partnership/ with Cerebras, to deploy 750 megawatts of scale, transforming key workloads with ultra high-speed inference.

About the Role

Cerebras is building a new generation of disaggregated AI inference systems that combine GPU-accelerated prefill with ultra-fast decode on the Cerebras Wafer-Scale Engine. We are hiring a Senior Software Engineer to build and evolve the ML API layer that makes this heterogeneous serving system accessible, reliable, and easy to use. You will own significant features and components across our inference APIs, model integration layer, and request-routing services, delivering a consistent experience across models and accelerator backends. This role sits at the intersection of machine learning systems, API design, model serving, and distributed systems. You will translate evolving model capabilities and customer requirements into clear designs, lead implementation across multiple components, and carry features through validation, rollout, and production operation. Your work will help ensure that streaming, sampling, tool use, structured outputs, multimodal inputs, and model configuration behave correctly and consistently. You will work closely with model enablement, compiler, runtime, cloud infrastructure, product, and customer-facing teams, as well as customers directly. This is a hands-on individual-contributor role with opportunities to lead small project teams, mentor engineers, and improve the quality and reliability of production inference software.

Responsibilities
  • Own features end to end.
  • Design, implement, test, and operate features for chat completions, text generation, streaming, model configuration, tool calling, structured outputs, multimodal inputs, and emerging inference capabilities.
  • Write clear design documents, evaluate technical tradeoffs, break projects into deliverable milestones, and coordinate implementation with a small group of engineers and partner teams.
  • Implement clear request and response semantics across GPU prefill, Cerebras decode, and other inference backends. Maintain compatibility with widely adopted inference interfaces and evolve owned APIs through validation, versioning, and deprecation best practices.
  • Integrate foundation models, tokenizers, chat templates, generation configuration, sampling methods, and model-specific features into the serving platform.
  • Extend inference services and their integration with frameworks and runtimes such as vLLM, PyTorch, Hugging Face libraries, LLGuidance, and Cerebras runtime components.
  • Implement routing, state-transfer integration, error handling, retries, cancellation, and request lifecycle management to coordinate GPU prefill with Cerebras decode.
  • Diagnose bottlenecks and improve time to first token, streaming latency, throughput, and resource efficiency through changes to tokenization, serialization, batching, scheduling, and communication.
  • Build conformance tests, model-validation suites, numerical comparisons, integration tests, and performance benchmarks that catch regressions in owned features and support safe releases.
  • Improve service metrics, tracing, logging, alerts, and diagnostic tools. Investigate production incidents, resolve root causes, and follow through on improvements to resilience and incident response.
  • Partner with product, support, and solutions teams to understand user needs and make practical tradeoffs. Improve configuration, documentation, examples, and debugging workflows.
  • Provide thoughtful design and code reviews, mentor less-experienced engineers, share domain knowledge, and proactively remove delivery roadblocks.
Minimum Qualifications
  • 5+ years of software engineering experience, including independent ownership of production services, distributed systems, or substantial software components.
  • Strong programming ability in Python and proficiency in Go, C++, Rust, or another language used to build production services.
  • Experience designing and delivering features that span multiple components, including documenting designs, evaluating tradeoffs, and coordinating implementation with other engineers.
  • Experience building and evolving production APIs, with sound practices for validation, error handling, compatibility, and testing. Familiarity with REST, gRPC, or streaming protocols.
  • Experience integrating software across service, framework, runtime, or infrastructure boundaries.
  • Experience operating services on Linux using containers, Kubernetes or comparable orchestration systems, and CI/CD.
  • Ability to diagnose correctness, reliability, and performance issues across distributed components and deliver measurable improvements.
  • Strong communication and collaboration skills, including experience leading technical delivery for a small project and supporting other engineers through reviews or mentorship.
  • Ability to turn ambiguous requirements into maintainable software while balancing customer needs, delivery timelines, and operational quality.
  • Bachelor's degree in C
Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Senior Software Engineer, ML API – AI Inference
Senior Software Engineer, ML API – AI Inference

FDE Times • Toronto

On-site
CAD 209,000 - 293,000
Staff Software Engineer, Inference API
Staff Software Engineer, Inference API

Cerebras • Toronto

On-site
CAD 140,000 - 190,000
Staff Software Engineer, GPU Inference
Staff Software Engineer, GPU Inference

FDE Times • Toronto

On-site
CAD 251,000 - 335,000
Full Stack LLM Engineer
Full Stack LLM Engineer

FDE Times • Toronto

On-site
CAD 120,000 - 190,000
Competitive salary
Career growth
Dynamic work environment
+1
Software Engineer, Inference Platform
Software Engineer, Inference Platform

Cerebras • Toronto

On-site
CAD 256,000 - 370,000
Staff Software Engineer, GPU Inference
Staff Software Engineer, GPU Inference

Cerebras • Toronto

On-site
CAD 150,000 - 210,000
Senior Research Engineer - Inference ML
Senior Research Engineer - Inference ML

We Love AI Jobs • Ottawa

Hybrid
CAD 180,000 - 240,000
Staff Software Engineer, Inference Platform
Staff Software Engineer, Inference Platform

Cerebras • Toronto

On-site
CAD 293,000 - 405,000
Software Engineer, GPU Inference
Software Engineer, GPU Inference

Cerebras • Toronto

On-site
CAD 140,000 - 210,000
Distributed Software Engineer
Distributed Software Engineer

Cerebras • Toronto

On-site
CAD 140,000 - 210,000