Staff GPU Inference SDET

Cerebras Systems; Lower Sackville; On-site; CAD 90,000 - 130,000; Full time

Job summary

Cerebras Systems seeks a Staff GPU Inference SDET to lead quality and validation for a new GPU Inference Development team. You will design, build, and scale end-to-end release qualification and automated test ecosystems for our GPU inference stack and rack-scale accelerated compute fleets.

In this high-impact role you will create automated test suites to validate multi-node GPU cluster bring-up, verify prefill optimizations, test open-source and custom serving engines, and ensure numerical

Qualifications

• 8+ years of software engineering experience as an SDET, Infrastructure Quality Lead, or Systems Test Engineer.
• Hands-on experience bringing up, provisioning, and validating multi-node GPU clusters across public cloud or data centers.
• Deep understanding of LLM serving engines and distributed runtimes.
• Expert-level Python programming with test automation frameworks and CI/CD integration.
• Proficiency with container orchestration and high-performance interconnects.
• Root-cause analysis across software/hardware boundaries and fault injection.

Responsibilities

• Build GPU Release Qualification Systems: automated test frameworks, regression gates, release pipelines for the GPU inference stack.
• Inference Serving & Workload Validation: benchmark/distributed serving frameworks, prefill vs decode performance, continuous batching.
• Performance & Modeling Verification: workload replay and performance model validation with TTFT, ITL, throughput, P99, capacity.
• Numerical Correctness & Quality Gates: validation to ensure accuracy, precision stability, determinism across updates.
• Fault Injection & Fleet Resilience: chaos engineering and node failure simulations with automated recoveries.
• Observability & CI/CD Integration: tie tests to telemetry tools for repeatable gates and monitoring.

Skills

SDET Experience

Test Automation

Distributed Systems

Performance Testing

Chaos Engineering

Linux/Unix

Tools

Python

Kubernetes

Slurm

Ray

NCCL

Job description

Cerebras Systems builds the world's largest AI chip, 56 times larger than GPUs. This architecture allows Cerebras to deliver industry-leading training and inference speeds; over 10 times faster than GPU-based hyperscale cloud inference services.

This order of magnitude increase in speed is transforming the user experience of AI applications, unlocking real-time iteration and increasing intelligence via additional agentic computation.

Cerebras works with the leading model labs, global enterprises, and cutting-edge AI-native startups. OpenAI recently announced a multi-year partnership with Cerebras, to deploy 750 megawatts of scale, transforming key workloads with ultra high-speed inference.

About the Role

As a Staff GPU Inference SDET, you will be the founding quality, reliability, and validation lead for a new GPU Inference Development team. Working closely with engineering leads and cross-functional systems infrastructure teams, you will design, build, and scale the end-to-end release qualification and automated test ecosystem for our GPU inference stack and rack-scale accelerated compute fleets. In this high-impact role, you will be responsible for building automated test suites to validate multi-node GPU cluster bring-up, verifying prefill worker optimizations, testing open-source and custom serving engines, and ensuring numerical correctness and performance stability under real-world streaming workloads. You will be the primary technical anchor ensuring production-grade reliability, fault isolation, and peak inference performance across accelerated GPU infrastructure.

WHAT YOU'LL DO

Build GPU Release Qualification Systems:

Design and implement automated test automation frameworks, regression gates, and release qualification pipelines for the complete GPU inference stack—spanning custom API services, model-serving workers, container runtimes, serving engines, driver stacks, and firmware.

Inference Serving & Workload Validation:

Benchmark and stress-test distributed LLM serving frameworks, focusing on prefill vs. decode worker performance, continuous batching, prefix caching, KV-cache efficiency, and tensor/expert parallelism.

Performance & Performance Modeling Verification:

Build automated workload replay and benchmarking tools to validate GPU performance models. Track critical serving metrics including Time-to-First-Token (TTFT), Inter-Token Latency (ITL), request throughput, tail latency (P99), and capacity efficiency.

Numerical Correctness & Quality Gates:

Build validation infrastructure to ensure model accuracy, precision stability (FP16/FP8/quantization), determinism, and output correctness across software updates, kernel fusions, and hardware revisions.

Fault Injection & Fleet Resilience:

Engineer chaos engineering and fault-injection suites to simulate node failures, inter-node network degradation, GPU memory leaks, driver/firmware mismatches, and automated recovery paths for multi-node GPU clusters.

Observability & CI/CD Integration:

Integrate automated test pipelines with telemetry tools (e.g., Prometheus, Grafana) to turn one-off investigations into repeatable engineering gates and continuous performance monitoring.

REQUIREMENTS:

8+ years of software engineering experience as an SDET, Infrastructure Quality Lead, or Systems Test Engineer.

GPU & Cluster Infrastructure Expertise:

Hands-on experience bringing up, provisioning, and validating multi-node GPU clusters (NVIDIA or AMD ecosystem) across public cloud infrastructure or enterprise data center environments.

Inference Stack Knowledge:

Deep understanding of LLM serving engines and distributed runtimes, including prefill vs. decode disaggregation, KV-cache management, and dynamic batching.

Automation & Scripting:

Expert-level Python programming skills with extensive experience designing custom test automation frameworks, diagnostic tooling, and CI/CD integration.

Orchestration & Networking:

Strong proficiency with container orchestration tools (e.g., Kubernetes, Slurm, Ray) and high-performance cluster interconnects (e.g., InfiniBand, RoCE, NCCL).

Failure Analysis & Debugging:

Proven background in root-cause analysis across software/hardware boundaries, stress testing, and node failure simulation in distributed systems.

NICE TO HAVES:

• Direct experience with either AMD (ROCm / HIP) or NVIDIA software stacks.
• Experience building workload replay tools, ML evaluation pipelines, or MLPerf Inference benchmark suites.
• Familiarity with low-level kernel profiling tools (PyTorch Profiler, NVTX, ROCm profilers) or C++

Why Join Cerebras

People who are serious about software make their own hardware. At Cerebras, we have built a breakthrough architecture that is unlocking new opportunities for the AI industry. With dozens of model releases and rapid growth, we've reached an inflection point in our business. Members of our team tell us there are five main reasons they joined Cerebras:

1. Build a breakthrough AI platform beyond the constraints of the GPU.
2. Publish and open source their cutting-edge AI research.
3. Work on one of the fastest AI supercomputers in the world.
4. Enjoy job stability with startup vitality.
5. Our simple, non-corporate work culture that respects individual beliefs.

Find out more about what it's like to work at Cerebras here!

Apply today and become part of the forefront of groundbreaking advancements in AI!

Cerebras Systems is committed to creating an equal and diverse environment and is proud to be an equal opportunity employer. We celebrate different backgrounds, perspectives, and skills. We believe inclusive teams build better products and companies. We try every day to build a work environment that empowers people to do their best work through continuous learning, growth and support of those around them.

This website or its third-party tools process personal data. For more details, click here to review our CCPA disclosure notice.