Senior Systems Engineer - AI Infrastructure

Clockwork.io; Palo Alto (CA); On-site; USD 150,000 - 210,000; Full time

Benefits offered by this job

Catered lunch

Equity program

Benefits package

Job summary

Clockwork.io in Palo Alto is building infrastructure for fault-tolerant, high-performance distributed GPU training. This systems role digs into internals, designing and implementing low-level software that runs at scale and improves reliability across runtimes and frameworks.

You will own the development from design to production, work with CUDA/NCCL and PyTorch internals, and help shape the next generation of distributed GPU systems at Clockwork.io.

Qualifications

• Must have systems-building experience (not just deployment).
• Experience with distributed storage, databases, or coordination systems.
• Familiarity with runtimes, profilers, or performance tooling.
• Experience with large-scale systems at the systems layer.

Responsibilities

• Design and implement low-level systems software for GPU clusters.
• Modify and extend frameworks like PyTorch, NCCL, CUDA runtime.
• Build components to improve reliability and efficiency of large-scale GPU training.
• Debug complex distributed, concurrent systems where failures are subtle.
• Own systems end-to-end: from design through production.

Skills

C/C++

Concurrency

Distributed systems

Large codebases

Tools

CUDA

NCCL

PyTorch internals

RDMA

Job description

About Clockwork Systems

Clockwork.io – Software Driven Fabrics to increase GPU cluster utilization

Clockwork Systems was founded by Stanford researchers and veteran systems engineers who share a vision for redefining the foundations of distributed computing. As AI workloads grow increasingly complex, traditional infrastructure struggles to meet the demands of performance, reliability, and precise coordination. Clockwork is pioneering a software-driven approach to AI fabrics by delivering cross-stack observability to catch and quickly resolve problems, workload fault tolerance to keep jobs running through failures, and performance acceleration that dynamically routes and paces traffic to avoid congestion.

We're building infrastructure for fault-tolerant, high-performance distributed GPU training. You'll work at the intersection of GPU systems, high-speed networking, and distributed coordination—designing and implementing systems that run at scale.

This is a systems building role. You'll dig into internals, understand why things break under pressure, and design solutions that handle the messy reality of distributed systems.

What You'll Do

• Design and implement low-level systems software for GPU clusters
• Work with internals of frameworks like PyTorch, NCCL, CUDA runtime—not as a user, but modifying and extending them
• Build components that make large-scale GPU training more reliable and efficient
• Debug complex distributed/concurrent systems where failures are subtle and non-deterministic
• Own systems end-to-end: from design through production

What We're Looking For

Required: Systems building experience

You've designed and built complex systems—not just deployed or operated them. Examples:

• Distributed storage, databases, or coordination systems
• Runtimes, profilers, or performance tooling
• Network stacks, protocols, or high-performance I/O systems
• Large-scale infrastructure at the systems layer

Core technical skills:

• Strong C/C++ in systems contexts
• Deep understanding of concurrency, memory models.
• Experience reasoning about distributed system behavior: consistency, ordering, partial failures
• Comfortable reading and modifying large, unfamiliar codebases

Nice to have:

• GPU programming (CUDA) or GPU systems experience
• High-performance networking (RDMA, InfiniBand)
• ML framework or runtime internals
• Cluster scheduling or orchestration systems

We believe strong systems engineers pick up domain-specific tools quickly. We value your ability to reason about complex systems over checkbox familiarity with our specific stack.

• A friendly and inclusive workplace culture.
• Competitive compensation.
• A great benefits package.
• Catered lunch.

Compensation for this position will vary based on the skills and experience you bring, as well as internal equity considerations. For candidates hired at the posted level, the expected base salary range is $150,000 - $210,000. In addition to cash compensation, this role is eligible to participate in the company’s equity program, which may include stock options granted in accordance with the company’s equity plan and subject to approval and applicable vesting schedules.

Clockwork Systems is an equal opportunity employer. We are committed to building world-class teams by welcoming bright, passionate individuals from all backgrounds. All qualified applicants will receive consideration for employment without regard to race, color, ancestry, religion, age, sex, sexual orientation, gender identity or expression, national origin, disability, or protected veteran status. We believe diversity drives innovation, and we grow stronger together.