Senior Agent Runtime & Systems Engineer – Distributed AI

AI Startups UK; Greater London; Hybrid; GBP 120,000 - 160,000; Full time

Benefits offered by this job

Visa sponsorship

Relocation benefits

Private healthcare

London office

In-person work

Job summary

Callosum in London is seeking a seasoned engineer to own the runtime and programming model for long-running agent systems. You will build durable execution, checkpointing, and secure tool execution across distributed environments.

The role emphasizes Python, systems languages, and a high-leverage stack; you will design APIs, propose new execution models, and drive observability, reliability, and performance at scale.

Qualifications

• Strong distributed systems background with hands-on experience in durable workflows, actor systems, schedulers, workflow engines, distributed databases, or stream processors
• Experience designing a library, framework, runtime, or DSL that other engineers or researchers adopted
• Deep Python expertise plus proficiency in a systems language such as Rust, C++, or Go - and the instinct to move between abstraction layers when it matters
• Hands-on debugging skills across APIs, runtimes, and distributed infrastructure - with a practical understanding of nondeterminism, side effects, and reproducibility

Responsibilities

• Design and build stateful, asynchronous agent runtimes across distributed environments
• Define event histories, checkpoint and resume semantics, deterministic reconstruction, retries, cancellation, timeouts, and recovery from partial failure
• Build traces and replay, time-travel debugging, and hierarchical tracing across model calls, tools, handoffs, state changes, and runtime decisions
• Develop clear Python APIs, libraries, and DSLs with stable extension points for tools, backends, and new execution models
• Design scheduling, placement, backpressure, resource accounting, sandboxing, permissions, and isolation for secure tool execution at scale
• Develop tooling for trajectory collection and visualisation, performance profiling, experiment comparison, failure attribution, and regression detection across models, prompts, tools, and runtime versions

Skills

Distributed systems

Library/framework design

Python

Rust

C++

Go

Debugging across APIs

Tools

Ray

Temporal

Kafka

PyTorch

JAX

TensorFlow

Job description

Callosum in London is seeking a seasoned engineer to own the runtime and programming model for long-running agent systems. You will build durable execution, checkpointing, and secure tool execution across distributed environments.

The role emphasizes Python, systems languages, and a high-leverage stack; you will design APIs, propose new execution models, and drive observability, reliability, and performance at scale.