ML Systems Engineer - Transferable Intelligence Team

Octaipipe; City Of London; On-site; GBP 90,000 - 130,000; Full time

Job summary

OctaiPipe is seeking an ML Systems Engineer to own the systems engineering behind our edge computing and federated AI platform. You will orchestrate workloads, ensure efficient inference on diverse hardware, and collaborate with scientists to translate ideas into scalable experiments.

You will work with RL and causal models, contribute to staging environments, and help build robust deployment workflows. This is a hands-on role in a fast-growing team creating technology to reduce energy use in

Qualifications

• Strong software engineering fundamentals and Python proficiency.
• Experience building and operating end-to-end ML systems in production.
• Distributed systems experience with state consistency and versioning across unreliable networks.
• Experience coordinating multiple workloads on shared hardware.
• Experience optimising model inference for production.
• Deep working knowledge of the ML training stack (PyTorch or similar).
• Experience building monitoring with Grafana and the Elastic Stack.
• Ability to debug complex failures and work autonomously with scientists.

Responsibilities

• Own workload coordination on the device: RL inference, fine-tuning, LLM explainer and analytics.
• Guarantee model lifecycle correctness across the fleet: versioning, atomic swaps, reconciliation with portal state.
• Build deployment workflows on the central portal: packaging, staged rollout, shadow-mode deployment, rollback.
• Scale staging environment and training systems: distributed training, orchestration, checkpointing, recovery.
• Profile and optimise on-device performance across heterogeneous hardware including LLM inference.
• Develop monitoring and observability on Grafana and Elastic Stack for drift detection.
• Build telemetry and data pipelines for on-device analytics and data lineage.
• Debug failures across the full stack to unblock scientists.
• Raise engineering bar through runbooks and post-mortems.

Skills

Python

Software engineering

Distributed systems

RL/inference optimization

Debugging complex failures

Autonomy

Collaboration with scientists

Tools

PyTorch

Grafana

Elastic Stack

ONNX Runtime

Job description

ML Systems Engineer

Transferable Intelligence Team

The Company

OctaiPipe is a young, ambitious company with the vision to be the global driving force for the next paradigm of foundational, physical AI that ensures our connected world, and its critical infrastructure, is safe, secure and sustainable. We are growing fast, having closed a recent funding round and looking to accelerate rapidly. OctaiPipe is offering the right candidate an exciting role on this adventure!

OctaiPipe is on a mission to revolutionise the optimisation of energy in data centres through decentralised artificial intelligence (AI). To do this, OctaiPipe is harnessing an elegant but revolutionary idea. Rather than move the data from the source, move the algorithms to the data to learn at the data source. This learning can be achieved with the intelligence of many devices through novel federated AI technology. OctaiPipe is developing the AI for Cooling Efficiency (ACE) application to be deployed using its own in-house distributed AI platform.

The Role

The Transferable Intelligence team builds the capability that lets what OctaiPipe's AI learns at one site benefit others: reinforcement learning agents for control, and causal inference and root cause analysis models, that train at the edge and transfer between heterogeneous data centre environments.

We are looking for an ML Systems Engineer to own the systems engineering behind this. Models are deployed through a central portal that orchestrates what runs on each edge device. The devices are heterogeneous on-prem hardware inside data centres, running a demanding mix of concurrent workloads: RL policy inference, fine-tuning, an LLM-based explainer, and on-device analytics served through a UI. This demands disciplined systems engineering: the right models, at the right versions, running efficiently within their resource budgets, across the entire fleet.

The role also has a secondary research dimension. You will work side-by-side with the team's applied and research scientists as a hands-on force multiplier: building the training systems and experiment infrastructure that take new algorithms and system designs from idea to result quickly, reproducibly and at scale, much of it within our staging environment of large virtual device fleets.

Duties and Responsibilities

• Own workload coordination on the device: scheduling, prioritisation, resource budgets and isolation across RL inference, fine-tuning, the LLM explainer and analytics.
• Guarantee model lifecycle correctness across the fleet: versioning, atomic model swaps, and reconciliation between portal intent and device state, working closely with the internal tools team building our model registry.
• Build the team's deployment workflows on the central portal: packaging, staged rollout, shadow-mode deployment, guardrails, and automatic fallback and rollback.
• Help build and scale the staging environment and the training systems inside it: continuous, distributed and federated training and fine-tuning across real and staging fleets, with automated orchestration, checkpointing, recovery and cross-site adaptation workflows.
• Profile and optimise on-device performance across heterogeneous hardware, including efficient LLM inference.
• Develop monitoring, alerting and observability on our Grafana and Elastic stack, enabling early detection of drift, anomalies and degradation.
• Build the telemetry and data pipelines that feed on-device analytics, causal inference and root cause analysis, with the data quality and lineage those methods demand.
• Debug failures across the full stack and unblock applied and research scientists quickly.
• Raise the engineering bar through code review, documentation, runbooks and post-mortems.

Your Profile

• Strong software engineering fundamentals and excellent Python.
• Experience building and operating end-to-end ML systems in production, from data pipelines through training to serving and monitoring.
• Solid distributed systems experience with care for correctness: state consistency, versioning and reconciliation across unreliable networks and heterogeneous hardware.
• Experience coordinating multiple workloads on shared hardware: resource budgeting, scheduling and isolation.
• Experience optimising model inference for production (profiling, quantisation, compilation) to run efficiently on constrained or shared hardware.
• Deep working knowledge of the ML training stack (PyTorch or similar), including distributed training and checkpoint/recovery.
• Experience building monitoring and observability with Grafana and the Elastic Stack, or similar.
• Skilled at debugging complex failures across the stack.
• Pragmatic, low-ego and hands-on; comfortable with autonomy while collaborating closely with scientists and engineers.

You Also Might Have

• Reinforcement learning infrastructure experience (e.g. Ray/RLlib, environment APIs, simulators) or deployment of closed-loop control systems.
• Experience building simulation or virtual test environments for distributed systems.
• Familiarity with LLM serving stacks (e.g. vLLM, llama.cpp) or optimisation toolchains such as ONNX Runtime and TensorRT.
• Experience with federated learning, transfer learning or privacy-preserving ML.
• Experience with on-prem GPU infrastructure or network-restricted environments.
• Experience with edge fleet management: device provisioning, remote deployment and over-the-air updates.
• Experience with telemetry and streaming systems (Kafka, MQTT).
• Exposure to security and compliance in critical infrastructure or regulated environments.

Why Join OctaiPipe

• Work on real-world sustainability impact at global scale
• Influence how AI is responsibly applied to critical infrastructure
• Join a well-funded, rapidly growing scale-up with ambitious goals
• Collaborate with experts across AI, infrastructure, and operations
• Shape a product that can materially reduce energy use and carbon emissions worldwide

The above statements are not intended to encompass all functions and qualifications of the position; rather, they are intended to provide a general framework of the requirements of the position. Job incumbents may be required to perform other functions not specifically addressed in this description.