ML Systems Engineer (RL)

Roc Search Inc.; Greater London; Hybrid; GBP 100.000 - 110.000; Vollzeit

Zusammenfassung

AI Startup in London is seeking a Machine Learning Systems Engineer to design and scale the deployment layer of a distributed platform. You will orchestrate concurrent apps across a vast on-premise network, ensuring reliable execution within memory and compute constraints.

You will collaborate with research and engineering teams, build testing environments, and scale distributed pipelines while turning experimental concepts into production software. Hybrid in-office pattern with weekly schedule.

Qualifikationen

• Proficient production-grade Python development with strong testing patterns.
• Experience delivering distributed systems and concurrent workloads.
• Familiarity with monitoring and observability tooling in distributed apps.
• Knowledge of memory and compute constraints in large-scale deployments.
• Experience with ML platforms and model deployment workflows.

Aufgaben

• Resource Management: design runtime isolation, scheduling, and allocations for multiple local processes.
• System Synchronization: build reconciliation to ensure atomic updates and version alignment across environments.
• Release Management: architect deployment flows with progressive rollout, validation modes, guardrails, and recovery loops.
• Simulation and testing: create robust testing environments for distributed workloads.
• Pipeline Automation: build fault-tolerant networks with state saving and cross-site data flows.
• Performance optimization: profile and tune code and hardware acceleration.
• Telemetry: establish data streams and dashboards for system health.
• Engineering Rigour: assist teams with blockers and enforce coding standards.
• Documentation and knowledge sharing for robust operations.

Kenntnisse

Python

MLops

Pytorch

Grafana

Elastic Search

TensorRT

Jobbeschreibung

Machine Learning Systems Engineer (Distributed Systems)

London (1-2 days per week in office)

AI Startup

£100,000-£110,000 DOE + Bonus

Skills: Python, MLops, Pytorch, Grafana, Elastic Search, TensorRT

About the Company

Our client is a venture-backed technology company building software to optimise large-scale physical infrastructure. Their platform processes data locally at the source using decentralized networks. Their mission is to make global industrial operations more resilient, secure, and sustainable through advanced automation.

About the Role

Our client is seeking an Infrastructure Engineer to design and scale the deployment layer of their distributed technology platform. The core challenge involves orchestrating complex, concurrent software applications and analytical workloads across a massive network of diverse, on-premise hardware installations.

In this position, you will own the systems engineering required to guarantee that these disparate applications execute reliably within strict memory and compute constraints. You will also collaborate directly with their research and engineering teams, building robust testing environments, scaling distributed pipelines, and converting experimental concepts into dependable production software.

Key Responsibilities

Network Orchestration & System Performance

• Resource Management: Design runtime isolation, task scheduling, and resource allocations for multiple concurrent local processes sharing the same hardware.
• System Synchronization: Build robust reconciliation mechanisms to ensure atomic updates and version alignment across remote environments.
• Release Management: Architect deployment flows supporting progressive rollout strategies, passive validation modes, safety guardrails, and automated recovery loops.

Scalable Infrastructure & Automation

• Simulation Frameworks: Develop and maintain large-scale virtualized environments to safely emulate real-world networks and system behaviors for validation.
• Pipeline Automation: Construct fault-tolerant distributed processing networks that support automated state saving, failure recovery, and cross-site data flows.
• Performance Optimization: Profile system execution to improve processing performance through code optimization, resource tuning, and hardware acceleration on varied architectures.

Diagnostics & Engineering Standards

• Data Streams: Establish reliable telemetry and ingestion channels that preserve data lineage for downstream analytic workflows.
• System Telemetry: Implement comprehensive dashboard metrics, unified logging, and warning systems to detect system degradation or variance early.
• Engineering Rigour: Troubleshoot deep architectural bugs, assist engineering teams with technical blockers, and enforce high coding standards via thorough review processes.

Candidate Profile

Core Technical Experience

• Software Foundations: Exceptional software engineering capabilities in production-level Python, with a strong focus on clean testing patterns and modular design.
• Distributed Computing: Extensive experience managing state alignment, messaging, and system execution across inconsistent networks and varied hardware form factors.
• Resource Partitioning: Demonstrated skill in managing system memory, computing bounds, and storage across competing local application tasks.
• Systems Infrastructure: Deep operational familiarity with managing background workloads, handling checkpointing/recovery, and optimizing software performance.
• Production Operations: Solid track record establishing telemetry, tracking system health, and managing alerting rules in distributed or containerized ecosystems.

Preferred Technical Exposure

• Experience building infrastructure for simulation software, virtual testbeds, or automated control systems.
• Experience with Reinforcement Learning tools such as Ray RLlib
• Familiarity with high-efficiency runtime environments or specialized hardware acceleration toolchains.
• Background in remote system provisioning, telemetry transport protocols (such as messaging queues), or remote software updates.
• Experience maintaining custom hardware environments, private network setups, or software for highly regulated environments.