Software Engineer

Lemon.io

Greater London

On-site

GBP 95,000 - 130,000

Full time

9 days ago

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Job summary

Lemon.io is seeking a senior Infra Agent for its Systems team in London. You will design and build production AI agents that diagnose and remediate infrastructure issues across a massive GPU fleet, shaping the platform they run on.

In this role, you will own end-to-end services, develop the memory and knowledge graphs, and improve agent performance using evaluation frameworks and telemetry. Strong English and ownership are essential.

Qualifications

  • 5+ years building production backend systems, distributed systems, or infrastructure platforms
  • 5+ years of production experience with Python, Go, or TypeScript
  • Hands-on experience building AI agents (harness, tools, memory, evaluation), not only RAG pipelines or LLM-powered product features
  • Proven results in significantly improving agent accuracy and effectiveness
  • Experience designing human-in-the-loop controls and access boundaries for agents acting on live systems
  • Experience with Kubernetes, GitOps (e.g., ArgoCD), infrastructure-as-code, and cloud platforms
  • Daily use of AI coding tools in your workflow
  • Builder mindset and high ownership: you've played a major role in designing and shipping complex systems
  • Strong communication skills and full professional English proficiency
  • GPU infrastructure, data centers, bare-metal systems, hardware failure modes, BMC/IPMI or cluster schedulers
  • Building knowledge graphs; experience with graph databases
  • Event-driven systems and messaging (NATS, Kafka)
  • Observability platforms such as Prometheus and Grafana
  • Building evaluation frameworks for LLM-powered systems
  • Training or fine-tuning small specialized models, including classical ML such as anomaly detection

Responsibilities

  • Design and build production AI agents that diagnose, investigate, and help remediate infrastructure issues across a massive GPU fleet
  • Build the distributed services, orchestration framework, knowledge graph, and retrieval systems that power these agents
  • Keep agent knowledge accurate and up to date, and combine telemetry, infrastructure state, and incident history to build fleet intelligence
  • Improve agent performance through evaluations, improved tools, improved retrieval, and production feedback loops
  • Own services end-to-end, from architecture and implementation to deployment, observability, and production operations

Skills

Backend systems
Distributed systems
Infrastructure platforms
Python
Go
TypeScript
AI agents
Agent accuracy optimization
Human-in-the-loop
Kubernetes
GitOps
IaC
Cloud platforms
AI coding tools
Ownership mindset
Communication skills
English proficiency
GPU infrastructure
Bare-metal systems
Cluster schedulers
Knowledge graphs
Graph databases
Event-driven systems
Messaging (NATS/Kafka)
Prometheus
Grafana
LLM evaluation frameworks
Model training (ML)
Anomaly detection

Tools

Kubernetes
ArgoCD (GitOps)
Cloud platforms

Job description

We help companies in the US and Europe hire experienced engineers, from funded startups to teams like Klarna, Retool, and Pinterest. We've been doing this since 2015, and more than 1,500 companies have hired engineers through us.

About the client

A fast-growing AI cloud company with strong roots in open-source AI research. Its platform supports some of the most widely used AI products on the market and runs on one of the world's largest GPU fleets. At this scale, infrastructure operations are increasingly handled by software and autonomous AI agents, and the company is investing heavily in building them.

The role

You'll join the Infra Agent Systems team, which builds production AI agents that diagnose hardware failures, investigate incidents, and automate operations across the GPU fleet, along with the platform they run on. The core challenge is making AI agents trustworthy enough to operate production infrastructure at scale. It's foundational work that hasn't really been done before.

What you'll do
  • Design and build production AI agents that diagnose, investigate, and help remediate infrastructure issues across a massive GPU fleet
  • Build the distributed services, orchestration framework, knowledge graph, and retrieval systems that power these agents
  • Keep agent knowledge accurate and up to date, and combine telemetry, infrastructure state, and incident history to build fleet intelligence
  • Improve agent performance through evaluations, improved tools, improved retrieval, and production feedback loops
  • Own services end-to-end, from architecture and implementation to deployment, observability, and production operations
Must-have requirements
  • 5+ years building production backend systems, distributed systems, or infrastructure platforms
  • 5+ years of production experience with Python, Go, or TypeScript
  • Hands-on experience building AI agents (harness, tools, memory, evaluation), not only RAG pipelines or LLM-powered product features
  • Proven results in significantly improving agent accuracy and effectiveness
  • Experience designing human-in-the-loop controls and access boundaries for agents acting on live systems
  • Experience with Kubernetes, GitOps (e.g., ArgoCD), infrastructure-as-code, and cloud platforms
  • Daily use of AI coding tools in your workflow
  • Builder mindset and high ownership: you've played a major role in designing and shipping complex systems
  • Strong communication skills and full professional English proficiency
  • GPU infrastructure, data centers, bare-metal systems, hardware failure modes, BMC/IPMI or cluster schedulers
  • Building knowledge graphs; experience with graph databases
  • Event-driven systems and messaging (NATS, Kafka)
  • Observability platforms such as Prometheus and Grafana
  • Building evaluation frameworks for LLM-powered systems
  • Training or fine-tuning small specialized models, including classical ML such as anomaly detection
Team and setup
  • Team of 10+ engineers, working directly with principal engineers and team leads.
  • Agents are used daily by infrastructure and data center teams through APIs, CLI, dashboards, and Slack
  • Near-term focus: improving retrieval, the memory backend and the agent harness, and training small specialized models on the company's own infrastructure
Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Senior Software Engineer — Infra Agent Systems UK
Senior Software Engineer — Infra Agent Systems UK

FDE Times • United Kingdom

Remote
GBP 90,000 - 130,000
Senior Software Engineer — Infra Agent Systems UK
Senior Software Engineer — Infra Agent Systems UK

Together AI • Greater London

Remote
GBP 100,000 - 170,000
Senior Software Engineer — Infra Agent Systems UK
Senior Software Engineer — Infra Agent Systems UK

Togetherai • Greater London

Remote
GBP 120,000 - 180,000
AI Agent Developer
AI Agent Developer

Vallum Associates • Greater London

On-site
GBP 120,000 - 180,000
Senior Software Engineer – AI Infrastructure
Senior Software Engineer – AI Infrastructure

Favia • United Kingdom

Remote
GBP 90,000 - 130,000
Senior AI Infrastructure Engineer
Senior AI Infrastructure Engineer

FDE Times • United Kingdom

Remote
GBP 90,000 - 130,000
HPC Infrastructure Engineer - GPU Clusters
HPC Infrastructure Engineer - GPU Clusters

ElevenLabs • Greater London

Hybrid
GBP 120,000 - 180,000
Remote-friendly
Global team
Office options: London, New York, San 
+1
Member of Technical Staff, ML Infrastructure
Member of Technical Staff, ML Infrastructure

Stealth - AI x Bio • Greater London

On-site
GBP 89,000 - 134,000
30 days annual leave + public holidays
UK pension 10% employer contribution +
AI Agent Developer
AI Agent Developer

Vallum Associates • City Of London

On-site
GBP 120,000 - 180,000
Backend Engineer, AI Systems
Backend Engineer, AI Systems

Act • Greater London

On-site
GBP 70,000 - 110,000