Platform Engineer (AI / LLM Operations)

Better Placed

United Kingdom

À distance

GBP 70 000 - 110 000

Plein temps

Il y a 4 jours
Soyez parmi les premiers à postuler

Obtenez une réponse de cet employeur

Votre CV et votre lettre de motivation, adaptés exactement à ce que l’employeur recherche.

Passez les filtres ATS

Résumé du poste

Better Placed is seeking a hands-on LLMOps Engineer to own monitoring, evaluation and reliability for live AI systems in a fully remote UK role. You will build and run production pipelines for LLM outputs, manage tool calls, observability and regression checks, and collaborate with AI engineers to improve performance and quality.

Ideal candidates have real commercial experience operating AI systems in production, strong Python skills, AWS familiarity and a track record of delivering reliable,

Qualifications

  • Hands-on experience building production LLM evaluation and monitoring pipelines.
  • Experience beyond prototype work; production systems required.
  • Strong Python development and debugging skills.
  • Experience with AWS and observability, logging, tracing, alerting.

Responsabilités

  • Build and maintain monitoring, observability and evaluation pipelines for production AI and LLM applications.
  • Develop automated evaluation frameworks to measure response accuracy, retrieval quality, hallucinations, consistency and performance.
  • Monitor and troubleshoot agentic AI workflows, including tool calls, orchestration failures, context handling and multi-step execution.
  • Identify and investigate changes in model behaviour, output quality and performance over time.
  • Build dashboards, alerts and quality metrics to help the team understand AI performance.
  • Support model upgrades, prompt changes, version management and provider transitions.
  • Investigate production incidents across AI applications, data pipelines, APIs and supporting infrastructure.
  • Work closely with AI Engineers to improve reliability and operational processes.

Connaissances

Production AI monitoring
Python development
Observability & debugging
AWS experience
CI/CD & automated testing
Problem-solving
Production monitoring

Outils

Kubernetes
Docker
Terraform
LangChain
Langfuse
LangSmith

Description du poste

Fully Remote (UK) Permanent

My client is a substantial, private equity-backed UK business investing heavily in AI and automation as part of a major technology transformation.

They're looking for a hands‑on LLMOps Engineer with proven experience building and operating monitoring and evaluation pipelines for LLM applications in live production environments.

This isn't a traditional Machine Learning Engineer, MLOps or AI application development position.

The focus is on ensuring production LLM and agentic AI systems deliver reliable, accurate and measurable results.

You’ll be working on automated LLM output evaluation, hallucination and groundedness checks, agent tracing, regression testing, observability and production reliability.

Essential: Recent, hands‑on experience personally building production LLM evaluation and monitoring pipelines. Experience limited to traditional ML model monitoring or building RAG applications without ongoing evaluation will not be sufficient.

Strong Python engineering skills are required, with AWS experience highly desirable.

The Role

You’ll take ownership of monitoring, troubleshooting and improving live AI systems, working across LLMs, agentic workflows, retrieval pipelines and cloud infrastructure.

A key focus will be ensuring AI applications don’t just remain available, but continue to produce accurate, consistent and useful results.

Key responsibilities:
  • Build and maintain monitoring, observability and evaluation pipelines for production AI and LLM applications.
  • Develop automated evaluation frameworks to measure response accuracy, retrieval quality, hallucinations, consistency and performance.
  • Monitor and troubleshoot agentic AI workflows, including tool calls, orchestration failures, context handling and multi-step execution.
  • Identify and investigate changes in model behaviour, output quality and performance over time.
  • Build dashboards, alerts and quality metrics to help the team understand how AI applications are performing.
  • Support model upgrades, prompt changes, version management and provider transitions, ensuring changes don’t negatively impact quality.
  • Investigate production incidents across AI applications, data pipelines, APIs and supporting infrastructure.
  • Work closely with AI Engineers and the wider technology team to improve reliability, performance and operational processes.
What They’re Looking For

We're particularly interested in candidates who can demonstrate real commercial experience operating AI systems in production, rather than solely building prototypes or proofs of concept.

  • Hands‑on experience building or maintaining monitoring and evaluation pipelines for production AI, ML or LLM applications.
  • Strong Python development and debugging skills.
  • Experience with observability, logging, tracing, alerting and troubleshooting production systems.
  • Understanding of LLM behaviour, including prompting, context management, output consistency and evaluation.
  • Experience deploying, monitoring or supporting applications in AWS.
  • Familiarity with CI/CD, automated testing and production software operations.
  • Strong problem‑solving skills and the ability to investigate issues across multiple interconnected systems.
Particularly interesting experience:
  • LLM evaluation frameworks, automated regression testing and hallucination detection.
  • Agentic AI frameworks such as LangGraph, LangChain or similar.
  • LLM observability tools such as Langfuse, LangSmith, Helicone or equivalent.
  • RAG pipelines, retrieval evaluation and vector databases.
  • MLflow, Databricks, SageMaker or Bedrock.
  • Model routing, fallback strategies, provider management and inference monitoring.
  • Kubernetes, Docker and Terraform.

You don't need experience with every tool listed. We're more interested in what you've personally built, monitored and improved in production.

Why Consider It?

This is an opportunity to join a business making a significant investment in applied AI, with the chance to influence how its AI systems are monitored, evaluated and operated.

You’ll be working on real production applications, collaborating with an established engineering team and helping shape the standards for AI reliability and quality across the organisation.

Obtenez votre examen gratuit et confidentiel de votre CV.

ou faites glisser et déposez votre fichier ici.

Similar jobs

Postes similaires à comparer

AI / Machine Learning Engineer – Agentic LLM Systems (Contract)
AI / Machine Learning Engineer – Agentic LLM Systems (Contract)

Experis UK • Greater London

Hybride
GBP 80 000 - 120 000
6 Months Initial Contract
Hybrid | London
Backend Software Engineer Python LLM - Finance
Backend Software Engineer Python LLM - Finance

Client Server Limited • Greater London

Hybride
GBP 81 000 - 99 000
Pension
Life Assurance
Income Protection
+5
Artificial Intelligence Engineer
Artificial Intelligence Engineer

Intelix.AI • Greater London

Hybride
GBP 150 000 - 200 000
Equity
Benefits package
MLOps Engineer
MLOps Engineer

EPAM Systems Inc • Greater London

Hybride
GBP 90 000 - 130 000
EPAM Employee Stock Purchase Plan (ES"
AI Engineer
AI Engineer

Hyre Ai • City Of London

Sur place
GBP 70 000 - 120 000
AI Engineer - LLM & Agentic Systems
AI Engineer - LLM & Agentic Systems

Intec Select • Greater London

Sur place
GBP 111 000 - 120 000
Remote UK LLMOps Engineer: Production AI Reliability
Remote UK LLMOps Engineer: Production AI Reliability

Better Placed • Grande-Bretagne

À distance
GBP 70 000 - 110 000
AI Engineer (GenAI / LLM / RAG / Agentic AI)
AI Engineer (GenAI / LLM / RAG / Agentic AI)

Ampstek • Greater London

Hybride
GBP 101 000 - 157 000
Artificial Intelligence Engineer
Artificial Intelligence Engineer

Altech Group Ltd • Grande-Bretagne

À distance
GBP 70 000 - 120 000
AI Engineer
AI Engineer

eFinancialCareers • Greater London

Hybride
GBP 90 000 - 170 000