LLM Ops Engineer

Quantios; Fleet; On-site; GBP 80,000 - 100,000; Full time

Job summary

A leading AI solutions firm in England is seeking an LLMOps Engineer to design and maintain LLM pipelines and RAG architectures. In this hands-on role, you will build scalable AI infrastructure using cloud services and support AI product development. The ideal candidate has over 4 years of software and data engineering experience, particularly in cloud environments, and strong skills in Python and modern AI frameworks. Collaboration with multidisciplinary teams is essential.

Qualifications

• 4+ years of experience in software engineering, data engineering, or DevOps.
• Hands-on experience with modern AI frameworks and Python.
• Strong understanding of CI/CD and version control.

Responsibilities

• Design and maintain LLM pipelines and RAG architectures.
• Build and operate AI infrastructure using cloud services.
• Ensure observability and quality in LLM systems.

Skills

Python

Cloud-based AI solutions

CI/CD

Kubernetes

Observability tooling

Problem-solving

Collaboration

Education

Bachelor’s degree in Computer Science or related field

Tools

Azure AI Foundry

LangChain

Azure DevOps

Job description

As an LLMOps Engineer at Quantios, you will play a foundational role in building and operating the company’s first generation of Large Language Model–powered agentic products. You will work closely with AI developers, architects, DevOps engineers, and Product Owners to design, deploy, monitor, and optimise LLM pipelines, RAG architectures, and agent-based systems. This is a hands‑on role suited to someone who enjoys solving complex technical problems, building scalable AI infrastructure, and shaping early‑stage best practices.

Job Responsibilities

• Model, Data, and RAG Pipelines
  • Design, implement, and maintain ingestion pipelines for LLM training and retrieval‑augmented generation (RAG) datasets.
  • Develop and optimise chunking, embedding, enrichment, and indexing processes using LangChain or equivalent frameworks.
  • Manage the lifecycle of prompt templates, embedding models, LLM chains, evaluators, and model configurations.
  • Support experimentation, evaluation, and benchmarking of foundation models, prompts, and retrieval strategies.
• LLM Infrastructure & Operations
  • Build and operate infrastructure for AI components using Azure AI Foundry, Azure OpenAI, Azure App Services, and related cloud services.
  • Implement secure hosting for RAG applications, vector databases, and agent runtimes.
  • Define and maintain CI/CD pipelines for LLM artefacts (datasets, prompts, model configs, evaluation suites) using Azure DevOps.
  • Collaborate with DevOps engineers to support environment provisioning, scalability, reliability, and performance.
• Observability, Quality & Monitoring
  • Establish foundational observability for LLM‑based systems, including telemetry, latency tracking, cost visibility, and model diagnostics.
  • Monitor and surface signals such as hallucination rates, evaluation scores, retrieval quality, and content safety triggers.
  • Implement automated evaluation pipelines for prompts, responses, and RAG relevance metrics.
  • Ensure LLM quality gates are integrated into CI/CD workflows.
• Security, Governance & Compliance
  • Apply responsible AI principles in line with Quantios’ AI and ISMS policies.
  • Ensure privacy, access control, and logging for all model interactions and vector index operations.
  • Support red‑team style penetration testing for prompt injection, leakage, and model‑based social engineering risks.
  • Contribute to documenting LLM pipelines, governance patterns, and internal standards.
• Collaboration & Delivery
  • Work with AI developers to integrate LLM and RAG components into product features.
  • Partner with Portfolio Architects to evaluate new AI technologies, patterns, and architectural approaches.
  • Collaborate with Product Owners to shape technical feasibility, performance considerations, and release planning for AI‑enabled features.
  • Participate in Agile ceremonies, contribute to estimation, and help the team deliver high‑quality AI capabilities.
• Continuous Improvement & Innovation
  • Stay up to date with emerging tools in LLMOps, RAG optimisation, evaluation methodologies, and vector search technologies.
  • Propose improvements to scalability, model performance, prompt engineering practices, and developer workflows.
  • Contribute to establishing early LLMOps best practices that will scale as the organisation’s AI capability grows.

Job Requirements

• Bachelor’s degree in Computer Science, Software Engineering, Data Engineering, or a related field; or equivalent industry experience.
• 4+ years of experience in software engineering, data engineering, machine learning engineering, or DevOps—preferably within cloud environments.
• Hands‑on experience with Python and modern AI frameworks (e.g., LangChain, Semantic Kernel, MC‑based tools, or equivalent).
• Experience operating cloud‑based AI solutions using Azure AI Foundry, Azure OpenAI, Azure App Services, Azure Storage, or similar services.
• Familiarity with vector databases, embeddings, and retrieval pipelines (Azure AI Search, Pinecone, Chroma, Redis Vector, or similar).
• Strong understanding of CI/CD, version control, and environment management (Azure DevOps preferred).
• Experience with container orchestration using Kubernetes (AKS or equivalent) and containerized deployments.
• Experience with observability tooling and practices (Azure Monitor, logging, tracing, metrics).
• Knowledge of modern front‑end or service development technologies (React, TypeScript, C#, or equivalent) is beneficial.
• Strong problem‑solving, analytical, and debugging skills with a passion for building reliable AI‑driven systems.
• Excellent communication skills and ability to collaborate across multidisciplinary teams