MLOps Engineer

Valarian Technologies; Greater London; Hybrid; GBP 90,000 - 150,000; Full time

Benefits offered by this job

Equity

Competitive salary

Employer pension contributions

Private health insurance

Hybrid work setup

Gym memberships

Company retreats

Job summary

Valarian Technologies, a frontier AI and sovereign data security company, is seeking an experienced Senior MLOps/LLMOps Engineer to lead deployment, monitoring, and orchestration of production LLM applications within a large Kubernetes environment. You will bridge AI development and production infrastructure, ensuring secure, scalable, multi-step workflows.

You will design end-to-end AI pipelines, implement robust telemetry, and optimize TTFT and throughput while collaborating in a hybrid

Qualifications

• 3+ years of experience in MLOps, LLMOps, Platform Engineering, AI Infrastructure, or SRE focusing on ML workloads.
• Hands-on experience deploying complex containerized apps on Kubernetes (EKS, GKE, AKS, or bare metal) at scale.
• Strong Python skills for backend automation, tooling, and integration development.
• Experience with monitoring/logging tools (Prometheus, Grafana, Datadog, OpenTelemetry) and LLM-specific tracing/evaluation platforms (LangSmith, Langfuse, TruLens).
• Familiarity with CI/CD pipelines (GitHub Actions, GitLab CI, ArgoCD) and IaC tools (Terraform, Pulumi).
• Deep understanding of agentic architectures, prompt chaining, function calling, and multi-agent coordination.

Responsibilities

• Design, deploy, and manage LLM services and agentic AI workloads in high-throughput Kubernetes clusters.
• Automate deployments, scalable inference endpoints, and rollout strategies (blue/green, canary) via CI/CD pipelines.
• Maintain production operation of agentic workflows: multi-step tool execution, state management, multi-agent coordination.
• Triage and resolve issues across AI app logic, orchestration, external APIs, and cloud infrastructure.
• Build end-to-end monitoring and Eval frameworks to measure AI output accuracy, drift, latency, cost, and hallucination.
• Implement trace observability for agent execution paths (prompts, completions, tool calls, steps).
• Monitor and optimize data pipelines, feature stores, vector indexing, and embedding updates.
• Manage resource allocation (GPU/CPU), memory, autoscaling, and cloud spend to optimize TTFT and throughput.

Skills

MLOps / LLMOps

Kubernetes

Python

CI/CD

Telemetry & monitoring

Agentic AI concepts

Tools

Kubernetes

Python

Prometheus

Grafana

Datadog

OpenTelemetry

GitHub Actions

GitLab CI

ArgoCD

Terraform

Pulumi

Job description

Valarian: A New Era of Sovereign Intelligence

Valarian is a dual use technology company building critical tools to safeguard the future in an era of evolving global security challenges. Backed by a Series A funding round of $50m led by NEA, we are rethinking security beyond traditional military domains to address asymmetric threats that impact our technological advantage, economic strength, and democratic institutions. We provide the control infrastructure that allows government and regulated enterprise to put frontier-class AI to work safely.

We are bringing a frontier AI experience inside secure boundaries for the first time. Our technology supports mission critical workflows across government, healthcare, and enterprise, including:

• Scaling open-weight models without external inference endpoints.
• Putting frontier class AI to work across organisations through partnerships with other leading AI startups such as Perplexity.
• Increasing operational resiliency for air gapped, on prem, and cloud based systems.
• Securing intellectual property and maintaining auditability for high stakes enterprise environments.
• Facilitating secure information flow during disaster response, M&A due diligence, and privileged legal communications.

We’ve only scratched the surface. Built in Britain by veterans of the Ministry of Defence, British Forces, Darktrace, and elite engineering backgrounds, we are driven by a shared mission to safeguard western democracy . Whether you join our London HQ or our wider global organisation, you’ll be part of a collaborative, high performing team creating the cutting edge software and infrastructure that defines the frontier of enterprise AI.

The Role

We are looking for an experienced Senior MLOps / LLMOps Engineer to lead the deployment, monitoring, orchestration, and operational reliability of our production LLM applications and agentic AI platform. In this role, you will bridge the gap between AI development and production infrastructure, ensuring multi-step agentic workflows, data pipelines, and retrieval-augmented systems run reliably, securely, and at scale within a large-scale Kubernetes environment.

What you’ll do:

• Design, deploy, and manage LLM services, agentic applications, and supporting AI workloads inside high-throughput Kubernetes clusters.
• Automate containerized deployments, scalable inference endpoints, and blue/green or canary rollout strategies for LLM/Agent releases via modern CI/CD pipelines.
• Support and maintain the production operation of agentic AI workflows, including multi-step tool execution, state management, and multi-agent coordination.
• Rapidly triage and resolve issues across AI application logic, orchestration frameworks, external API integrations, and underlying cloud infrastructure.
• Build end-to-end monitoring and evaluation (Eval) frameworks to measure and maintain AI output accuracy, drift, latency, cost, and hallucination metrics.
• Implement trace-level observability for agent execution paths (tracking prompts, completions, tool calls, and agent steps).
• Monitor and optimize real-time and batch data pipelines powering AI workloads, including feature stores, vector indexing, and embedding updates.
• Manage resource allocation (GPU/CPU utilization), memory footprint, autoscaling strategies, and cloud infrastructure expenditure to optimize time-to-first-token (TTFT) and throughput.

What we are looking for:

• 3+ years of experience in MLOps, LLMOps, Platform Engineering, AI Infrastructure, or SRE with a focus on machine learning workloads.
• Hands‑on experience managing and deploying complex containerized applications on Kubernetes (EKS, GKE, AKS, or bare metal) at scale.
• Strong proficiency in Python for backend automation, system debugging, custom tooling, and integration development.
• Proven experience with standard monitoring/logging tools (e.g., Prometheus, Grafana, Datadog, OpenTelemetry) as well as LLM‑specific tracing and evaluation platforms (e.g., LangSmith, Phoenix/Arize, Langfuse, TruLens).
• Familiarity with modern CI/CD pipelines (GitHub Actions, GitLab CI, ArgoCD) and Infrastructure as Code (IaC) tools (Terraform, Pulumi).
• Deep operational understanding of agentic architectures, prompt chaining, function calling/tool execution, and non-deterministic application behavior.

Nice to have:

• Practical experience operating Retrieval-Augmented Generation (RAG) pipelines and production vector databases (e.g., Pinecone, Qdrant, Milvus, pgvector, Weaviate).
• Hands‑on exposure to agent development and orchestration frameworks (e.g., LangChain, LangGraph, LlamaIndex, AutoGen, CrewAI).
• Experience with high-performance LLM serving engines and frameworks (e.g., vLLM, TGI, Ollama, TensorRT-LLM).
• Understanding of AI security practices (prompt injection mitigation, data privacy, PII masking) and rate limiting/cost guardrails.

Benefits:

• Equity – because you have the right to own what you’re building
• A competitive salary – because we value your unique skills
• Employer pension contributions – because you deserve a secure future
• Private health insurance - because your health is important to us
• Hybrid work setup – because everyone has different needs
• Gym memberships - because staying active fuels productivity
• Rewarding company retreats and meetups that respect your work/life balance – because we love getting to know each other

Life at Valarian

Our culture is built on inclusivity, compassion and flexibility – we want everyone to be empowered to achieve their goals at Valarian.

The work we do is vital, but so are the connections that make it happen. We thrive on the shared energy, spontaneous conversations, and mutual trust built when we spend time together. We operate on an office first hybrid model, regularly gathering in our London office to support one another and collaborate.

Valarian Technologies Limited is an equal opportunity employer and welcomes applications from invdividuals regardless of race, colour, religion, sex, sexual orientation, gender, identity or expression, national origin, age, disability, genetic information, marital status, veteran, amnesty, or any other legally protected characteristic.

We are committed to ensuring a fair and inclusive recruitment process and providing employment opportunities to all applicants. Decision recruitment, hiring, and employment are based solely on qualifications, skills, and experience relevant to the job requirements.

We may use artificial intelligence (AI) tools to support parts of the hiring process, such as reviewing applications, analysing resumes, or assessing responses and identifying potential inconsistencies or verification signals in application materials based on available information. These tools assist our recruitment team but do not replace human judgement. Final hiring decisions are ultimately made by humans. If you would like more information about how your data is processed, please contact us.