AI Assurance Lead

Fruition IT; Greater London; Hybrid; GBP 100,000 - 120,000; Full time

Job summary

Fruition IT in London offers a hybrid, permanent AI Safety & Assurance Lead role. It combines hands-on machine learning engineering with safety, model evaluation, and assurance across regulated AI products.

The role emphasizes testing models in real-world environments, building end-to-end evaluation frameworks, and contributing to broader assurance programs. The position focuses on hands-on technical work (about 75%), with collaboration across engineering, product and data science teams.

Qualifications

• Production Python is used for building scalable ML systems.
• Hands-on experience with LLMs and modern ML frameworks.

Responsibilities

• Design evaluation frameworks and test harnesses for AI models.
• Evaluate and benchmark LLMs against real-world use cases.
• Measure precision, recall, grounding, calibration and hallucination.
• Red team systems for prompt injection, jailbreaks and data leakage.
• Stress-test models with adversarial and out-of-distribution data.
• Test agent guardrails, permissions and containment.
• Build automated evaluation into CI/CD pipelines.
• Monitor deployed models for drift and regression.
• Support fine-tuning and model adaptation.
• Produce technical assurance evidence and present findings to stakeholders.

Skills

Production Python

PyTorch

LLM experience

RAG

Vector search

Prompting

Agent orchestration

CI/CD

Cloud deployment

Containers

Statistics

Adversarial ML

AI safety

Tools

PyTorch

Hugging Face Transformers

Job description

AI Safety & Assurance Lead

London | Hybrid | Permanent

Security Clearance: Current SC or eligibility to obtain SC

Salary: £100,000 - £120,000 + benefits

We’re working with a specialist UK technology business delivering advanced AI solutions into Defence, Government and other highly regulated environments.

They’re looking for an experienced Machine Learning Engineer or Data Scientist to lead technical AI safety, model evaluation and assurance across a growing portfolio of AI products.

This is not a governance-only role. Around 75% of the position is hands‑on technical work, testing whether models, pipelines and AI agents perform safely and reliably in real-world environments.

The role

You’ll take technical ownership of AI evaluation and assurance, working closely with engineering, product and data science teams.

Responsibilities will include:

• Designing model evaluation frameworks and test harnesses
• Evaluating and benchmarking LLMs against real-world use cases
• Measuring areas including precision, recall, grounding, calibration and hallucination
• Red teaming systems for prompt injection, jailbreaks, data leakage and tool misuse
• Stress‑testing models using adversarial and out‑of‑distribution data
• Testing agent guardrails, permissions and containment
• Building automated evaluation into CI/CD pipelines
• Monitoring deployed models for drift and regression
• Supporting fine‑tuning and model adaptation
• Producing technical assurance evidence and presenting findings to senior stakeholders and customers

You’ll also contribute to the wider AI assurance programme, including internal audit, certification and customer assurance activity.

What we’re looking for

You’ll bring a strong Machine Learning Engineering or Data Science background alongside hands‑on experience taking AI systems into production.

Key experience includes:

• Production Python
• PyTorch, Hugging Face Transformers or similar
• Strong hands‑on LLM experience
• RAG, vector search, prompting, tool use and agent orchestration
• Designing and running model evaluations
• Cloud deployment, containers and CI/CD
• Understanding AI attack surfaces and failure modes
• Applied statistics and experimental design

Experience in AI safety, adversarial ML, AI red teaming, technical assurance or Defence/Secure Government environments would be highly advantageous.

A rare opportunity to combine hands‑on AI engineering with safety and assurance, working on technically challenging systems where reliability and trust genuinely matter.