AI Evaluation Engineer — Quality & Risk for AI Systems

SmartSourcing Ltd; Greater London; Hybrid; GBP 118,000 - 159,000; Full time

Job summary

SmartSourcing Ltd is seeking a hands-on AI evaluation engineer for a 4-month contract in London, Bristol, or Manchester with two days per week onsite. You will design and deploy AI evaluation tooling for AI/ML and LLM-based products, build evaluation harnesses in Python, and collaborate with technical and non-technical stakeholders to ensure quality, safety, reliability and governance.

Active SC clearance is preferred or eligible; suitable candidates will have deep expertise in AI evaluation

Qualifications

• Strong AI/ML evaluation expertise with hands-on testing and quality assurance experience.
• Deep expertise in evaluating AI/ML/LLM-based systems in production.
• Proven leadership across teams and organisational boundaries.
• Experience designing AI evaluation frameworks and tools (Python).
• Active SC clearance preferred / eligible and willing to undergo SC Clearance.

Responsibilities

• Design, develop and deploy AI evaluation and risk management tooling.
• Build and maintain evaluation frameworks for AI, ML and LLM-based products.
• Create and develop code solutions, harnesses and testing capabilities from the ground up using Python.
• Define and execute evaluation strategies across model, agentic and application layers.
• Assess AI system quality, performance, reliability, safety and risk.
• Work closely with technical and non-technical stakeholders across multiple departments and organisations.
• Provide technical leadership and guidance on AI evaluation best practice.

Skills

AI evaluation

Python development

Leadership

Stakeholder engagement

Quality assurance

ML/LLM evaluation

Production systems

Tools

RAGAS

DeepEval (DP-Eval)

Job description

SmartSourcing Ltd is seeking a hands-on AI evaluation engineer for a 4-month contract in London, Bristol, or Manchester with two days per week onsite. You will design and deploy AI evaluation tooling for AI/ML and LLM-based products, build evaluation harnesses in Python, and collaborate with technical and non-technical stakeholders to ensure quality, safety, reliability and governance.

Active SC clearance is preferred or eligible; suitable candidates will have deep expertise in AI evaluation