Senior Applied AI scientist

Few&Far; United States; Remote; GBP 111,000 - 129,000; Full time

Job summary

Few&Far is hiring a Senior Applied AI Scientist to drive evaluation and production-readiness for LLM-based AI applications. You will design evaluation frameworks, establish metrics, and create gold datasets to prove performance and reliability across PoC to production transitions.

This remote UK role requires hands-on AI/ML expertise, strong Python coding, and experience with LLMs, RAG, and retrieval evaluation.

Qualifications

• Strong hands-on background in Applied AI/ML, evaluating and improving LLM-based systems.
• Experience with LLMs and RAG for evaluation.
• Ability to build evaluation frameworks from scratch and establish benchmarks.
• Experience with golden datasets and human evaluation methodologies.
• Familiarity with retrieval evaluation and optimization.
• Proficiency in Python and modern AI engineering practices.

Responsibilities

• Design and build evaluation frameworks for LLM and RAG-based applications.
• Define metrics, benchmarks, and acceptance criteria for production readiness.
• Create golden datasets and ground-truth benchmarks with subject matter experts.
• Evaluate retrieval precision, recall, groundedness, faithfulness, and end-to-end quality.
• Develop regression tests to ensure quality as models evolve.
• Use LLM-as-a-judge approaches and human judgement as calibration.
• Take PoC AI apps through to production with clear evaluation gates and evidence.

Skills

Applied AI/ML

LLMs

RAG

Evaluation frameworks

Golden datasets

Human evaluation

Retrieval evaluation

Python

Tools

Ragas

DeepEval

LangSmith

promptfoo

Job description

I’m currently working on an exciting contract opportunity for a Senior Applied AI Scientist to join a growing AI business, working on taking AI applications from proof of concept through to production.

Up to £700 a day- Ideally around the £600-£650 mark though!Fully remote

The key requirement for this role is strong experience in AI evaluations and setting up evaluation frameworks. You’ll be responsible for establishing how AI systems are measured, tested and improved, ensuring there is clear evidence behind the quality and reliability of the applications being built.

What you’ll be doing:

• Designing and building evaluation frameworks for LLM and RAG-based applications
• Defining metrics, benchmarks and acceptance criteria to determine whether an AI application is ready for production
• Building golden datasets and ground-truth benchmarks, working with subject matter experts to establish what good looks like
• Evaluating areas including retrieval precision, recall, groundedness, faithfulness and end-to-end answer quality
• Developing regression testing to ensure quality remains consistent as models, prompts and data evolve
• Working with LLM-as-a-judge methodologies, calibrated against human judgement
• Using evaluation results and production feedback to identify failure modes and continuously improve AI systems
• Taking AI applications from PoC through to production, establishing the evaluation gates and evidence required for release
• Working closely with engineers and AI specialists across LLMs, RAG, retrieval and knowledge graphs

What we’re looking for:

Someone with a strong hands-on background in Applied AI / ML, particularly around evaluating and improving LLM-based systems.

You should have experience with:

• LLMs and RAG
• Building evaluation frameworks from scratch
• Golden datasets and benchmark creation
• LLM-as-a-judge / human evaluation
• Retrieval evaluation and optimisation
• Python and modern AI engineering practices

Experience with tools such as Ragas, DeepEval, LangSmith or promptfoo would be highly beneficial.

This is a great opportunity for someone who enjoys being responsible for proving whether AI actually works, rather than simply building the models themselves.