Senior Applied AI scientist

Few&Far

United States

Remote

GBP 111,000 - 138,000

Full time

2 days ago
Be an early applicant

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Job summary

Few&Far is seeking a Senior Applied AI Scientist to drive evaluation frameworks for AI applications from PoC to production, with a fully remote setup. The role focuses on rigorous measurement, evidence-driven release gates, and building repeatable evaluation processes.

You will define metrics for LLM/RAG systems, create golden datasets, perform retrieval and end-to-end quality assessments, and collaborate with engineers to ensure reliable AI outcomes.

Qualifications

  • Hands-on AI/ML experience focused on evaluation.
  • Experience turning PoC into production with evidence.
  • Experience with LLM-based systems and retrieval evaluation.

Responsibilities

  • Design and build evaluation frameworks for LLM and RAG apps.
  • Define metrics, benchmarks and acceptance criteria for production readiness.
  • Create golden datasets and ground-truth benchmarks with SMEs.
  • Evaluate retrieval precision, recall, groundedness, faithfulness and end-to-end quality.
  • Develop regression tests to maintain quality over model iterations.
  • Collaborate with engineers and AI specialists across LLMs, RAG, retrieval, and knowledge graphs.
  • Lead PoC to production evaluations and establish release gates.

Skills

LLMs & RAG
Evaluation frameworks
Golden datasets
Human evaluation
Python

Tools

Ragas
DeepEval
LangSmith
PromptFoo

Job description

I’m currently working on an exciting contract opportunity for a Senior Applied AI Scientist to join a growing AI business, working on taking AI applications from proof of concept through to production.

Up to £750 a day- Ideally around the £600-£650 mark though!

Fully remote

The key requirement for this role is strong experience in AI evaluations and setting up evaluation frameworks. You’ll be responsible for establishing how AI systems are measured, tested and improved, ensuring there is clear evidence behind the quality and reliability of the applications being built.

What you’ll be doing:
  • Designing and building evaluation frameworks for LLM and RAG-based applications
  • Defining metrics, benchmarks and acceptance criteria to determine whether an AI application is ready for production
  • Building golden datasets and ground-truth benchmarks, working with subject matter experts to establish what good looks like
  • Evaluating areas including retrieval precision, recall, groundedness, faithfulness and end-to-end answer quality
  • Developing regression testing to ensure quality remains consistent as models, prompts and data evolve
  • Working with LLM-as-a-judge methodologies, calibrated against human judgement
  • Using evaluation results and production feedback to identify failure modes and continuously improve AI systems
  • Taking AI applications from PoC through to production, establishing the evaluation gates and evidence required for release
  • Working closely with engineers and AI specialists across LLMs, RAG, retrieval and knowledge graphs
What we’re looking for:

Someone with a strong hands-on background in Applied AI / ML, particularly around evaluating and improving LLM-based systems.

You should have experience with:

  • LLMs and RAG
  • Building evaluation frameworks from scratch
  • Golden datasets and benchmark creation
  • LLM-as-a-judge / human evaluation
  • Retrieval evaluation and optimisationii>
  • Python and modern AI engineering practices

Experience with tools such as Ragas, DeepEval, LangSmith or promptfoo would be highly beneficial.

This is a great opportunity for someone who enjoys being responsible for proving whether AI actually works, rather than simply building the models themselves.

If you have strong experience across LLM evaluations, RAG and evaluation frameworks and are available for an ASAP start.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

AI Evaluation Engineer
AI Evaluation Engineer

DeepRec.ai • Denver (CO)

Remote
USD 162,000 - 198,000
AI Systems Engineer
AI Systems Engineer

Harnham • United States

Hybrid
GBP 60,000 - 95,000
Hybrid working with flexibility
Technical progression opportunities
Supportive environment
AI Engineer – Applied AI & Agentic Systems
AI Engineer – Applied AI & Agentic Systems

Signify Technology • United States

Remote
USD 150,000 - 180,000
Senior AI Observability & Evaluation Engineer
Senior AI Observability & Evaluation Engineer

CloudBigD • Dallas (TX)

On-site
USD 89,000 - 126,000
Senior AI Observability & Evaluation Engineer
Senior AI Observability & Evaluation Engineer

Valiant Technologies LLC • Dallas (TX)

On-site
USD 89,000 - 126,000
Senior AI Observability & Evaluation Engineer
Senior AI Observability & Evaluation Engineer

Quantum Technologies LLC • Dallas (TX)

On-site
USD 91,000 - 124,000
Senior AI Observability & Evaluation Engineer
Senior AI Observability & Evaluation Engineer

Robotic Process Automation LLC • Dallas (TX)

On-site
USD 89,000 - 126,000
Senior AI Observability & Evaluation Engineer
Senior AI Observability & Evaluation Engineer

Ethereum Technologies LLC • Dallas (TX)

On-site
USD 65,000 - 107,000
Senior AI Engineer
Senior AI Engineer

arosplatforms | AI Consulting & Services • North Township (IN)

On-site
USD 120,000 - 170,000
AI Engineer
AI Engineer

LA International • United States

Remote
USD 129,000 - 154,000