Senior AI Evaluation Scientist — PoC to Production (Remote)

Few&Far; United States; Remote; GBP 111,000 - 129,000; Full time

Job summary

Few&Far is hiring a Senior Applied AI Scientist to drive evaluation and production-readiness for LLM-based AI applications. You will design evaluation frameworks, establish metrics, and create gold datasets to prove performance and reliability across PoC to production transitions.

This remote UK role requires hands-on AI/ML expertise, strong Python coding, and experience with LLMs, RAG, and retrieval evaluation.

Qualifications

• Strong hands-on background in Applied AI/ML, evaluating and improving LLM-based systems.
• Experience with LLMs and RAG for evaluation.
• Ability to build evaluation frameworks from scratch and establish benchmarks.
• Experience with golden datasets and human evaluation methodologies.
• Familiarity with retrieval evaluation and optimization.
• Proficiency in Python and modern AI engineering practices.

Responsibilities

• Design and build evaluation frameworks for LLM and RAG-based applications.
• Define metrics, benchmarks, and acceptance criteria for production readiness.
• Create golden datasets and ground-truth benchmarks with subject matter experts.
• Evaluate retrieval precision, recall, groundedness, faithfulness, and end-to-end quality.
• Develop regression tests to ensure quality as models evolve.
• Use LLM-as-a-judge approaches and human judgement as calibration.
• Take PoC AI apps through to production with clear evaluation gates and evidence.

Skills

Applied AI/ML

LLMs

RAG

Evaluation frameworks

Golden datasets

Human evaluation

Retrieval evaluation

Python

Tools

Ragas

DeepEval

LangSmith

promptfoo

Job description

Few&Far is hiring a Senior Applied AI Scientist to drive evaluation and production-readiness for LLM-based AI applications. You will design evaluation frameworks, establish metrics, and create gold datasets to prove performance and reliability across PoC to production transitions.

This remote UK role requires hands-on AI/ML expertise, strong Python coding, and experience with LLMs, RAG, and retrieval evaluation.