AI Evaluation Engineer: Agentic Benchmarks & Production

Teli Labs

Greater London

On-site

GBP 90,000 - 130,000

Full time

40 hours ago
Be an early applicant

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Job summary

PhysicsX is London-based and seeks an AI Engineer: Evals to design and build the evaluation discipline for agents across production systems. You will provide shared infrastructure, frameworks, and coaching to product, delivery, and customer teams, enabling robust eval suites and credible benchmarks for real engineering workflows.

You will own cross-product evaluation, publish credible benchmarks, and work with Research and Post-Training to ensure integrity of evaluations while enabling teams to

Qualifications

  • Experience building evaluation systems for agents or LLM-powered products in production.
  • Ability to translate product goals into datasets, evaluators and decision criteria.
  • Understanding failure modes of probabilistic systems and using statistical judgement.
  • Knowledge of deterministic evaluators, domain experts, human review and LLM-as-a-judge.

Responsibilities

  • Lead the technical strategy and implementation of how PhysicsX evaluates agents.
  • Build reusable evaluation frameworks and reference implementations.
  • Help teams establish robust eval suites and publish open benchmarks.
  • Develop datasets and evaluators combining deterministic checks, domain judgement and human review.
  • Connect offline evaluation with production telemetry to detect drift.

Skills

Evaluation systems design
Dataset design
Production tooling
Cross-team collaboration

Education

Bachelor's degree in CS/Math/Engineering

Tools

Python
LLM evaluation tooling

Job description

PhysicsX is London-based and seeks an AI Engineer: Evals to design and build the evaluation discipline for agents across production systems. You will provide shared infrastructure, frameworks, and coaching to product, delivery, and customer teams, enabling robust eval suites and credible benchmarks for real engineering workflows.

You will own cross-product evaluation, publish credible benchmarks, and work with Research and Post-Training to ensure integrity of evaluations while enabling teams to

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

AI Evaluation Architect for Production Agents
AI Evaluation Architect for Production Agents

PhysicsX • Greater London

On-site
GBP 90,000 - 120,000
AI Engineer: Evals
AI Engineer: Evals

Teli Labs • Greater London

On-site
GBP 90,000 - 130,000
AI Engineer: Evals
AI Engineer: Evals

PhysicsX • Greater London

On-site
GBP 90,000 - 120,000
Agentic Evaluation Engineer — Benchmarks & Systems
Agentic Evaluation Engineer — Benchmarks & Systems

AI Startups UK • Greater London

Hybrid
GBP 90,000 - 130,000
Competitive salary
Equity ownership
Private healthcare
+1
Staff Evaluations Engineer — AI Benchmarks
Staff Evaluations Engineer — AI Benchmarks

Ai • Greater London

On-site
GBP 85,000 - 130,000
30 days annual leave
Pension plan with 10% employer contrib
Bupa private health cover
+1
Staff Research Engineer: Agent Evals & Post-Training (London)
Staff Research Engineer: Agent Evals & Post-Training (London)

Nominal • Greater London

On-site
GBP 180,000 - 220,000
Unlimited PTO
Free meals
Dev stipend
+4
Staff Research Engineer: Agentic Evaluation & Benchmarks
Staff Research Engineer: Agentic Evaluation & Benchmarks

Callosum Technologies Ltd. • Greater London

Hybrid
GBP 90,000 - 140,000
Competitive salary
Equity & Ownership
Private healthcare
+2
AI Safety Benchmark Engineer
AI Safety Benchmark Engineer

Moonfire • Greater London

Hybrid
GBP 90,000 - 130,000
Equity
Hybrid work model
Office in London/Paris
+3
AI Evaluation Engineer — Hybrid, Public-Impact Tech
AI Evaluation Engineer — Hybrid, Public-Impact Tech

i.AI • City Of London

Hybrid
GBP 46,000 - 63,000
Senior Applied AI Engineer: Production, Evaluation & Impact
Senior Applied AI Engineer: Production, Evaluation & Impact

Scale AI • City Of London

On-site
GBP 90,000 - 150,000