AI Harness Engineer — Equity + Hybrid Work

Valarian; Greater London; Hybrid; GBP 90,000 - 130,000; Full time

Benefits offered by this job

Equity

Competitive salary

Employer pension

Private health insurance

Hybrid work setup

Team retreats

Job summary

Valarian Technologies is a dual-use technology company building critical tools to safeguard the future in an era of evolving global security challenges. We’re seeking an AI Harness Engineer to own the experimental design, evaluation methodologies, and benchmark infrastructure for our AI models and autonomous agent workloads.

You will bridge data science, statistical validation, and production scaffolding to drive robust harness architecture, curate evaluation datasets, and calibrate

Qualifications

• Expertise in hypothesis testing, bootstrapping, and power analysis within probabilistic ML systems.
• Experience validating LLM judge setups against human annotations and tracking agreement metrics.
• Experience designing evaluation datasets, scoring rubrics and measuring inter-annotator agreement.
• Ability to evaluate non-deterministic multi-step agent trajectories and tool execution sequences.
• Strong judgment in defining metrics tied to real-world outcomes and detecting benchmark gaming.

Responsibilities

• Architect Evaluation Runtimes & Harnesses: design scalable Python evaluation harnesses for multi-step agent systems focusing on accuracy, latency and cost.
• Design Experiments & Validate Results: apply statistical methods to determine true improvements vs stochastic noise.
• Build & Calibrate LLM‑as‑a‑Judge Pipelines: automate judging validated against human ground truth and measure agreement metrics.
• Curate Benchmark Datasets & Rubrics: define sampling, annotation guidelines and scoring rubrics with high benchmark integrity.
• Deep Error & Trajectory Analysis: perform failure analysis and translate findings into actionable improvements.
• Drive Harness Scaffolding Iteration: implement architectural improvements across agent scaffolding, prompt design, and safety guardrails.

Skills

Statistics & design

LLM evaluation

Non-deterministic eval

Python

LLM integration

Tools

LangChain

Job description

Valarian Technologies is a dual-use technology company building critical tools to safeguard the future in an era of evolving global security challenges. We’re seeking an AI Harness Engineer to own the experimental design, evaluation methodologies, and benchmark infrastructure for our AI models and autonomous agent workloads.

You will bridge data science, statistical validation, and production scaffolding to drive robust harness architecture, curate evaluation datasets, and calibrate