Remote AI Evaluation & Benchmark Engineer

Intergral GmbH; United Kingdom; Remote; GBP 60,000 - 75,000; Full time

Job summary

Intergral UK is seeking an AI Evaluation & Benchmarking Engineer to design and implement systems that measure the quality of OpsPilot. You will lead automated evaluations, create synthetic workloads, and benchmark models, prompts, tools, and workflows against repeatable baselines.

You will work with cross-functional teams to extend evaluation across customer journeys, APIs, and backend services, using OpenTelemetry to ensure representative benchmarks and correlate results with metrics, logs, and

Qualifications

• Practical experience working with LLMs, AI agents or AI evaluation.
• Strong software engineering skills, particularly Python or a similar language.
• Experience building automated evaluation, benchmarking, testing or experimentation infrastructure.
• Experience creating synthetic workloads, datasets or evaluation scenarios.
• An understanding of non-deterministic evaluation, including repeated measurement, variance and distinguishing meaningful changes from noise.
• The ability to turn complex system behaviour into measurable criteria.
• Comfort working across APIs, distributed systems and multiple layers of a software product.

Responsibilities

• Evaluate the agent and the product it runs on.
• Across the wider product, extend evaluation across important customer journeys, APIs, backend services and UI.
• Turn failures and real-world problems into new evaluation scenarios.
• Identify recurring failure patterns and capability gaps.
• Test potential improvements against baselines, holdouts and unseen scenarios.
• Detect regressions, benchmark overfitting and improvements that don\'t generalise.
• Work directly with AI, software, platform and SRE engineers to investigate findings and improve the product.

Skills

LLM experience

Python

AI evaluation

Automation/testing infrastructure

Synthetic workloads/datasets

Non-deterministic evaluation

APIs and distributed systems

Tools

OpenTelemetry

Job description

Intergral UK is seeking an AI Evaluation & Benchmarking Engineer to design and implement systems that measure the quality of OpsPilot. You will lead automated evaluations, create synthetic workloads, and benchmark models, prompts, tools, and workflows against repeatable baselines.

You will work with cross-functional teams to extend evaluation across customer journeys, APIs, and backend services, using OpenTelemetry to ensure representative benchmarks and correlate results with metrics, logs, and