Data Analyst: AI Agent Search Quality & Evaluations

Nebius Group; Greater London; On-site; GBP 45,000 - 70,000; Full time

Benefits offered by this job

Competitive compensation

Career growth

Flexibility and ownership

Collaborative culture

Impactful AI projects

International teams

Fast-moving environment

Job summary

Nebius Group in London is seeking an experienced Data Analyst to own end-to-end quality for AI-powered search components. You will design evaluation systems, build pipelines, and partner with engineers and product managers to improve outcomes.

The role demands hands-on data engineering, solid statistical thinking, and the ability to translate user needs into measurable metrics. You will influence product direction and ensure robust, scalable evaluation is at the heart of development.

Qualifications

• Substantial hands-on experience designing and building evaluation or quality measurement systems for complex technology products.
• Strong Python skills and an engineering mindset.
• Foundation in mathematical statistics and practical experience with experimental design, sampling, uncertainty and measurement bias.
• Strong analytical judgment and ability to translate understanding into actionable evaluations.
• Excellent communication and collaboration skills to influence technical and product decisions.

Responsibilities

• Own the quality measurement and improvement cycle end-to-end for AI search components.
• Design, build, and evolve evaluation systems, including datasets, benchmarks, metrics, and automated pipelines.
• Evaluate response quality across relevance, completeness, freshness, factual accuracy, and usefulness.
• Investigate quality regressions and metric anomalies, diagnose root causes and propose impactful improvements.
• Prioritize product and engineering roadmap with explicit quality trade-offs.

Skills

Python

Statistics

Experiment design

Data evaluation

Communication

Job description

Nebius Group in London is seeking an experienced Data Analyst to own end-to-end quality for AI-powered search components. You will design evaluation systems, build pipelines, and partner with engineers and product managers to improve outcomes.

The role demands hands-on data engineering, solid statistical thinking, and the ability to translate user needs into measurable metrics. You will influence product direction and ensure robust, scalable evaluation is at the heart of development.