AI Evaluation & Quality Analyst

Nebius Group; Greater London; Hybrid; GBP 55,000 - 90,000; Full time

Benefits offered by this job

Competitive compensation

Career growth and learning

Flexibility and ownership

Collaborative and innovative culture

Impactful AI projects

International environment and talented

Job summary

Nebius is seeking an experienced Data Analyst to own end-to-end quality in search for AI agents. You will design evaluation systems, build datasets and metrics, and work with engineers to improve outcomes across a fast-moving AI platform.

The role requires strong Python skills, statistical expertise, and the ability to translate user needs into actionable evaluations that guide product decisions. London-based hybrid work and competitive benefits are offered.

Qualifications

• Substantial hands-on experience designing and building evaluation or quality measurement systems for complex technology products.
• Experience owning ambiguous problems from framing through implementation and measurable improvement.
• Strong Python skills and engineering mindset for reliable data pipelines.

Responsibilities

• Own the quality measurement and improvement cycle end-to-end: identify user needs, define success, build evaluations, diagnose failures, and work with engineers and product managers to improve outcomes.
• Design, build, and evolve evaluation systems, including datasets, benchmarks, metrics, and automated pipelines that support development and launch decisions.
• Evaluate response quality across dimensions: relevance, completeness, freshness, factual accuracy, usefulness to downstream agents.
• Investigate quality regressions and metric anomalies, uncover root causes, identify improvements with greatest impact.
• Help prioritize the product and engineering roadmap, making quality trade-offs explicit and assessing impact for users.

Skills

Python

Statistical analysis

Experimental design

Data quality assurance

Communication & collaboration

Education

Bachelor's degree in a quantitative field

Tools

SQL

Pandas/Numpy

Airflow

Notebook environments

Job description

Nebius is seeking an experienced Data Analyst to own end-to-end quality in search for AI agents. You will design evaluation systems, build datasets and metrics, and work with engineers to improve outcomes across a fast-moving AI platform.

The role requires strong Python skills, statistical expertise, and the ability to translate user needs into actionable evaluations that guide product decisions. London-based hybrid work and competitive benefits are offered.