Lead AI Evaluation Engineer

Nebius Group; Greater London; On-site; GBP 65,000 - 85,000; Full time

Benefits offered by this job

Competitive compensation

Career growth and learning opportuniti

Flexibility and ownership

Collaborative and innovative culture

Opportunity to work on impactful AI项目

International environment and talented

Job summary

Nebius is seeking an experienced Data Analyst to own end-to-end quality evaluation for AI search systems. You will build datasets, benchmarks, metrics, and pipelines, and work with engineers and product managers to improve outcomes.

Strong Python and statistics background, plus experience turning data into reliable quality signals, is essential. Join Nebius to shape evaluation strategies in a fast-paced, international environment and contribute to cutting-edge AI projects with real impact across

Qualifications

• Substantial hands-on experience designing and building evaluation or quality measurement systems for complex technology products.
• Ability to own ambiguous problems from framing through implementation and measurable improvement.
• Strong Python skills and an engineering mindset for reliable pipelines and tools.
• Solid foundation in statistical methods and experimental design, sampling, uncertainty, and measurement bias.
• Strong analytical judgement and ability to translate user needs into actionable evaluations.

Responsibilities

• Own the quality measurement and improvement cycle end-to-end: identify user needs, define success, build evaluations, diagnose failures, and work with engineers and product managers to improve outcomes.
• Design, build, and evolve evaluation systems, including datasets, benchmarks, metrics, and automated pipelines that support development and launch decisions.
• Evaluate response quality across multiple dimensions, including relevance, completeness, freshness, factual accuracy, and usefulness to downstream agents.
• Investigate quality regressions and metric anomalies, uncover root causes, and identify the improvements with the greatest potential impact.
• Help prioritize the product and engineering roadmap, making quality trade-offs explicit and assessing whether changes deliver meaningful improvements for users.

Skills

Python programming

Data analysis

Statistical methods

Experiment design

Problem solving

Communication

Tools

Pandas

SQL

Job description

Nebius is seeking an experienced Data Analyst to own end-to-end quality evaluation for AI search systems. You will build datasets, benchmarks, metrics, and pipelines, and work with engineers and product managers to improve outcomes.

Strong Python and statistics background, plus experience turning data into reliable quality signals, is essential. Join Nebius to shape evaluation strategies in a fast-paced, international environment and contribute to cutting-edge AI projects with real impact across