AI Metrics & Model Evaluation Lead

The Portfolio Group; Greater London; On-site; GBP 80,000 - 85,000; Full time

Job summary

The Portfolio Group in London is seeking an AI Product Analyst to own AI metrics and model evaluation for cutting-edge, client-facing AI products used across highly regulated industries. You will report to the AI leadership team and collaborate with Product, Engineering and in-house experts to shape next‑stage AI development.

You will define metrics, own observability, build dashboards, and translate data findings into clear recommendations that influence release decisions and product direction.

Qualifications

• Solid experience as a product, data or commercial analyst, owning measurement end to end.
• Strong hands-on SQL and Python to build analysis and dashboards.
• Judgement to choose the right method for questions and lines of enquiry.
• Comfort with AI outputs; understand AI metrics vs deterministic products.
• Track record of extracting knowledge from experts into reusable structures.
• Ability to present recommendations to senior leaders and defend them.

Responsibilities

• Define the AI metrics that matter, including retrieval quality and user actions.
• Own production observability from regeneration rates to task abandonment and drift.
• Build analysis and dashboards; specify data needs when the product changes.
• Analyze data to identify patterns, outliers, and commercial impact.
• Choose appropriate methods: segmentation, cohort analysis, trends, etc.
• Own the model evaluation programme and measure coverage and conclusions.
• Own golden datasets for model evaluation, update them, and elicit expert judgement.
• Turn findings into actionable recommendations that influence releases.
• Flag when model evaluation coverage isn't strong enough for a release.

Skills

SQL

Python

Data analysis

Stakeholder communication

Job description

The Portfolio Group in London is seeking an AI Product Analyst to own AI metrics and model evaluation for cutting-edge, client-facing AI products used across highly regulated industries. You will report to the AI leadership team and collaborate with Product, Engineering and in-house experts to shape next‑stage AI development.

You will define metrics, own observability, build dashboards, and translate data findings into clear recommendations that influence release decisions and product direction.