AI Product Analyst (AI Metrics & Model Evaluation)

The Portfolio Group; Greater London; On-site; GBP 80,000 - 85,000; Full time

Job summary

The Portfolio Group in London is seeking an AI Product Analyst to own AI metrics and model evaluation for cutting-edge, client-facing AI products used across highly regulated industries. You will report to the AI leadership team and collaborate with Product, Engineering and in-house experts to shape next‑stage AI development.

You will define metrics, own observability, build dashboards, and translate data findings into clear recommendations that influence release decisions and product direction.

Qualifications

• Solid experience as a product, data or commercial analyst, owning measurement end to end.
• Strong hands-on SQL and Python to build analysis and dashboards.
• Judgement to choose the right method for questions and lines of enquiry.
• Comfort with AI outputs; understand AI metrics vs deterministic products.
• Track record of extracting knowledge from experts into reusable structures.
• Ability to present recommendations to senior leaders and defend them.

Responsibilities

• Define the AI metrics that matter, including retrieval quality and user actions.
• Own production observability from regeneration rates to task abandonment and drift.
• Build analysis and dashboards; specify data needs when the product changes.
• Analyze data to identify patterns, outliers, and commercial impact.
• Choose appropriate methods: segmentation, cohort analysis, trends, etc.
• Own the model evaluation programme and measure coverage and conclusions.
• Own golden datasets for model evaluation, update them, and elicit expert judgement.
• Turn findings into actionable recommendations that influence releases.
• Flag when model evaluation coverage isn't strong enough for a release.

Skills

SQL

Python

Data analysis

Stakeholder communication

Job description

AI Product Analyst (AI Metrics & Model Evaluation) | London | £80,000-£85,000

Join an award-winning, internationally recognised consultancy that's growing fast and building client-facing, cutting-edge AI products used across highly regulated industries, where real users act on high-level advice relating to legislative, regulatory and legal frameworks.

As AI Product Analyst you'll play a key role in providing high-level insight into how much the AI products are being used, how well the AI models and agents are working, and reporting on the key analytics - segmenting the data to highlight trends by client, sector and question type. You will own the AI metrics and model evaluation capability that shapes the next stage of generative and agentic AI development.

The role splits into two dynamic elements: AI metrics tell you how the product is performing in the hands of actual users, and model evaluation tells you whether a change is good enough to release. You'll report into the AI leadership team and work day to day with Product, Engineering and the in-house experts who know what a correct answer looks like. The scope will include;

• Defining the AI metrics that matter, retrieval quality, correctness of output, and what users go on to do with what they're given
• Owning production quality observability, from thumbs-down and regeneration rates through to abandoned tasks and drift in how the product is being used
• Building the analysis and dashboards the team works from, and specifying what needs capturing when the product itself has to change to get the AI metrics you need
• Digging into the data to work out what's really going on, the patterns, the outliers and the likely causes, then deciding which of it matters commercially
• Choosing the right method for the question, whether that's segmentation, cohort comparison, trend analysis or something built for the purpose, rather than reporting the same six metrics every month
• Owning the model evaluation programme: what gets tested, how much it covers, and what the results actually tell us
• Owning the golden datasets that underpin model evaluation, building them, extending them, versioning them and keeping them honest and running elicitation sessions with subject-matter experts to capture the professional judgement behind a right answer
• Turning findings into recommendations and following them through, so the insight changes something rather than sitting in a report
• Saying so clearly when model evaluation coverage isn't strong enough to call a release

Essential Experience

• Solid experience as a product, data or commercial analyst, having owned measurement for a product or service end to end
• Strong hands-on SQL and Python, to the point where you build your own analysis and your own dashboards rather than queuing for someone else to do it
• The judgement to choose the right method for a question, and to know which lines of enquiry are worth your time
• Comfort working with the output of AI or machine-learning systems, and a clear view of why AI metrics and model evaluation differ from measuring a deterministic product
• A track record of getting knowledge out of experts and turning it into something structured and reusable
• The ability to put a recommendation in front of senior people and hold your position, particularly when they don't want to hear it

Desirable experience

• Hands-on model evaluation with LLMs: golden datasets, scoring and regression testing.
• Retrieval-augmented systems and the AI metrics used to assess them
• Experiment design and A/B testing
• A regulated professional services environment

This is a defining appointment for a fast-growing AI function, and a rare opportunity to build an AI metrics and model evaluation capability from first principles rather than inherit one. You will set the standards against which product quality is judged and release decisions are made, so your work determines what ships, not what gets reported afterwards. The role carries genuine autonomy, direct exposure to senior leadership, and the backing of an executive team with a clear AI roadmap.

INDAMS