Job Description:
You will own the behaviourof a specific AI system or workflow inside a larger product. You will work witha team and a defined set of internal stakeholders. Your primary stakeholdersare business teams who will use the workflow and engineering teams who willbuild it. You will be responsible for creating a spec that the business teamscan approve and engineering teams can build efficiently. Your success ismeasured by whether the AI system behaves as specified in production over time.
You will work with QA andengineering to build and review evaluations. This is a critical part of therole. Evaluations are how you verify the AI system keeps behaving as specifiedonce its live.
Building a workingunderstanding of how AI systems work, and what that means for product andanalytics, is critical to this role. You will also act as a proxy for thebusiness team and run UAT.
Alongside that, the rolecarries the usual weight of product ownership: picking up an unfamiliar domainfast, holding the line on scope, and keeping stakeholders aligned on what isshipping and when.
What you will ownBacklogand delivery
- Own and prioritise the backlog for yoursystem; run refinement, planning, and acceptance with the delivery team.You could also be running kanban boards for your team for a fast pacedproject.
- Turn business requirements into AIsystem requirements for the engineering team.
- The requirements should meet the Jeavioguardrails of AI governance, privacy, security and responsible AIstandards, and highlight it when a requirement falls outside them.
- Support adoption: training, workflowchanges, review Evals, user manuals, release notes etc. that might berequired to get the feature to production.
POCompetency
- Pick up a new domain quickly and get tothe point where you can ask informed questions about the business.
- Should be able to adopt AI tools likeClaude or similar to work upon eliciting and/or prototyping requirements(including UX designs) independently or collaboratively with technicalpeers.
- Should be able to participate in andlead (where appropriate) discovery sessions to be able to understand thelay of the land, vision and translate it back into documentation that canbe used by both the business and engineering teams.
- You should be able to figure out whichare the relevant metrics for users and business.
- AI can be more accurate but slower, moreflexible but less predictable, cheaper but occasionally wrong. You shouldbe able to weigh tradeoffs from a product point of view - i.e. whether acertain trade off is okay or breaks user experience.
- The Evals you define must align with theproduct requirements.
StakeholderManagement
- Steer stakeholders towards the betteroption or push back on scope during scope based negotiations by workingalongside the engineering team.
- Be able to influence the roadmap byunderstanding what the team can deliver vs what the stakeholders want.
- Should also be able to present projectstatus to stakeholders and give an update on delivery timelines byaligning with the technical teams.
- Should be able to understand andcommunicate the limitations of the system, clearly to stakeholders.
Behaviouralspecification
- Write acceptance criteria forprobabilistic outputs.
- Work with Engineering and QA colleaguesto define system guardrails
- Specify the confidence thresholds thattrigger human review, a fallback path, or a graceful exit.
- Read and assess system prompts wellenough to tell whether they match the spec.
- Write the spec for two readers at once:the AI engineer who implements it and the business user who will live withit.
Evaluationand human oversight
- Own the evaluation dataset for yoursystem across its lifecycle.
- Interpret results and translate them fornon technical stakeholders.
Security& Compliance:
- Define and prioritize securityrequirements in product backlog.
- Ensure data protection, privacy, andcompliance with ISO 27001 policies.
- Collaborate with engineering andsecurity teams for secure product delivery.
- Manage risks related to features,integrations, and data handling.
- Support audit readiness and continuoussecurity improvements.
What we are looking for
Required
- 6+ years in product ownership, productmanagement, or business analysis, including at least two years owning abacklog with a delivery team.
- Direct experience shipping or operatingan AI/ML or LLM-based feature in production — not only prototyping.
- Fluency in AI tools - ChatGPT or Claudeincluding an understanding of skills, plugins, and contextmanagement.
- Demonstrable understanding of how LLMswork and how they impact product decisions
- Demonstrated ability to write acceptancecriteria for a non-deterministic output.
- Working fluency in Evals.
- Judgment about where human in the loopuse case fits in.
- Strong written communication.
- Should be able to keep up with evolvingtrends and is open to unlearn and learn quickly.
Goodto have
- Experience with agentic systems: tooluse, multi-step orchestration, retrieval, or observability tooling.
- Familiarity with our stack: AWS, Claude,LangSmith, JIRA.
Requirements: