Product Owner (AI & Prompt Strategy)

Kaleidoscope Innovation; Fort Worth (TX); Hybrid; USD 100,000 - 130,000; Full time

Job summary

Kaleidoscope Innovation is seeking a Product Owner (AI & Prompt Strategy) in Fort Worth, TX. You will lead the strategy for conversational AI initiatives, working across teams to ensure effective prompt and persona management. The ideal candidate has 5–7 years of experience in product management, particularly with AI/ML products.

Your responsibilities include designing multi-agent architectures, defining metrics for effectiveness, and ensuring stakeholder alignment. This role is hybrid and offers a dynamic work environment focused on innovation and collaboration.

Qualifications

• 5–7 years of experience in Product Management or Ownership, ideally in mortgage or fintech.
• Experience with AI/ML products and conversational AI.
• Strong analytical background and ability to build effectiveness dashboards.

Responsibilities

• Lead product vision and strategy for conversational AI agents.
• Architect multi-agent systems and oversee prompt development.
• Define metrics for tracking prompt effectiveness and agent performance.

Skills

Product Management

AI/ML Products

Conversational AI

Stakeholder Management

Analytical Skills

Tools

AWS

A/B Testing

Job description

Location

Dallas, TX (hybrid)

Job Description

We are seeking a Product Owner (AI & Prompt Strategy) to serve as a prompt-development focused leader for conversational AI initiatives. Operating within a Federated Hub-and-Spoke model, you will sit at the orchestration level across work teams to own the company-wide prompt and persona strategy. Partnering closely with the core engineering team and leading a team of Product Analysts, you will drive the parallel development and deployment of voice AI agents across CDL/Sales, Servicing, and Fulfillment.

Key Responsibilities

• Strategy & Orchestration: Lead the overall product vision for conversational AI agents, owning the company-wide persona, brand voice, and overarching prompt strategy.
• Agent & Sub-Agent Architecture Design: Architect the multi-agent orchestration layer, defining how primary routing agents interpret user intent and seamlessly hand off tasks to specialized, domain‑specific sub‑agents (e.g., specific agents for Sales vs. Servicing vs. Fulfillment). Establish the logic boundaries, tool access, and context parameters for each sub‑agent.
• Iterative Prompt Refinement: Continuously develop, test, and tune complex system prompts, system instructions, and few‑shot examples. Optimize these prompts across the AWS tech stack to balance conversational quality, token efficiency, and response latency.
• Prompt Effectiveness Tracking & Analytics: Establish robust tracking frameworks to measure prompt and agent efficacy at a granular level. Monitor specialized metrics such as hallucination rates, contextual accuracy, task completion/containment rates, and user drop‑off to drive continuous prompt optimization cycles.
• Stakeholder Alignment: Serve as the central orchestrator across CDL/Sales, Servicing, and Fulfillment teams to negotiate and refine business requirements, ensuring parallel development without technical bottlenecks.
• Testing & Quality Assurance: Define the "Golden Dataset" of test scenarios to feed into engineering’s automated LLM‑as‑a‑judge evaluation pipelines. Oversee structured A/B testing initiatives and data analysis to iteratively optimize agent responses against baseline configurations.
• Performance & ROI: Define product strategy and track ROI by aligning AI agent performance to key business KPIs, including Customer Satisfaction (CSAT) and Average Handle Time (AHT).
• Cross‑Functional Collaboration: Partner with the engineering team to integrate prompt designs with the core infrastructure, latency management pipelines, and real‑time safety guardrails.

What You’ll Bring

• 5–7 years of Product Management or Product Ownership experience, ideally within mortgage, fintech, or complex customer servicing environments.
• Demonstrated experience with AI/ML products, specifically conversational AI, Large Language Models (LLMs), and deep prompt engineering strategies within an enterprise AWS environment.
• Proven ability to design multi‑agent or complex state‑machine conversational architectures.
• Strong analytical background with a track record of building effectiveness dashboards, running A/B tests, and utilizing AI‑specific metrics (e.g., hallucination tracking, prompt latency) to drive improvements.
• Ability to translate complex business logic into precise technical configurations and natural language flows.
• Exceptional leadership and stakeholder management skills.

Applicants must be authorized to work for ANY employer in the U.S. We are unable to sponsor or take over sponsorship of an employment Visa at this time.

Kaleidoscope, an Infosys Company, is an equal opportunity employer, and all qualified applicants will receive consideration without regard to race, color, religion, sex, sexual orientation, gender identity, national origin, protected veteran status, spouse of protected veteran, or disability.