Senior Director, Technical Product Management — AI Quality & Evaluation

Leadout Capital; San Francisco (CA); On-site; USD 180,000 - 240,000; Full time

Job summary

Leadout Capital’s Salesforce-backed incubation unit seeks a technical product leader with deep AI expertise to turn frontier capabilities into reliable customer products. This hands-on IC leadership role emphasizes product ownership, technical judgment, and cross-functional influence while mentoring designers and program managers.

You'll work with AI researchers, engineers, and customers to define strategy for AI quality, lead evaluation-driven delivery, and shape features from early learning to

Qualifications

• Experience building AI/ML products with measurable customer outcomes.
• Hands-on experience building AI agents and evaluation capabilities.
• Ability to inspect data and traces, prototype with coding agents, and reason about experiments.

Responsibilities

• Set a coherent product strategy and roadmap for AI quality and evaluation.
• Lead customer investigations and experiments from learning to delivery.
• Design credible evaluations with researchers, applied scientists, and engineers.
• Coach product partners, junior designers, and technical program managers.
• Communicate differentiation and investment choices to customers and senior stakeholders.

Skills

AI product leadership

Hands-on AI/ML

Agent prototyping

Customer investigations

Cross-functional leadership

Data/traces inspection

Job description

What You’ll Be Doing & Why It Matters

Help build the next generation of Agentforce. We’re an entrepreneurial incubation team within Salesforce, combining the pace and ownership of a startup with access to enterprise customers and Salesforce’s distribution. We’re looking for a technical product leader with deep AI expertise who wants to turn frontier capabilities into products customers can rely on.

Our team works closely with AI researchers, applied scientists, and engineers, uses coding agents extensively, and builds and experiments outside the core platform to iterate quickly. You’ll shape product strategy for AI quality and evaluation, establish how we measure progress, and personally lead a customer quality workstream and a focused feature from early learning through delivery.

This is a hands‑on individual contributor leadership role, with an IC focus expected through 2027. You’ll lead through product ownership, technical judgment, and cross‑functional influence while coaching colleagues, including junior designers and technical program managers. People‑management responsibilities may develop as the product and team grow. You should enjoy inspecting traces and data, challenging evaluation results, prototyping with coding agents, and working directly with customers. Engineering partners own technical architecture and production implementation.

Core Responsibilities & Scope

• Set a coherent product strategy and roadmap for AI quality and evaluation, connecting customer needs, technical opportunities, and measurable business outcomes.
• Establish shared quality definitions, evaluation requirements, coverage priorities, and release criteria with research, engineering, and product partners.
• Make explicit investment tradeoffs across customer quality improvements, evaluation capabilities, and product features; align teams on priorities and decision ownership.
• Personally lead customer investigations and experiments. For example, investigate why an agent gives an incorrect answer or fails a tool‑driven workflow, define the expected behavior, and partner with researchers and engineers to test a change and validate its customer impact.
• Build an operating approach that connects customer feedback, evaluation evidence, experiments, releases, and learning across teams.
• Partner with researchers, applied scientists, and engineers to design credible evaluations. Apply AI/ML fundamentals such as training/test separation, overfitting, representative coverage, and independent validation to assess non‑deterministic model behavior and grader reliability.
• Lead delivery of capabilities that help customers understand and improve AI performance. For example, prototype an experience that turns a failed agent interaction into an evaluation case, makes the quality gap clear, and helps a customer verify whether a proposed change works.
• Coach product partners, junior designers, and technical program managers in problem framing, experimentation, and delivery; raise the quality of decisions through mentorship and practical example.
• Bring verified competitive insight into strategy and communicate differentiation, limitations, and investment choices clearly to customers and senior stakeholders.

Minimum Requirements

• A record of building AI/ML products and delivering measurable customer outcomes, with increasing ownership, scope, and impact.
• Deep hands‑on experience building AI agents and evaluation capabilities in close partnership with AI researchers, applied scientists, or AI engineers. One to two years of substantive generative AI experience can be sufficient when supported by strong AI/ML fundamentals and demonstrated impact.
• Strong applied AI/ML judgment across model behavior, prompting, retrieval, agent/tool workflows, and the tradeoffs among model, data, system, and product improvements.
• Current hands‑on ability to inspect data and traces, prototype with coding agents, and reason about experiments, overfitting, training/test splits, edge cases, and uncertainty. You can define what good looks like for a non‑deterministic system and challenge conclusions with technical partners.
• Principled, flexible thinking: frame the underlying customer problem, test assumptions, and change direction when evidence warrants it. You can explain when lessons from a previous product apply and when the context demands a different approach.
• Experience establishing shared standards and resolving cross‑team disagreements while preserving clear accountability and delivery momentum.
• Evidence of coaching and mentoring colleagues without relying on reporting authority, and the ability to deliver independently in a fast‑moving environment.
• Clear communication with customers, technical teams, and senior leaders, including an honest account of what is known, uncertain, or still unproven.

Preferred Qualifications

• Evaluation platforms, agent systems, AI observability, retrieval, or enterprise AI products.
• Building in an AI‑native startup or entrepreneurial environment, taking a new product from early customer learning to adoption, and applying lessons from both successful and unsuccessful experiments.
• Hands‑on prototyping with coding agents and delivery of customer‑facing performance or outcome experiences.
• Leading initiatives that require agreement across research, engineering, product, and customer‑facing organizations.

What success looks like

Customers see credible improvement in real agent workflows, supported by evaluations they can trust. You independently take an initial quality problem from failure analysis through experimentation and customer validation, and deliver a focused feature with engineering. Teams make better release and investment decisions because of the evidence, product judgment, and coaching you bring.