Sr. Product Manager, AI

Pantomath Inc; San Francisco (CA); On-site; USD 124,000 - 182,000; Full time

Job summary

Pantomath Inc in San Francisco is hiring a Senior Product Manager for AI with hands-on experience building agentic systems. You will own the strategy and execution of our AI product surface, determining how agents see data, what actions they can take, and how outputs are evaluated.

The role spans context, resolution agents, and MCP-based distribution across customer surfaces. You will work closely with engineering to deliver reliable agents, define golden datasets and evaluation rubrics, and set

Qualifications

• Hands-on experience building with LLMs in a technical role.
• 3+ years of product management experience on complex AI/data platforms.
• Strong knowledge of foundation models, agent patterns, and evaluation for non-deterministic systems.

Responsibilities

• Own the AI product strategy and roadmap for agentic features.
• Evaluate feasibility, failure modes, and maintainability of agentic capabilities.
• Define agent boundary: autonomous actions vs. human confirmation and evolving trust.
• Translate customer needs and model capabilities into clear PRDs for engineers.
• Partner with engineering to design, build, and ship reliable agents with accuracy, latency, cost, and security.
• Build evaluation harnesses with golden data, rubrics, and regression testing.
• Replicate real-world data environments to validate agent behavior for production readiness.
• Define success metrics (accuracy, acceptance rate, time to root cause, token cost).
• Support GTM with technical enablement and customer-facing docs.

Skills

Hands-on LLMs

Product management

Agent architectures

Evaluation methodologies

Cross-functional collaboration

Education

Bachelor's degree in Computer Science or related field

Tools

MCP framework

API integration

Job description

Job Summary

Pantomath is hiring a Senior Product Manager for AI with direct, hands-on experience building agentic systems to own the strategy and execution of our AI product surface.

This role sits squarely at the intersection of product leadership and applied AI engineering. You will be responsible for defining how Pantomath's agents resolve data incidents: what context they can see, what actions they are trusted to take, how their output is evaluated, and where the boundary sits between what the agent does and what the human decides. That work spans our context layer (structural, behavioral, baseline, risk, and semantic context), our resolution and triage agents, and our MCP-based distribution into the surfaces customers already work in (Teams, Slack, ServiceNow, Jira).

The bar for this role is hands-on. You deeply understand frontier model capabilities and limitations, fluency in the parts that decide whether an agent is trustworthy: what goes into its context and what gets left out, tool and API surfaces the model can actually use correctly, evaluation for non-deterministic systems, and the cost/latency/quality frontier and the unique UX considerations of non-deterministic systems. Enough depth to read a failed trace and name the defect - retrieval, tool definition, context assembly, or prompt.

You will use that experience to set technical direction, write precise requirements, and partner closely with engineering to deliver agents that are reliable enough to earn autonomy in production data environments.

About Pantomath

Pantomath is building the autonomous layer for enterprise Data Operations. Our agents catch data incidents, trace them to root cause across the stack and resolve them, pinpointing their root cause, and resolve them with operational context from across your entire stack.

Backed by a $30M Series B led by General Catalyst (alongside Sierra Ventures, Bowery Capital, Epic Ventures, and Hitachi Ventures), Pantomath is pioneering the transition enterprises from passive data monitoring to real-time, autonomous incident resolution.

Across our entire company, Pantomath brings together seasoned startup founders, operators with prior successful exits, and veteran talent from market-defining technology companies including Google, Meta, Stripe, Databricks, Microsoft and Amazon. We combine the high-ownership, high-velocity culture of a scaling venture-backed startup with the deep execution rigor and track record of proven industry leaders.

What You'll Do

• Own the AI product strategy and roadmap, making informed, technical decisions about which workflows an agent should own end to end, which it should assist, and which it should stay out of.

• Leverage your applied AI background to evaluate feasibility, failure modes, and long-term maintainability of agentic features.

• Define and defend the agent boundary: what the agent is trusted to do autonomously, what requires human confirmation, and how that boundary widens as trust is earned.

• Translate customer needs, technical constraints, and model capabilities into clear, detailed PRDs that engineers can execute against without ambiguity.

• Partner closely with engineering to design, build, evaluate, and ship agents that meet strict standards for accuracy, latency, cost, observability, and security.

• Build and own the evaluation harness: define golden datasets, quality rubrics, and regression testing so that model and prompt changes ship on evidence rather than anecdote.

• Replicate real-world customer data environments and incident scenarios to validate agent behavior, and ensure production readiness.

• Define success metrics for AI features (resolution accuracy, acceptance rate, time to root cause, deflection, token cost per outcome) and actively monitor, diagnose, and iterate based on data and feedback.

• Work with Go-to-Market teams to support launch readiness, technical enablement, and customer-facing documentation, including how to explain what the agent knows and why it is right.

• Stay current on trends and best practices across foundation models, agent frameworks, MCP and tool-calling standards, retrieval, and evaluation to inform roadmap decisions.

• Advocate for a best-in-class experience for data engineers and analysts, ensuring AI capabilities are transparent, correctable, and trustworthy at scale.

Required Qualifications (Non-Negotiable)

Hands-on experience building with LLMs in a technical role (e.g., shipping production AI features, building agent workflows, prompt and context engineering, running model evaluations).

3+ years of product management experience working on technically complex products such as AI/ML platforms, data platforms, developer tools, or APIs.

Strong, working knowledge of:

• Foundation model capabilities, limitations, and cost/latency trade-offs

• Agent architecture patterns: tool calling, context assembly, retrieval, orchestration, and guardrails

• Evaluation methodology for non-deterministic systems

• Demonstrated judgment about when AI is the right answer and when it is not.

• Proven ability to partner deeply with engineering teams and make informed trade-offs.

• Ability to clearly communicate complex technical concepts to both technical and non-technical stakeholders.

• Bachelor's degree in Computer Science, Data Science, Information Systems, Engineering, or a related technical field (or equivalent practical experience).

Preferred Qualifications

• Experience with MCP, agent frameworks, or embedding AI into third-party surfaces such as Teams, Slack, ServiceNow, or Jira.

• Prior ownership of an AI or agent product within a B2B SaaS or data product.

• Familiarity with data observability, lineage, governance, or metadata platforms.

• Hands-on background in data engineering or data architecture, or experience building AI on top of enterprise data systems.

• Experience in a high-growth SaaS or developer-focused product environment.

Who This Role Is (and Isn't) For

This role is ideal for a PM who:

• Has actually built with models, not just written strategy decks about them

• Is energized by the hard part of agentic products, which is proving they are right often enough to be trusted

• Enjoys being close to the technical details and using that depth to drive product decisions

• This role is not a fit for PMs whose AI experience is limited to evaluating vendors or shipping a chat interface.

Pay Range for this role is 124,000 - 182,000