Senior Product Manager Observability Tools

ASAPP; Bengaluru; On-site; INR 400,000 - 700,000; Full time

Benefits offered by this job

Stock options

Insurance – ICICI Lombard

Lunch/dinner stipend

Connectivity stipend

Wellness perks

Mac equipment

Learning stipend

Parental leave

Job summary

ASAPP is seeking a Senior Product Manager to own tools for simulating and previewing AI agent behavior before production, and to lead review/audit tooling for complex conversations. You will collaborate with Research, Engineering, Sales, and Customer Success to drive quality, trust, and measurable outcomes in a fast-growing AI-driven platform.

The role emphasizes ownership, cross-functional collaboration, and delivering high-velocity, customer-focused improvements in a scalable way, with a

Qualifications

• 5+ years of product management experience in enterprise B2B SaaS or AI-enabled products.
• Direct experience with LLMs, AI agents, or applied AI systems and evaluating model output.
• Experience shipping tools to simulate/test/preview system behavior before production.
• Experience building review or audit tooling for unstructured content.
• Comfort partnering with research/engineering on system internals to ground product decisions.

Responsibilities

• Own scenario creation via the simulation agent to generate realistic test conversations.
• Own eval surfaces, including mistake monitoring and structured data evals.
• Own the Previewer to show configurations in simulated conversations.
• Own Conversation Explorer for reviewers to read full production conversations.
• Build the case for simulation-driven quality with cross-functional partners.
• Partner with Research and Engineering on agent architecture and tooling.
• Manage the trust layer of the review experience to ensure confidence.

Skills

Product management

AI/LLMs

Quality testing

Analytics

Stakeholder communication

Job description

Job Description:

At ASAPP, our mission is simple: deliver the best AI-powered customer experience—faster than anyone else. To achieve that, we’re guided by principles that shape how we think, build, and execute. We value customer obsession, purposeful speed, ownership, and a relentless focus on outcomes. We work in tight, skilled teams, prioritize clarity over complexity, and continuously evolve through curiosity, data, and craftsmanship. We’re seeking technologists and problem solvers who thrive in fast-paced environments, love collaborating with great talent, and approach every day like it’s Day 1.

Were a globally diverse team with hubs in New York City, Mountain View, Latin America, and India. If youre driven by continuous learning, rapid pivots, and the challenges of building in a high-growth startup, we’d love to talk. This is more than a job—it’s a journey.

This is not a reporting role. You will own the tools that let builders and reviewers see, test, and trust how GenerativeAgent (GA) actually behaves — before it ever talks to a customer, and after. If builders cant confidently preview a configuration change, and reviewers cant efficiently audit what GA said, thought, and did in production, quality erodes silently and trust with our customers erodes with it. Making agent behavior visible, testable, and reviewable at scale is your product.

What You Will Do

• Own scenario creation via the simulation agent. Lead product for the tools that let builders generate and curate realistic test scenarios, so configuration changes can be validated against representative conversations before they ship.

• Own our evals surfaces, including mistake monitoring and structured data evals. Define how we detect, surface, and categorize GA mistakes at scale, and how structured data extraction is evaluated for accuracy — turning noisy conversational output into clear, actionable quality signals for builders and customers alike.

• Own the Previewer. Own the tool builders rely on to see exactly how their configurations show up in real simulated conversations, closing the loop between making a change and understanding its downstream impact.

• Own Conversation Explorer. Lead the primary surface reviewers use to read through full production conversations — including agent thoughts and actions — balancing depth for investigative review with speed for high-volume auditing.

• Build the case for simulation-driven quality. Partner with Sales, Customer Success, and Research to show customers and internal stakeholders how pre-production testing and post-production review compound into fewer mistakes, faster iteration, and more defensible GA performance.

• Partner deeply with Research and Engineering on the underlying agent architecture — reasoning traces, tool calls, scenario generation methods — so product decisions are grounded in whats technically sound and what actually predicts real-world behavior.

• Manage the trust layer of the review experience. Proactively identify and resolve issues that erode confidence in what builders and reviewers see — desyncs between simulated and real behavior, incomplete traces, unclear mistake categorization — and design the UX and guardrails that keep them confident in what theyre looking at.

Who You Are

We evaluate candidates against a set of principles. These arent interview talking points — they describe how PMs are expected to operate here every day.

• Growth & Abundance Mindset: You approach new domains with curiosity and learn fast. You see collaboration as additive, not competitive. When a bet doesnt work, you extract the signal and move on.

• Extreme Ownership: You take responsibility to deliver. You dont wait to be asked — you identify the gap and close it. You are the PM; the outcome is yours.

• High-Velocity Outcomes: You measure yourself by outcomes, not output. You distinguish between shipping a feature and moving a metric. You can answer "what changed for the customer?" for everything on your roadmap.

• Effective Communication, Collaboration, and Trust: You influence without authority across engineering, research, design, sales, and customer success. You overcommunicate to keep stakeholders aligned.

• Real-Time Feedback: You give and receive feedback continuously rather than saving it for quarterly reviews. When something is wrong, you say so directly and constructively.

What You Will Need

• 5+ years of product management experience, with meaningful time spent on testing, quality, or review/analytics products in enterprise B2B SaaS.

• Direct experience building products around LLMs, AI agents, or other applied AI systems — you can reason about agentic behavior, prompt and scenario design, or model output evaluation well enough to make sharp product tradeoffs, not just talk about AI at a high level.

• Experience shipping tools that simulate, test, or preview system behavior before it reaches production, including the judgment to know when simulated behavior is a reliable proxy for the real thing and when it isnt.

• Experience building review or audit tooling for complex, unstructured content, and turning ambiguous quality signals into clear, defensible categorizations.

• Comfort partnering with research and engineering teams on system internals — reasoning traces, tool-use logs, evaluation methodology — to ground product decisions in technical reality.

• Track record of running discovery with internal or external users and translating it into structured requirements — evals, quality metrics, or review workflows.

• Excellent written and verbal communication — you can write a clear PRD, a release announcement, and comfortable presenting directly to customers.

What We Would Like to See

• Experience in contact centers, conversational AI, or customer experience (CX) analytic - a plus.

• Experience with evaluation frameworks or methodologies for generative AI systems (offline evals, human-in-the-loop review, automated mistake detection).

• Prior experience in a startup or fast-scaling environment where the roles scope evolved as the company did

Benefits

• Competitive compensation
• Stock options
• ICICI Lombard General Insurance LTD
• Onsite lunch & dinner stipend
• Connectivity (mobile phone & internet) stipend
• Wellness perks
• Mac equipment
• Learning & development stipend
• Parental leave, including 6 weeks paternity leave

ASAPP is committed to creating a diverse environment and is proud to be an equal opportunity employer. All qualified applicants will receive consideration for employment without regard to race, color, religion, gender, gender identity or expression, sexual orientation, national origin, disability, age, or veteran status. If you have a disability and need assistance with our employment application process, please email us [e-mail] obtain assistance.