Senior Product Manager, Observability Tools

Own scenario creation via the simulation agent. Lead product for the tools that let builders generate and curate realistic test scenarios, so configuration changes can be validated against representative conversations before they ship.

Own our evals surfaces, including mistake monitoring and structured data evals. Define how we detect, surface, and categorize GA mistakes at scale, and how structured data extraction is evaluated for accuracy — turning noisy conversational output into clear, actionable quality signals for builders and customers alike.

Own the Previewer. Own the tool builders rely on to see exactly how their configurations show up in real simulated conversations, closing the loop between making a change and understanding its downstream impact.

Own Conversation Explorer. Lead the primary surface reviewers use to read through full production conversations — including agent thoughts and actions — balancing depth for investigative review with speed for high-volume auditing.

Build the case for simulation-driven quality. Partner with Sales, Customer Success, and Research to show customers and internal stakeholders how pre-production testing and post-production review compound into fewer mistakes, faster iteration, and more defensible GA performance.

Partner deeply with Research and Engineering on the underlying agent architecture — reasoning traces, tool calls, scenario generation methods — so product decisions are grounded in what's technically sound and what actually predicts real-world behavior.

Manage the trust layer of the review experience. Proactively identify and resolve issues that erode confidence in what builders and reviewers see — desyncs between simulated and real behavior, incomplete traces, unclear mistake categorization — and design the UX and guardrails that keep them confident in what they're looking at.

5+ years of product management experience, with meaningful time spent on testing, quality, or review/analytics products in enterprise B2B SaaS.

Direct experience building products around LLMs, AI agents, or other applied AI systems — you can reason about agentic behavior, prompt and scenario design, or model output evaluation well enough to make sharp product tradeoffs, not just talk about AI at a high level.

Experience shipping tools that simulate, test, or preview system behavior before it reaches production, including the judgment to know when simulated behavior is a reliable proxy for the real thing and when it isn't.

Experience building review or audit tooling for complex, unstructured content, and turning ambiguous quality signals into clear, defensible categorizations.

Comfort partnering with research and engineering teams on system internals — reasoning traces, tool-use logs, evaluation methodology — to ground product decisions in technical reality.

Track record of running discovery with internal or external users and translating it into structured requirements — evals, quality metrics, or review workflows.

Excellent written and verbal communication — you can write a clear PRD, a release announcement, and comfortable presenting directly to customers.