Product Manager, Agentic AI

About the role

What you'll do

• Own the roadmap for Kinetic's agent orchestration layer: models and routing, tools and memory, observability, and permissions and guardrails. 
• Run the agent registry. Every agent has exactly one owner, one documented purpose, one autonomy tier, and one place its behavior is logged.
• Define how agents are deployed, versioned, and rolled back, with prompts, tool definitions, and model versions managed like code.
• Track usage, cost, and latency per agent. Cost per orchestration matters more than cost per token, and you'll know both.
• Lead the cleanup of the agents we already have: inventory every script and workflow touching AI, then retire, replatform, or sunset each one.
• Keep business logic, specs, and evaluation data portable so a future framework or model change is an engineering project, not a rewrite.

• Partner with PMs to define the problem or opportunity, and make the call on whether an agent is the right tool or simpler automation will do.
• Translate use cases into agent specs: inputs, tools, autonomy boundary, human escalation path, success metric, and baseline.
• Write, test, and iterate on system prompts, tool descriptions, and structured outputs side by side with agent and data engineers.
• Tie every agent to one business metric before it scales, such as time-to-quote, bind rate, override rate, or hours saved.

• Build golden datasets with our data team for every agent, and run them as a regression suite whenever a prompt, tool, or model changes.
• Own the rollout gates from shadow mode to staged rollout to full production, with each go/no-go decided by evaluation thresholds rather than a calendar date.
• Write and maintain Kinetic's autonomy-tier policy, including which decisions always keep a human checkpoint, such as a decline, a denial, a bind, or a price.
• Produce the artifacts a regulator or acquirer will ask for as a byproduct of good practice: architecture map, evaluation history, audit trail, and before-and-after metrics.

What you've done

• 5+ years in Product Management, Technical Product Management, or a similar role, including 1+ years shipping LLM-powered or agentic products into production, not just demos.
• Hands-on prompt engineering: system prompts, tool definitions, multi-step chains, and structured outputs. 
• Built or ran evaluation for LLM systems, such as golden datasets, LLM-as-judge graders, and regression testing.
• Working knowledge of agent frameworks and platforms such as CrewAI, LangGraph, OpenAI Agents SDK, or AWS Bedrock AgentCore, along with RAG, tool calling, and MCP.
• Demonstrated ability to convert messy business needs into requirements, user stories, and acceptance criteria.
• Experience in startups or high-growth environments. You're comfortable building from scratch.
• Excellent communication skills, with the ability to explain probabilistic systems to business leaders.

• Prior experience at an insurtech, MGA, or carrier, or in another regulated industry such as financial services or healthcare.
• Familiarity with workers' compensation, commercial lines underwriting, or claims workflows.
• Experience with AI governance frameworks.
• Experience with multi-model routing, cost optimization, or human-in-the-loop design for agents.

• Fully distributed, remote team across all continental US time zones
• Medical, dental, and vision insurance
• 20 vacation days per year
• 9 federal holidays off 
• Parental Leave
• Pre-tax 401k contribution
• HSA with compatible health plans
• FREE: Health Advocate
• FREE: Telehealth Membership
• FREE: OneMedical Account