A product team watches an AI assistant complete a task in seconds. The demonstration looks convincing. Then the questions start.
Will it work with incomplete information? Can another team integrate it without weeks of support? Will customers understand what happened? And who decides whether it is ready to launch?
Those questions often land on someone with “AI product manager” in their title. Yet they represent different kinds of responsibility. A team can make progress on one and remain blocked on another.
In AI Product Management at Spotify: A View From Three Angles, Henrik (Henok) Lindström, Divita Vohra, and Humberto Corona describe their work across research, a shared recommendation model, and listener experiences. Their account makes an important distinction: the product manager’s customer, decisions, and evidence of success change with the part of the system they own.
For teams applying this idea, the practical question is where each important decision belongs. Here is a way to make those responsibilities explicit, even when one person covers several of them.
Start with the decision you own
Research ownership involves deciding which uncertainties deserve investigation. Model or platform ownership involves making a capability useful to the teams that depend on it. Experience ownership involves helping an end user accomplish something successfully.
These responsibilities connect, but their evidence is different.
| Responsibility | Decision to make | Evidence to bring |
|---|---|---|
| Research | Which approach deserves further investment | A comparison that resolves a meaningful uncertainty |
| Model or platform | Can teams depend on this capability | Task quality, operating limits, and a workable integration |
| Experience | Does the complete journey help the user | Task completion, correction effort, and observed usability |
A useful exercise is to take a current initiative and identify the next decision in each row. If the team can only describe a model milestone, it may be missing the work required to make that milestone useful.
Work backward from a specific customer task
Consider a fictional onboarding assistant for a project management application. Its purpose is to help a new customer create a usable first project from a short description of their work.
“Generate a project plan” is too loose to guide the whole team. A more useful definition is: the customer can review the proposed tasks, correct the assumptions, save the project, and understand what to do next.
That definition creates several distinct questions.
The research question might be whether retrieving examples from a curated library improves plans for unfamiliar industries. The platform question might be whether the assistant can produce valid tasks and preserve customer permissions. The experience question might be whether people recognize the assumptions and can edit them without starting again.
The same product outcome gives each group a reason to do its work. It also makes gaps visible. A technically valid plan can still be unusable if the customer does not understand the proposed sequence.
Make the research question small enough to answer
An exploration needs a decision at its end. Otherwise, a promising prototype can keep attracting investment without changing what anyone knows.
For the onboarding assistant, compare the current approach with an approach that retrieves approved examples. Use the same cases, including incomplete descriptions and requests from industries the team understands less well. Decide beforehand what improvement would justify additional complexity and operating cost.
The result might support retrieval, reveal that better input questions solve the problem, or show that the assistant should serve a narrower audience. Each result can inform a useful product decision.
A small team can perform this work without a dedicated research department. The discipline is to record the uncertainty, the alternatives, the test, and the decision the evidence will inform.
Define the capability other people can rely on
Spotify’s model perspective concerns a model trained for recommendations. Many smaller teams will use an existing model through an API. Their responsibility is still to define the behavior of the system around it, including retrieved information, tools, validation, and failure handling.
For the onboarding assistant, an integration agreement should describe what the caller supplies, what a successful response contains, and what happens when the information is insufficient. It should also specify operating constraints and how changes are tested.
“The model is better” provides little guidance to a team that needs valid task dates, dependable permission checks, and a clear response when a required service is unavailable.
Make those requirements testable. A generated task must reference an existing project. An uncertain date should remain unconfirmed. A failed save must be reported as a failed save. These are concrete behaviors a product team can discuss with engineering and design.
Evaluate the whole journey
An evaluation, often shortened to “eval,” is a structured test of whether the system meets defined expectations. For a product team, the important step is connecting each check to a user consequence.
In the fictional assistant, useful checks include whether the plan preserves the user’s stated constraints, whether edits survive a retry, and whether the project actually exists after the system reports success.
Anthropic’s guidance on agent evaluations distinguishes the record of an agent’s actions from the final state it produces. That distinction is useful here: a convincing confirmation message does not establish that a project was saved.
Use technical checks for facts the system can verify, and observed user sessions for questions about comprehension and effort. If an AI model helps grade subjective outputs, compare its judgments with human reviews before relying on the scores. Keep a separate set of cases for checking changes so the team does not mistake tuning to familiar examples for broader improvement.
Give failures a route back to the right owner
Imagine a user changes a project deadline and the assistant silently reverts it. The symptom appears in the interface, but its cause could sit in stored state, instructions, a tool call, or the way the interface submits the correction.
Record the intended outcome, the observed behavior, and the system version. Then assign a person to investigate the cause. Avoid assuming every poor result requires a more capable model.
The product manager keeps the investigation connected to user impact. The outcome should include a fix, a case that checks for the same failure, and a decision about whether related users need attention.
Write a brief the whole team can use
Before expanding an AI feature, answer these questions together:
- Who is trying to accomplish what task?
- What does the current experience deliver, and where does it fall short?
- Which uncertainty could change our decision to build?
- What must the AI system reliably produce or do?
- How can users inspect and correct the result?
- Which checks determine readiness, and who owns that decision?
- What evidence after release would make us expand, change, or pause it?
Attach examples to the answers. One difficult case with a clear expected outcome can resolve more ambiguity than a page of adjectives about quality.
The value of this approach is visible in the next product conversation. Research can explain what has been learned. Engineering can explain what the capability supports. Design can show how people use and recover from it. Product can connect those findings to a release decision the team understands.
Put this into practice
Start with the AI Evaluation Starter Kit and use it to document the launch checks. For the interface review, read Designing AI Experiences People Can Correct.
Sources
Henrik (Henok) Lindström, Divita Vohra, and Humberto Corona. AI Product Management at Spotify: A View From Three Angles. Spotify Engineering, November 9, 2026.
Anthropic. Demystifying evals for AI agents. January 9, 2026.
The onboarding assistant and decision brief are original illustrative applications of these ideas.