The starting point
Complaints about payment status and disputes have been rising for two quarters. The feedback comes from six separate channels: interviews, support tickets, sales and account notes, ops escalations, Jira requests, and compliance and risk feedback. The consequences:
- The product team reads and consolidates feedback by hand, one item at a time.
- Backlog items cannot be traced back to verified customer needs.
- Prioritisation runs on anecdote instead of the evidence that exists.
- Engineers get tickets with missing context and have to rework them.
Leadership asked for a controlled pilot: from gut feeling to evidence-based planning, without removing product managers' decision authority and without compromising security and compliance.
The proposal
A 12-week pilot in the payments domain as a Hybrid Agentic System. The AI only does cognitive tasks. Anything with consequences runs deterministically and under human control. BigQuery on the existing AI platform is the single source of truth, and an Appsmith interface replaces the spreadsheets.
The core idea: the same complaint from an interview, a support ticket and a CRM note becomes one prioritised opportunity with three pieces of evidence, instead of three separate backlog entries.
How it works
- Clean the data: the Google service Cloud DLP removes personal data and financial identifiers before any text touches the language model. This is not AI, it is fixed code.
- Split: the AI breaks an interview or ticket into individual problems, each with a summary, pain points, root cause and tags such as severity and product area. If the material is thin, it asks for clarification instead of guessing.
- Match: a vector search finds similar, already known themes and proposes whether it is the same problem or a new one.
- Gate 1: a product manager confirms the split, the tags and the match.
- Score: a transparent formula with tunable weights calculates the score from five inputs: customer value (Enterprise or SMB), urgency (worst reported severity and frequency), strategic alignment, operational risk and compliance sensitivity. The AI only explains the number, it does not calculate it.
- Gate 2: a product manager decides which opportunities are worth drafting now.
- Draft: the AI writes problem statement, user story, acceptance criteria and a rationale with evidence.
- Gate 3: the draft is checked against the original evidence and the complexity is estimated. Only then does a script, the only component holding the Jira credentials, send the ticket to Jira.
In addition, an analytics agent using the Model Context Protocol answers questions about the stored data in plain language, with no SQL and no CSV export, read-only. It shows product, process and AI metrics. The system also detects regressions, meaning when feedback keeps arriving about a topic that was already fixed in a release.
Four decisions stay with people
- Gate 1: split and matching. It is the only gate that can later become a spot check, because it has a verifiably correct answer and a sustained high agreement rate would prove it.
- Gate 2: which opportunities get built now. This depends on capacity and timing, so there is no rate it could ever converge to. It stays permanently.
- Gate 3: approval of the Jira draft. It also stays permanently.
- Decision after week 12: proceed, iterate or stop, based on measurement against the baseline.
Safe by design, not by instruction
- No Jira access for the AI: no output reaches engineering unreviewed. This is a structural guarantee, not a rule.
- No path to raw data: the AI's account cannot read the raw data. The residual risk is free-text residue that DLP misses. Pattern checks before ingestion and ongoing sampling address it.
- Traceable: every decision is a stored row, and BigQuery logs data access by default. The AI's proposal is stored next to the final value, so deviations and drift can be measured.
- Works council: the role is logged (for example Payment_PM), not the person, so there is no performance monitoring (§ 87 BetrVG).
- Legal: processing in the EU with a defined retention period, classified under the EU AI Act as an internal helper tool outside the high-risk tier.
- Tunable instead of a black box: because the weights are parameters, a systematic disagreement at review shows which weight is wrong and who can change it.
Five stakeholders, five conditions
The stakeholder map ranks everyone by influence and interest, and the system is built so each condition is met technically:
- Head of Product (sponsor): wants fast, evidence-based prioritisation for roadmap reviews. Reports come from a dashboard or a question in plain language.
- Risk & Compliance: can stop the pilot and requires data protection, traceability and human review.
- Engineering Lead: sceptical, wants incomplete or invented tickets kept out of the backlog. That is what Gate 3 and the AI's lack of Jira access are for.
- Senior Product Manager: an early adopter who wants the AI as an assistant, not a replacement.
- Support Operations: wants to be sure escalations do not get buried under feature requests. Operational risk is a separate factor in the formula.
The RACI matrix sets who is responsible, accountable, consulted and informed for each activity.
Risks and countermeasures
The entire risk register is available as an interactive Risk Hub. The five biggest risks:
- Data privacy leak (severity 20): incomplete redaction. Countermeasure: test the DLP rules against real sample data and have compliance approve them before live data flows.
- Rubber-stamping (16): reviewers who only confirm out of fatigue. Countermeasure: mandatory rationale at Gate 3 and a random 20 % sample with a manual complexity check.
- Works council (15): review logs used as performance monitoring. Countermeasure: role-only logging, approved by the works council.
- Delayed approvals (15): gates taking longer than 48 hours. Countermeasure: automatic alerts and an agreed escalation.
- AI interface outage (15): countermeasure: a manual mode that bypasses the AI.
Four decisions are needed before the start: the DLP rules, the GDPR erasure policy, the agreement with the works council and the 48-hour rule.
Timeline and success
- Weeks 1–4 (foundation): sign-off of the DLP template, database schema, baselines, stakeholder survey, RACI agreement, Appsmith, processing of the existing payments data.
- Weeks 5–10 (pilot): daily payments feedback runs through all gates.
- Weeks 11–12 (assessment): measurement against baseline, confidence survey, decision to proceed, iterate or stop.
Measurement comes first: the baseline is captured before the first AI step. Five goals are measured: less time on manual analysis, more consistent documentation, traceability from customer insight to Jira ticket, a faster path from feedback to backlog, and more confidence in prioritisation. The targets are at least 85 % agreement on tags, over 80 % hits when matching similar topics, over 85 % acceptance at Gate 3 and over 50 % less time spent on feedback management.
The project portal
The project includes an interactive prototype: the flow, the three gates, the roadmap, the stakeholders, the Risk Hub and the monitoring portal with the analytics. See the prototype (in English)
Project deliverables
Project brief, stakeholder map with RACI matrix, risk analysis, monitoring plan, an 11-table data model in BigQuery and the interactive prototype. The concept with its documentation was the deliverable. We presented it to the stakeholders so they can decide on that basis whether to run the pilot. It was not built. In the THRIVE programme, the concept was one of 20 winning submissions among 150 participants.