Customer success, rebuilt as an agentic operation
Extrovert · AI LinkedIn engagement SaaS for B2B sales teams · May to September 2026
🧭 What this was
Extrovert is an AI LinkedIn engagement product for B2B sales teams, with a founder and four engineers. I ran support, billing, the help center and founder-facing analytics as Head of Customer Success — the whole function, one person.
Rather than work the queue by hand, I rebuilt the function as an agentic operation on Claude Code: a router that decides what kind of question it is, an investigation skill wired directly into production, and a loop that turns every wrong answer into a rule so the same mistake does not come back.
The point was never automation for its own sake. It was that a support answer should be traced to the code path that produced it and confirmed against production, not assembled from a plausible story — and that one person should be able to hold that standard across a whole queue.
🔌 The workstation
A single setup document rebuilds the entire environment on a fresh machine in under an hour: data connections, skills, operating rules, and a smoke test per integration.
| Source | Access | Used for |
|---|---|---|
| Production Postgres, 6 schemas | Read-only | Account state, campaigns, jobs, billing, assistant threads |
| Grafana Loki | LogQL | Provider responses, request traces, job timing |
| Intercom | Read + article editing | Conversations, contacts, help-center articles |
| YouTrack | Read + write | Filing and tracking engineering tickets |
| Attio | Read | Churn list, subscription state, partner flags |
| Six product repos | Local clones | Tracing the exact code path behind a symptom |
One rule runs through all of it: the database is read-only and there is no send tool on the inbox. The agent produces SQL and drafts; I run them and I send them. Nothing reaches a customer or the database without a human in between.
🧩 How a ticket gets routed
Fourteen skills in three tiers. The router's first decision is whether the answer depends on this specific account's state. If it does not, the reply is grounded in the help center. If it does, the investigation skill traces the symptom to the code that produced it and confirms it in production before anyone gets a verdict.
| Tier | Skills | What they do |
|---|---|---|
| Triage | cs-user-help, ops-extrovert-debug, tool-youtrack | Read the ticket, route it, investigate across logs, database and source, file the ticket, draft the reply, log the case |
| Knowledge base | kb-read, kb-plan, kb-write, kb-formatting, kb-release-audit, kb-overhaul, kb-ai-agent-enrichment | Pull the authoritative article before answering; plan, draft and format new ones in house style; audit a sprint release into a work list; sweep every article against the current product |
| Analytics | ai-assistant-conversation-analysis, onboarding-funnel-analysis, churn-recovery-analysis, ops-trial-abuse | Answer founder questions from raw production data rather than dashboards |
♻️ The part that compounds
Every case gets written down. Every wrong answer gets dissected, and the lesson lands in the skill or in persistent memory the same day — wired to the exact place it will fire next time, not filed in a document nobody reopens.
| Artifact | Volume | Purpose |
|---|---|---|
| cs-log.md | 190 cases | One line per case: user, symptom, evidence, verdict, ticket |
| cs-error-log.md | 48 post-mortems | Wrong answer, reality, why it was missed, the guardrail that generalises |
| Skill rule sections | Dozens of dated entries | The guardrail wired where it fires next time |
| Memory | 37 files, 13 of them feedback rules | Facts and corrections that outlive a session |
| kb-queue.md | 131 closed items | Stale articles and gaps, consumed by the KB skills |
The triage skill grew from 30 lines to 141 over four months, and every line added cites the case that earned it. The hardest misses produced the rules I would keep anywhere: no verdict without a pinned mechanism, an empty result is provisional until the fetch path is proven, resolve identity by profile rather than by email, and read the provider's raw response before naming a cause.
🔍 Support by the numbers
190 investigations in 14 weeks, each ending in a verdict backed by code references and production evidence. The baseline was two independent sources per verdict, with logs pulled when the question turned on a specific moment the database does not record.
| Measure | Count | Note |
|---|---|---|
| Cases logged | 190 | Across 14 weeks |
| Working-as-designed verdicts | 33 | Each traced to the exact code path and confirmed in the database |
| Confirmed bugs | 15 | Reproduction, mechanism and code reference handed to engineering |
| Manual SQL fixes | 12 | Read-back query, transaction, verify query — run by a human |
| Tickets filed as reporter | 22 | Most resolved |
| Distinct tickets worked | 83 | Filed, commented on, or referenced in case logs |
Typical case shapes: a number the customer sees that does not match the database, a campaign that stopped, an invite that never went out, a post the customer saw on LinkedIn but not in the feed, a billing portal that rejects a paying customer.
Bugs found this way included an import breaking on a session parameter in the URL, a deadlock when deleting a topic during bulk reject, duplicate activities from two URL forms, posts silently not reaching a campaign, a 60-second access token breaking long operations, and a text sanitiser stripping Spanish punctuation.
📚 A help center with two audiences
Three AI agents answer on the company's behalf, and all three read the same help center:
- ·Intercom Fin — replies to customers in the support inbox, often before a human sees the conversation at all.
- ·The public MCP agent — a public endpoint that lets AI clients outside the product query it and answer from those same articles.
- ·The in-app assistant — walks users through setup inside the product itself.
When an agent walks a user through a setup, the article is effectively its prompt. So every article was rewritten to work as executable instructions and still read as plain help for a human.
| Measure | Count |
|---|---|
| Articles in the help center | 96 |
| Authored by me | 69 |
| New since May 2026 | 23 |
| Rewritten August to September | ~45 |
| Release audits | 5 sprints |
| Queue items closed | 131 |
House rules the skills enforce: no internal mechanics or provider names in customer copy, UI labels verified against the deployed frontend rather than the dev branch, and nothing goes live without a human seeing the diff — an edit to a published article is live the moment it is saved.
📊 Founder-facing analytics
Six standalone analyses, each answering a question the founder asked, each computed from raw production data rather than from a dashboard.
| Analysis | What it did |
|---|---|
| Churn recovery | Classified every account on the churn list by driver with explicit confidence levels, and separated the genuinely churned from the mislabelled. Surfaced accounts still paying with auto-renew off, two systemic bugs holding users, and real customers wrongly excluded as partners. Runs per-account dossiers in subagents so the main context stays clean, and ships with an eval set. |
| Assistant benchmark | A 25-question benchmark run against both the in-app assistant and Intercom Fin, scored question by question. Proved the assistant actually reads the help center, and separated its own failures from gaps in the knowledge base that both agents shared. |
| In-app assistant conversation audits | Every external thread read in full. Found two infrastructure failures the product's own error flag never caught, plus a whole class of onboarding failure. Became a standing weekly health check. |
| Onboarding funnel | Traced new owners from signup through content, campaign, prospects, start and first output — from durable database state rather than event tracking. |
| Voice in the in-app assistant | A five-document series on how the in-app assistant fills a user's writing voice: how it is used today, how the instructions should be designed, what to fix, and an implementation plan. |
| Customer reporting | Weekly exports for an agency customer, plus one-off feed and team-usage exports for individual accounts. |
💡 What actually made it work
- ·A human gate on every write. Read-only database access and no send tool is not a limitation, it is the thing that makes the rest safe to run fast.
- ·Verdicts need a mechanism. “Probably a sync issue” is not an answer. Two independent sources and a named code path, or it does not ship to the customer.
- ·Write the mistake down where it will fire. A post-mortem in a document is a diary entry. The same post-mortem as a rule inside the skill is a guardrail.
- ·Routing beats one big prompt. Most of the quality came from the first decision — does this answer depend on account state — not from the depth of any single step.
- ·Documentation is now an interface. Once agents answer on your behalf, help-center articles are prompts. They have to be executable, not just readable.
