Karissa Van Baulen - The Importance of Using Analytics and Feedback for your Documentation

Documentation and knowledge bases improve fastest when teams treat user analytics and in-page feedback as a single system—using numbers to find where users struggle, then using feedback to learn why. Karissa Van Baulen frames the core problem as a common trap: dashboards built around vanity metrics like page views or even benchmark scores can produce “a number” without actionable insight. The fix is to start with questions about user needs—what users like or dislike, where they drop off, which layouts feel intuitive, and whether content actually prevents support tickets—then design analytics and feedback around those questions.

A key theme is that documentation funnels behave differently from e-commerce. Instead of pushing users toward a purchase, teams aim for “negative conversion”: getting users to the right answer quickly so they don’t contact support. That shift changes how success metrics are interpreted. Van Baulen groups the work into four analytics/feedback categories: user feedback (to capture reasons), drop-off points (to locate where users fail), session visualization (to observe behavior), and self-service score (to measure whether users find answers without tickets).

For user feedback, she distinguishes between surveys and always-on feedback widgets. Surveys typically appear after a user indicates an article wasn’t helpful (e.g., clicking “no”), then ask what’s missing and can route users into follow-up via email. Feedback widgets live on the page and let users rate with emojis or stars and optionally leave comments. The practical payoff is catching small issues that reviews miss—like spacing problems in headings and paragraphs—because incoming feedback acts like an extra layer of editing.

To turn feedback into dashboard-ready metrics, she recommends an “approval rating” calculated as positive responses divided by total responses (positive and negative can come from widget ratings, surveys, or ticket-like “did you like this page?” buttons). Color thresholds help teams prioritize: high approval can be deprioritized, while low approval flags pages needing review. She pairs this with page view prioritization so the most urgent work sits at the intersection of high traffic and low approval.

Drop-off points extend the same logic along the support journey. Van Baulen describes a four-step funnel over a seven-day window: landing on a documentation page, reaching the contact form, filling out a ticket, and submitting it. When drop-off spikes, teams should investigate whether users are failing to get answers—and they should use feedback tools at those moments to learn the cause. She also introduces deflection rate (equated to drop-off rate in her setup) as a way to quantify how effectively content prevents tickets, which can be used to justify documentation investment to stakeholders.

Session visualization—via recordings and heat maps—adds the “watch and learn” layer. Instead of relying solely on analytics, teams can run weekly review sessions where they filter recordings to specific hypotheses, observe reading and interaction patterns, and generate a concrete backlog of improvements.

Finally, she addresses self-service score and why benchmarking alone can mislead. Self-service score is calculated as knowledge base sessions divided by tickets submitted, and it should be framed as speeding up the user experience, not reducing support. When benchmarks don’t clarify what to change, she recommends cohorting by source (e.g., overall vs. Google traffic vs. internal tool traffic) and by page or time-on-page to pinpoint where experiments should start. In Q&A, she adds practical guidance on handling negative feedback, filtering out cases driven by product expectations rather than documentation, and how lone writers can begin with a smaller set of tools and iterate.