How Novelcrafter researches a 230,000-user community in minutes

The challenge

Novelcrafter is writing software for people juggling too many ideas at once: a story bible for characters, locations and world building progressions, chapters you can rearrange without breaking continuity, and BYOK AI where the author keeps full control of the prompts and the models. Bootstrapped since September 2023, it has reached 230,000 registered users and a Discord community near 10,000 members on word of mouth alone, with a team of eight.

That community talks everywhere: Discord, Intercom, Canny, Reddit, and author forums Novelcrafter doesn't own. Each one came with its own search box.

In Canny alone, thousands of comments had accumulated. Reviewing and organizing that feedback took dedicated time: so much so Novelcrafter brought in a short-term contractor to work through the backlog and migrate it to their ticket tracker.

"The hardest part about feedback is that we need to play doctor," Leonie says. "Someone has an issue, and they don't tell us the issue. They tell us the solution they think might help." Separating a bug from a UX dead end from a genuine feature gap, in German, French, Chinese and everything else, is judgment work, and there was only ever one small team to do it. During a year focused on backend rewrites, the team kept receiving feedback on problems it was already working to solve, but didn't have the time to file away properly.

Why Modem

What Novelcrafter needed wasn't another inbox. It was memory that outlives whichever tool the feedback happened to arrive in.

Support tools already analyze their own data, but only their own. "Each tool is its own ecosystem. With Modem we can do the same for Discord, the same for Intercom, the same across other channels, because it connects to everything."

Modem in action: the daily hallucination report

Novelcrafter answers first-line support with an Intercom AI bot. Those conversations flow into Modem continuously, already aggregated and analyzed alongside every other channel. Each morning a Modem automation scans the previous day's worth and posts a short report to Slack on where the bot got it wrong. Three kinds of signal surface:

• Wrong answers, which usually trace back to documentation that contradicts itself across pages
• Missing features, which the bot doesn't know exist because nobody documented them
• Human hand-offs, the inverted signal: if support had to step in and correct the bot, that itself is evidence something was off

The bot is wrong because the docs are. It flags at least two such gaps a day, sometimes the same one recurring until the docs catch up. And the fix compounds well past Intercom.

Research on demand

When a Unicode text-handling bug surfaced in support, the team used Modem to look across Discord, Canny, and Intercom for related bugs people had reported in other languages, including Sanskrit and Turkish. One question brought together reports that would otherwise have required separate searches. Answers land in under five minutes.

Novelcrafter already knew it served a large romance-writing community. The ability to choose models suited to their stories is one reason those writers use the product. While preparing a landing page for romance writers, the team used Modem to find the questions they were asking in support channels that the marketing page could answer directly. Modem ranked those questions by how often they came up over six months.

That ranking helped shape the landing page's sections around recurring support questions. The team also used Modem to research a challenge it had known about since day one: models refusing to write certain kinds of content. This affected romance material, but also scenes involving school bullying, war, or workplace harassment. When Novelcrafter wanted to start working on AI guidance pages, Modem brought the relevant support reports together without the team having to search through every channel manually. That research informed a benchmark of whether models would write about different categories of difficult content. The benchmark measures models' openness to those subjects, not the quality of the writing.

The reach isn't limited to support channels. With PostHog connected, the team can use Modem to check which features may be underused, then investigate why: does a feature need better documentation, or are users simply not discovering it during their trial? Bringing product analytics and support feedback into the same place gives the team more context for those questions. "One tool we have everything connected to."

The results

The measurable win is documentation quality: at least two gaps surfaced every day, in a product whose docs are the first thing humans and chatbots both read.

The less visible win is staying in flow. "I do a bit of marketing, I need to do the business, I need to do tech work," Leonie says. "Modem lets me keep being in the flow a lot more."

Prefer it as a document? Download the case study PDF.