An AI that hunts AI-sounding prose: the signal scanner

PlotDevice helps you write a novel with AI. Which creates an obligation most writing tools dodge: helping you find the places where the novel sounds like it was written with AI. Three days of work on the review system produced the feature I suspect people will actually buy this app for — an AI signal scanner — plus the two-phase continuity engine underneath and a validation pass whose entire job is distrusting the reviewer.

Two phases, because context windows aren't continuity

A manuscript review covers nine categories: character, arc, setting, atmosphere, style, continuity, object, and the two newcomers this post is about. The naive approach (stuff the book into a prompt, ask for problems) fails structurally: continuity errors live in the gaps between chapters, and attention over 80,000 tokens is a rumour.

So review is two phases. Extract: every section independently gets distilled into structured JSON notes (characters, locations, time references, items, plot events, stated facts), with instructions to be exhaustive ("extract EVERY character mention, physical detail, gender indicator"). Items even track an owner and location history. Cross-reference: the notes, not the prose, get compared across sections for direct contradictions, items appearing or disappearing without explanation, impossible travel. Token budgeting is explicit (availableTokens = context_window - 4500), and if the notes overflow, they're batched by category rather than truncated.

The findings prompt has one demand I've come to consider the heart of the system: fixes must be "exact replacement prose ready to swap in, not a description of what to change and not multiple options (no 'or' alternatives). Pick the best fix and commit to it." A reviewer that hedges is a reviewer whose suggestions can't be applied with a button.

The signal list

The AI signal scanner is a ~950-word system prompt describing ten signals of AI-flavoured prose. Signal one is the word list, quoted here in full as both documentation and confession:

"delve", "tapestry", "nuanced", "multifaceted", "intricate", "pivotal", "landscape", "embark", "foster", "leverage", "moreover", "furthermore", "intricacies", "comprehensive", "facilitate", "testament to", "myriad", "plethora", "resonate", "underscores", "navigating", "realm"

The rest are patterns, not words: formulaic transitions ("Little did they know..."), stock emotion ("felt a pang of", "heart swelled with"), generic metaphors ("tapestry of", "symphony of", "mosaic of"), hedging, telling-not-showing, summary narration, and the weak comma-conjunction join, my favourite for its specificity: "It had taken her brother, and now it watched her" is weaker than "It had taken her brother. Now it watched her." That one earns its place because it's the AI tell I catch in my own drafting most often.

There are no regexes anywhere in this. Every signal is LLM-judged in context, which is the only way "intricate" in a sentence about lace survives while "intricate tapestry of emotions" gets flagged. The prompt even warns the scanner about its own failure mode: "Common traps: replacing one cliché with another (e.g., 'hit like a blow' → 'the world tilted on its axis')." An AI editor needs to be told that its instinctive rewrites are made of the same stuff it's hunting.

The em-dash gets its own department

Late last night, punctuation split into its own review category with its own prompt: em-dash overuse, mechanical Oxford commas, semicolon excess, comma splices, ellipsis abuse, the dramatic colon. The guard rail is the giveaway about what's really going on:

"Never suggest em-dashes as the 'human' or 'natural' alternative... Em-dash overuse is itself a top AI pattern."

And because prompts are requests, not guarantees, there's a hard filter behind it: any suggested fix containing — is dropped in code. The same commit added six punctuation-avoidance rules to the generation prompts. The cheapest place to fix AI-sounding prose is before it exists.

Fixes you can actually apply

A finding without a location is a complaint. Jump-to-source resolves a finding's quote back to a position in the editor through three tiers: exact indexOf, then a typographically-normalised regex (smart quotes, en/em dashes, ellipses, and nbsp all degraded to equivalence classes), then case-insensitive. Straight quotes in an LLM's output still match curly quotes in the manuscript. Tier two earns its keep hourly.

Applying fixes takes two paths. Signal findings carry their replacement already, so accepting one is a direct TipTap deleteSelection().insertContent(replacement). Consistency findings get a preview dialog (original and proposed side by side, with Regenerate), because a continuity fix rewrites meaning, and meaning deserves a look before a click.

The reviewer gets reviewed

Today's commit added the piece that makes me trust the rest: two-pass validation. Before findings reach the UI, a second LLM call re-checks each one against the actual prose and returns a boolean verdict per finding. The discard rules are a taxonomy of reviewer failure: claims an Oxford comma where there are only two items; vague gestures ("AI frequently employs..."); flagging deliberate literary devices; fixes that swap near-synonyms ("offered" → "gave"). Validation fails open: a parse error keeps all findings rather than silently dropping a section's review. Scores then get computed per category (each finding costs 15 points; signal and punctuation findings only 5, since style nits shouldn't tank a manuscript the way a continuity break does).

Honest ledger

Everything runs sequentially: one extract call per section, one validation call per section per phase. A 30-section book is roughly 120 LLM calls in single file. Parallelism is the obvious next move.
The Electron and web apps have two hand-mirrored copies of the 1,400-line review backend. Every fix lands twice or rots in one. This is unsustainable and I know it.
Auto-fix marks a finding "fixed" even when the quote can't be found in the document: success theatre in a status field.
The phase numbering in the progress UI skips 3. Nobody has noticed.

The shape worth keeping: generator, critic, validator — three different prompts, three different jobs, and the last one exists because the middle one lies. Anyone building LLM review of anything will land here eventually; fiction just makes the failure modes funnier. Testers will recognise the shape: the oracle problem doesn't disappear when the reviewer is a model. It just moves up a level.