There Is No Aesthetic Surprisal Arc

There is an old and seductive analogy between music and language: beauty is managed surprise. A great composition is neither monotonous nor noise; it rides a sweet spot between the two, shaping your expectation into a deliberate arc that builds tension and then resolves it. It is tempting to think writing works the same way: that a skilled author shapes how surprising each word is into a global arc over a page, and that this arc is both a fingerprint of human authorship and a mark of good prose. This study tests that idea directly, on public data, and reports a clean negative. There is no aesthetic surprisal arc. The shape of a text's surprise trajectory predicts neither whether a human or a machine wrote it nor how a human grades its quality. What carries the signal is mundane (the level and the variance of surprise) and the two behave very differently when someone tries to disguise machine text.

What "surprisal" means, in one plain sentence

Every modern language model reads a text one token at a time and, at each step, has an opinion about what comes next. Surprisal is just how wrong that opinion was: the negative log-probability the model assigned to the word that actually appeared. A word the model expected has low surprisal; a word it did not see coming has high surprisal. Run a whole text through the model and you get a sequence of these numbers, a surprise signal that rises and falls as the prose moves from the predictable to the unexpected and back. Everything in this study is built on that one sequence.

From that single signal you can pull three very different things, and keeping them apart is the whole point of the paper:

• The level, the average surprisal across the text. This is essentially perplexity: how predictable the writing is overall. Low level means the text was easy for the model to anticipate.
• The variance, how much the surprise jumps around from sentence to sentence. This is what detection researchers call burstiness. High variance means the writing lurches between the obvious and the unexpected; low variance means it hums along at a steady, even rate.
• The shape, the slow, global contour of the signal once you strip the level and the variance out of it. This is the "arc": does surprise build to a climax and resolve, the way tension does in a song? This is the part the aesthetic analogy is really about, and it is the part we set out to test.

The reason this matters is that two respected ideas make opposite predictions about that shape. The aesthetic view says skilled producers shape surprise, so the arc should carry real information. A long-standing hypothesis in psycholinguistics called Uniform Information Density (UID) says the opposite: efficient producers flatten surprise, spreading information as evenly as they can to make the text easy to process. Under UID, fluent writing should be flat, not arc-shaped, and the arc should carry nothing. We let the data referee.

How we set it up

We deliberately used only independent, public, multi-author benchmarks (no in-house corpus) and we computed surprisal with open language models (primarily GPT-2, with a second, larger model for robustness). For each text we measured the three quantities above, isolating the arc as the low-frequency contour of the surprise curve after removing its mean and variance, exactly the "shape independent of level and loudness" the aesthetic claim points to. Then we asked three concrete questions on three datasets, with five-fold cross-validation throughout.

TL;DR. Across three public benchmarks, the shape of a text's surprisal trajectory ("the arc") predicts neither AI authorship nor writing quality (detection AUC 0.52–0.55, essentially chance; ΔCV-R² = −0.001 for quality). What works is mundane: the level (perplexity) and variance (burstiness) of surprise. And under a paraphrase attack the two split: perplexity collapses from 0.80 to 0.58 AUC while burstiness barely moves (0.80 to 0.78), which makes burstiness the more durable tell. Every result replicates on a second, larger model, and even 30 book-length novels show no arc.

Experiment 1: Can the arc tell human from AI? (HC3)

The first test is the easy regime: short human-versus-ChatGPT answers from the HC3 benchmark, where raw machine text should be at its most obvious. Here the level and the variance of surprise separate the two classes almost perfectly. Mean perplexity scores a ROC-AUC of 0.969 and burstiness 0.953, near-flawless. The arc shape, by contrast, scores 0.552, barely above the 0.50 you would get from a coin flip. Plotted on top of each other, the average human arc and the average AI arc are visually indistinguishable, and adding the arc to the level-plus-variance baseline does not improve it at all. On the regime most favorable to the hypothesis, the shape already carries nothing.

Experiment 2: Does any signal survive a humanization attack? (MAGE)

The second test moves to long-form, multi-domain text from the MAGE benchmark (news, dialogue summaries, reviews and biomedical abstracts) and adds the case that matters most for real detection: a paraphrase attack, where the machine text is rewritten to disguise it. Two findings come out of it.

First, the arc is still at chance. In both the raw and the paraphrased conditions its AUC sits at 0.52, even in the long-form, adversarial setting where a global tension curve had its best chance to matter. The shape does not rescue detection when detection actually gets hard.

Second, and more useful, the attack is selective. It does not degrade the two surprise statistics equally:

Paraphrasing collapses mean perplexity from 0.80 toward chance at 0.58: it successfully restores a human-looking level of predictability. But it leaves burstiness almost untouched at 0.78. Whatever a paraphraser does to even out word-by-word predictability, it does not restore the human rhythm of long and short, dense and loose. We checked this in every domain separately; the same pattern holds, with perplexity dropping from raw to paraphrased everywhere (for instance, IMDB 0.97 to 0.84 and PubMed 0.93 to 0.80), so it is not an artifact of mixing domains together. Burstiness is the attack-robust surprise statistic: the variance survives the disguise that the level does not.

Experiment 3: Does the arc track good writing? (ASAP-AES)

The third test leaves detection behind and asks whether the arc predicts quality among human texts. We used ASAP-AES, student essays carrying independent human holistic scores. The dominant predictor of a score turns out to be the least glamorous one, length (per-set Spearman correlations of +0.75, +0.65 and +0.46). Among the surprise features, what mildly predicts a higher score is lower perplexity and more uniformity, the UID-consistent direction, and the exact opposite of what the aesthetic-arc view would predict. More uniform, more predictable essays score higher, not lower.

And the arc shape adds nothing. Layered on top of a baseline of length, mean, variance and local uniformity, the arc block is not statistically significant (nested F = 1.79, p = 0.076) and actually worsens out-of-sample fit, nudging cross-validated R² from 0.341 to 0.340: a change of −0.001. High-scoring and low-scoring essays have overlapping average arcs. Whatever makes an essay good, it is not the shape of its surprise curve.

We ran the whole thing on a second model

A fair worry about any surprisal result is that it depends on which model you measured with. So we repeated all three experiments with Pythia-410m, a model roughly three times the size of GPT-2, and every qualitative result reproduced. The arc stayed at chance for detection (0.53). Mean perplexity again collapsed under paraphrase, from 0.83 to 0.55, while burstiness again held, from 0.83 to 0.77. And the arc again failed to predict quality (nested F = 1.18, p = 0.31). The negative is model-robust, length-robust, regime-robust and domain-robust at once, and so is the perplexity-fragile / burstiness-durable contrast under attack.

We also pre-empted the obvious objection, that we only looked at the slow part of the surprise signal and a faster band might hide a signal the single arc missed. So we split the whole surprise spectrum into eight frequency bands and checked. It helps, but barely: the band view adds about +0.011 AUC for detection over the level-plus-variance baseline on raw text, roughly nothing under paraphrase, and about +0.005 to cross-validated R² for quality. The broader spectrum holds a faint, statistically-detectable but practically-unusable sliver of structure. The headline does not move: it is surprise structure in general that is null, not just one operationalization of the arc.

Even novels have no arc

The arc's last and best hope was book-length literary prose. Earlier work famously recovered six dominant emotional arc shapes from whole novels, so perhaps full-length stories shape surprise the same way. We computed book-length surprise arcs for 30 public-domain literary works and found the opposite of the hypothesis. The slow, global-arc component of the signal actually shrinks as texts get longer: the lowest-frequency band holds 0.30 of the spectrum in short answers, 0.15 in essays, and only 0.12 in whole books. Book-length surprise is the flattest, most broadband writing of all: the least arc-shaped, not the most. And book arcs do not collapse into a few recognizable shapes the way emotional arcs do: the top two components explain just 22% of the variance. Surprise arcs are not emotional arcs. The negative is complete across every scale we could reach: short answers, essays and entire novels.

So what does give AI text away?

The boring statistics. What separates human from machine writing, and what tracks human quality, is the level and the uniformity of surprise, not its shape. And the mechanism behind it has a name. Raw machine text is too even: it spreads information more uniformly and predictably than people do. In the language of the psycholinguistics, it overshoots Uniform Information Density. Humans aim for evenness but never fully reach it (we spike, we stall, we reach for the odd word) and that residual unevenness is exactly the texture a simple perplexity-and-burstiness check reads when it flags raw model output. This is why those detectors work in the first place: not because AI writing has a tell-tale shape, but because it is flatter and steadier than the human writing it imitates.

Why a humanizer lab studied surprise arcs

We build language tools. Our work centers on reading human writing carefully across registers, the same instinct behind our AI text detector, so a question about what actually distinguishes human from machine prose is squarely our problem. The appeal of the arc was that it promised a richer, more elegant signal than the blunt scalars everyone already uses. Testing it honestly, on independent benchmarks, with a falsifiable design and open code, is the only way to know whether the elegant idea beats the blunt one. It does not. The blunt statistics win, and knowing which of them is durable is more practically useful than the arc ever would have been.

What this means for spotting AI text

The cleanest practical takeaway is about robustness. The level of surprise and the variance of surprise are not equally hard to fake. A paraphrase pass can restore a human-like level of predictability (it knows how to make text look less machine-confident on average) but it does not restore human-like burstiness, the lived-in unevenness of real sentence rhythm. So a check that leans on variance has firmer ground under it than one that leans on average predictability alone, which is the half a rewrite most easily smooths away. If you want to understand a passage rather than just score it, you can run text through our detector and read it with this in mind: a suspiciously even rhythm is a stronger tell than a merely low average. For the companion story on how a real-corpus approach reads these same human-versus-machine textures from the product side, the 2,590-essay corpus deep dive is the read; the rest of our studies live on the research hub.

A note on honesty, since detection is a field full of overconfident numbers. None of this says detectors are infallible: the opposite. They lean on statistical texture, not on truth, and that texture misfires: careful, formal, second-language academic writing is naturally low in perplexity and even in rhythm, which is the same fingerprint raw machine text carries, and it gets wrongly flagged for it. Perplexity is also measured relative to whatever model does the scoring, which is one more reason a single number should never be a verdict, though it is worth noting our own result is stable across two different scoring models, so the pattern here is not an accident of one model's quirks. The finding in this paper is narrow and specific on purpose: the shape of surprise does not work, the level and variance do, and only the variance survives a disguise. It is not a license to trust any single score.

Open data, open code

This is not a black box. The full methods, every statistic, all the figures and a frank limitations section are written up in the preprint on ResearchGate, and the complete analysis code with a fully reproducible pipeline is openly available on GitHub. Every dataset is public (HC3, MAGE and ASAP-AES) and a single command reproduces every number in this study, on either of the two surprise models, so the negative can be checked and extended rather than taken on faith. The appealing idea that writers shape surprise into an arc does not survive contact with the data; the signal lives in the boring statistics, and we have left everything out in the open so you can confirm it yourself.