There Is No Aesthetic Surprisal Arc

There is an old and seductive analogy between music and language: beauty is managed surprise. A great composition is neither monotonous nor noise; it rides a sweet spot between the two, shaping your expectation into a deliberate arc that builds tension and then resolves it. It is tempting to think writing works the same way: that a skilled author shapes how surprising each word is into a global arc over a page, and that this arc is both a fingerprint of human authorship and a mark of good prose. This study tests that idea directly, on public data, and reports a clean negative. There is no aesthetic surprisal arc. The shape of a text's surprise trajectory predicts neither whether a human or a machine wrote it nor how a human grades its quality. What carries the signal is mundane (the level and the variance of surprise) and the two behave very differently when someone tries to disguise machine text.

What does "surprisal" mean in a language model?

Surprisal is how wrong a language model's expectation was at each word: the negative log-probability it assigned to the word that actually appeared. A word the model expected has low surprisal; a word it did not see coming has high surprisal.

Every modern language model reads a text one token at a time and, at each step, has an opinion about what comes next. Run a whole text through the model and you get a sequence of these numbers, a surprise signal that rises and falls as the prose moves from the predictable to the unexpected and back. Everything in this study is built on that one sequence.

From that single signal you can pull three very different things, and keeping them apart is the whole point of the paper:

• The level, the average surprisal across the text. This is essentially perplexity: how predictable the writing is overall. Low level means the text was easy for the model to anticipate.
• The variance, how much the surprise jumps around from sentence to sentence. This is what detection researchers call burstiness. High variance means the writing lurches between the obvious and the unexpected; low variance means it hums along at a steady, even rate.
• The shape, the slow, global contour of the signal once you strip the level and the variance out of it. This is the "arc": does surprise build to a climax and resolve, the way tension does in a song? This is the part the aesthetic analogy is really about, and it is the part we set out to test.

The reason this matters is that two respected ideas make opposite predictions about that shape. The aesthetic view says skilled producers shape surprise, so the arc should carry real information. A long-standing hypothesis in psycholinguistics called Uniform Information Density (UID) says the opposite: efficient producers flatten surprise, spreading information as evenly as they can to make the text easy to process. Under UID, fluent writing should be flat, not arc-shaped, and the arc should carry nothing. We let the data referee.

How was the study set up?

The study was set up on independent, public, multi-author benchmarks only (no in-house corpus), with surprisal computed by open language models: primarily GPT-2, plus a second, larger model for robustness. Three concrete questions then ran across three datasets, with five-fold cross-validation throughout.

For each text we measured the three quantities above, isolating the arc as the low-frequency contour of the surprise curve after removing its mean and variance, exactly the "shape independent of level and loudness" the aesthetic claim points to.

TL;DR. Across three public benchmarks, the shape of a text's surprisal trajectory ("the arc") predicts neither AI authorship nor writing quality (detection AUC 0.52 to 0.55, essentially chance; ΔCV-R² = −0.001 for quality). What works is mundane: the level (perplexity) and variance (burstiness) of surprise. And under a paraphrase attack the two split: perplexity collapses from 0.80 to 0.58 AUC while burstiness barely moves (0.80 to 0.78), which makes burstiness the more durable tell. Every result replicates on a second, larger model, and even 30 book-length novels show no arc.

Experiment 1: Can the arc tell human from AI? (HC3)

The arc cannot tell human from AI: its shape scores 0.552 on HC3, barely above the 0.50 you would get from a coin flip. Over the very same texts, the level and the variance of surprise separate the two classes almost perfectly, with mean perplexity at a ROC-AUC of 0.969 and burstiness at 0.953.

The first test is the easy regime: short human-versus-ChatGPT answers from the HC3 benchmark, where raw machine text should be at its most obvious. Plotted on top of each other, the average human arc and the average AI arc are visually indistinguishable, and adding the arc to the level-plus-variance baseline does not improve it at all. On the regime most favorable to the hypothesis, the shape already carries nothing.

Experiment 2: Does any signal survive a humanization attack? (MAGE)

One surprise statistic survives a humanization attack and one does not: burstiness holds at 0.78 while mean perplexity collapses from 0.80 to 0.58. The second test moves to long-form, multi-domain text from the MAGE benchmark (news, dialogue summaries, reviews and biomedical abstracts) and adds the case that matters most for real detection: a paraphrase attack, where the machine text is rewritten to disguise it. Two findings come out of it.

First, the arc is still at chance. In both the raw and the paraphrased conditions its AUC sits at 0.52, even in the long-form, adversarial setting where a global tension curve had its best chance to matter. The shape does not rescue detection when detection actually gets hard.

Second, and more useful, the attack is selective. It does not degrade the two surprise statistics equally:

Paraphrasing collapses mean perplexity from 0.80 toward chance at 0.58: it successfully restores a human-looking level of predictability. But it leaves burstiness almost untouched at 0.78. Whatever a paraphraser does to even out word-by-word predictability, it does not restore the human rhythm of long and short, dense and loose. We checked this in every domain separately; the same pattern holds, with perplexity dropping from raw to paraphrased everywhere (for instance, IMDB 0.97 to 0.84 and PubMed 0.93 to 0.80), so it is not an artifact of mixing domains together. Burstiness is the attack-robust surprise statistic: the variance survives the disguise that the level does not.

Experiment 3: Does the arc track good writing? (ASAP-AES)

The arc does not track good writing. On ASAP-AES, student essays carrying independent human holistic scores, the dominant predictor of a score turns out to be the least glamorous one, length (per-set Spearman correlations of +0.75, +0.65 and +0.46).

This third test leaves detection behind and asks whether the arc predicts quality among human texts. Among the surprise features, what mildly predicts a higher score is lower perplexity and more uniformity, the UID-consistent direction, and the exact opposite of what the aesthetic-arc view would predict. More uniform, more predictable essays score higher, not lower.

And the arc shape adds nothing. Layered on top of a baseline of length, mean, variance and local uniformity, the arc block is not statistically significant (nested F = 1.79, p = 0.076) and actually worsens out-of-sample fit, nudging cross-validated R² from 0.341 to 0.340: a change of −0.001. High-scoring and low-scoring essays have overlapping average arcs. Whatever makes an essay good, it is not the shape of its surprise curve.

Do the results hold on a second language model?

Every qualitative result reproduced on a second model, Pythia-410m, roughly three times the size of GPT-2. The arc stayed at chance for detection (0.53), mean perplexity again collapsed under paraphrase, from 0.83 to 0.55, burstiness again held, from 0.83 to 0.77, and the arc again failed to predict quality (nested F = 1.18, p = 0.31).

A fair worry about any surprisal result is that it depends on which model you measured with, which is why we repeated all three experiments end to end. The negative is model-robust, length-robust, regime-robust and domain-robust at once, and so is the perplexity-fragile / burstiness-durable contrast under attack.

We also pre-empted the obvious objection, that we only looked at the slow part of the surprise signal and a faster band might hide a signal the single arc missed. So we split the whole surprise spectrum into eight frequency bands and checked. It helps, but barely: the band view adds about +0.011 AUC for detection over the level-plus-variance baseline on raw text, roughly nothing under paraphrase, and about +0.005 to cross-validated R² for quality. The broader spectrum holds a faint, statistically-detectable but practically-unusable sliver of structure. The headline does not move: it is surprise structure in general that is null, not just one operationalization of the arc.

Do book-length novels have a surprisal arc?

Book-length novels carry less arc, not more. Across 30 public-domain literary works, the slow, global-arc component of the signal actually shrinks as texts get longer: the lowest-frequency band holds 0.30 of the spectrum in short answers, 0.15 in essays, and only 0.12 in whole books.

The arc's last and best hope was book-length literary prose. Earlier work famously recovered six dominant emotional arc shapes from whole novels, so perhaps full-length stories shape surprise the same way. They do not. Book-length surprise is the flattest, most broadband writing of all: the least arc-shaped, not the most. And book arcs do not collapse into a few recognizable shapes the way emotional arcs do: the top two components explain just 22% of the variance. Surprise arcs are not emotional arcs. The negative is complete across every scale we could reach: short answers, essays and entire novels.

So what does give AI text away?

What gives AI text away is the level and the uniformity of surprise, not its shape: the boring statistics carry the signal that separates human from machine writing and tracks human quality. Raw machine text is too even, spreading information more uniformly and predictably than people do, which in the language of psycholinguistics means it overshoots Uniform Information Density.

Humans aim for evenness but never fully reach it (we spike, we stall, we reach for the odd word) and that residual unevenness is exactly the texture a simple perplexity-and-burstiness check reads when it flags raw model output. This is why those detectors work in the first place: not because AI writing has a tell-tale shape, but because it is flatter and steadier than the human writing it imitates.

Why did a humanizer lab study surprise arcs?

Surprise arcs drew us in because they promised a richer, more elegant signal than the blunt scalars every detector already leans on, and what actually distinguishes human from machine prose is squarely our problem. Reading human writing carefully across registers is the same instinct that sits behind our AI text detector.

Testing the idea honestly, on independent benchmarks, with a falsifiable design and open code, is the only way to know whether the elegant idea beats the blunt one. It does not. The blunt statistics win, and knowing which of them is durable is more practically useful than the arc ever would have been.

What does this mean for spotting AI text?

Spotting AI text comes down to robustness: the level of surprise and the variance of surprise are not equally hard to fake. A paraphrase pass can restore a human-like level of predictability (it knows how to make text look less machine-confident on average) but it does not restore human-like burstiness, the lived-in unevenness of real sentence rhythm.

So a check that leans on variance has firmer ground under it than one that leans on average predictability alone, which is the half a rewrite most easily smooths away. If you want to understand a passage rather than just score it, you can run text through our detector and read it with this in mind: a suspiciously even rhythm is a stronger tell than a merely low average. For the companion story on how a real-corpus approach reads these same human-versus-machine textures from the product side, the 2,590-essay corpus deep dive is the read; the rest of our studies live on the research hub.

A note on honesty, since detection is a field full of overconfident numbers. None of this says detectors are infallible: the opposite. They lean on statistical texture, not on truth, and that texture misfires: careful, formal, second-language academic writing is naturally low in perplexity and even in rhythm, which is the same fingerprint raw machine text carries, and it gets wrongly flagged for it. Perplexity is also measured relative to whatever model does the scoring, which is one more reason a single number should never be a verdict, though it is worth noting our own result is stable across two different scoring models, so the pattern here is not an accident of one model's quirks. The finding in this paper is narrow and specific on purpose: the shape of surprise does not work, the level and variance do, and only the variance survives a disguise. It is not a license to trust any single score.

Can anyone reproduce these results?

Anyone can. Every dataset in the study is public (HC3, MAGE and ASAP-AES), and a single command reproduces every number reported here, on either of the two surprise models.

The full methods, every statistic, all the figures and a frank limitations section are written up in the preprint on ResearchGate, and the complete analysis code with a fully reproducible pipeline is openly available on GitHub, so the negative can be checked and extended rather than taken on faith. The appealing idea that writers shape surprise into an arc does not survive contact with the data; the signal lives in the boring statistics, and we have left everything out in the open so you can confirm it yourself.