Abstract. People judge whether text was written by a model from checklists of “tells”, habits said to mark machine writing, yet few have been measured against matched human writing. We gave seven models (three local, two from OpenAI, two from Anthropic) the same 800 Hacker News comments and Stack Exchange questions that people answered between 2019 and 2022, and counted 23 tells under a pre-registered plan (tested on 600 human and 4,199 model replies). “Rather than” was 4.3 times as frequent in model replies (95% CI 2.7 to 6.7) and lists of three 3.3 times (2.5 to 4.3). The automated count of lists of three failed our human check; corrected for its errors, the ratio stays above 2. Em dashes and flagged words like “delve” hold up on average but split by model: the Claude models used almost no em dashes. Newer words such as “quietly” also hold up, but appear in only 1.3% of human replies. Seven tells were more common in the human replies. The models wrote under one bare prompt, so chat products may differ. Next we test whether readers use these tells.
Introduction
MRL builds writing tools; these results will inform them.
The problem. Lists of the signs that a text was written by a large language model (LLM) circulate on Hacker News and Reddit, in Wikipedia’s guide to AI writing and in advice from editors and teachers. The same items recur: em dashes, “delve” and “crucial”, contrasts such as “it’s not X, it’s Y” and “Y rather than X”, lists of three, summary closers. People use these lists to judge other people’s writing, and writers, us among them, edit their own to steer clear of them. Most items on the lists have never been measured against human writing.
Why it matters. A tell that is common in human writing produces false accusations: a student or a job applicant who writes “Note that” is flagged for a habit that, in our data, humans have more than models do. A tell that only one model has tells you about that model, not about LLMs. And a style guide that bans a pattern on the strength of a rumour makes writing worse for no gain. Our own house style bans several of these patterns, and its section on what is allowed was waiting on this study.
Why it’s hard. A fair comparison needs human and model text written for the same purpose, at the same length, from before models could have written the human side. It needs more than one model, because models differ. And each tell has to be counted by a fixed rule that someone else can rerun, with a check that the rule counts what it claims to. The tells also move over time.
What was known. Word frequencies shifted in scientific abstracts after ChatGPT: in 2024, “delves” appeared in 28 times as many PubMed abstracts as its trend from 2021 to 2022 predicted. Kobak et al. 2025 Several words flagged in early 2024, “delve” among them, then dropped sharply in arXiv abstracts soon after they were pointed out, while commoner flagged words such as “significant” kept rising. Geng and Trotta 2025 Juzek and Ward 2025 found no evidence that training data, model architecture or algorithm choices explain such words; their tests were consistent with reinforcement learning from human feedback (RLHF) playing a role, though they call that evidence mixed. GPT-4o and instruction-tuned Llama 3 use present participial clauses about 2 to 5 times as often as humans when continuing the same text, and base models are close to human rates. Reinhart et al. 2025 People are poor judges: in six experiments with 4,600 participants, profiles written by 2022 models for dating, hospitality and professional contexts were identified with 50% to 52% accuracy Jakesch et al. 2023, and annotators who rarely use LLMs for writing were near chance on non-fiction articles. Annotators who use them often caught 93% of the AI-written articles while flagging 3% to 4% of the human ones, and named vocabulary and “listing three items” among their cues. Russell et al. 2025 What we did not find was a test of the popular tells one at a time, against human writing on the same task, across several current models, with a verdict for each that a reader could check.
What we did. We took 800 human replies from before ChatGPT and gave seven 2026 models exactly what each human saw, with the same target length. We counted 21 tells by fixed rules and had models code two more (the coders disagreed, so those two get no verdict), compared each with the human rate under a plan written down before any text was collected, and checked the two hardest counters against a human coder. The contributions:
- A verdict for each of 23 claimed tells, pooled over seven models and reported per model Section 03
- Two tells that are more common than in human writing in every model, two that depend on the model and one that is rare on both sides Section 03, Table 2
- Seven tells that were more common in the human writing Section 03
- Released data: post IDs, labels, counts, prompts and code, so anyone can rerun the analysis Data and code
Method
Human text. We used 800 replies written between January 2019 and November 29, 2022, the day before ChatGPT’s release: 400 Hacker News comment replies, drawn at random by item ID, and 400 Stack Exchange answers from 10 non-programming sites (cooking, travel, parenting, personal finance and six others, 40 per site), each with a score of at least 1 and not edited after the cutoff. Both are replies, so each human text has a prompt we can give to a model. Wikinews was planned as a third register (kind of text) and dropped: too few articles per year.
Models. Each model got what the human saw (the story title and the parent comment, or the question) and the request “Write a reply to this comment, about N words long. Give only the text of the reply.”, with N the human reply’s length. There was no other system prompt, and each model wrote one reply per prompt.
| Model | Where | Reasoning | Sampling |
|---|---|---|---|
| Qwen 3.8 27B | LM Studio, 4-bit | Off | Model card defaults, seeded |
| Gemma 4 E4B | LM Studio, 4-bit | Off | Model card defaults, seeded |
| Meta Muse Glimmer 30B | LM Studio, 4-bit | Lowest level, “none”, set by one line of system prompt with no style instruction (it cannot be switched off) | LM Studio defaults (temperature 0.8, top-p 0.95, top-k 40), seeded |
| gpt-6.1-sol | OpenAI API | low, its lowest | API defaults, seeded |
| gpt-6-luna | OpenAI API | Off | API defaults, seeded |
| Claude Sonnet 5.5 | Anthropic Batch API | Thinking between_tools, its off setting (disabled is rejected); effort at the default, high | API defaults; the API takes no seed |
| Claude Opus 5.5 | Anthropic Batch API | Effort low (thinking cannot be switched off) | API defaults; the API takes no seed |
The three local models cover three companies and sizes that run on one laptop. From OpenAI we took a mid-tier model and the smallest, the only one that runs with reasoning off; from Anthropic, the current Sonnet and Opus. A reply that was empty, opened like a refusal or was mostly non-English was regenerated once. Opus refused one prompt twice (a cybersecurity question) and that reply is excluded.
Tells. The 23 come from our review of the claims, finished on Nov 6, 2026: about 2,440 Hacker News comments from 2023 to 2026, Wikipedia’s guide to signs of AI writing, editors’ and teachers’ advice, news write-ups and academic studies. We took the tells that recur across those sources and can be counted or coded in a short reply; 10 were registered as primary tests and the rest as secondary. We counted 21 of them by fixed rules, written down before any text was collected: regular expressions for word and phrase tells, and a dependency parser (spaCy) for lists of three and trailing “-ing” clauses. Markdown was rendered the same way on both sides; formatting tells (lists, headings, bold) are counted only on Stack Exchange, where the human writers had Markdown too. Two content tells, “generic” and “hype”, have no countable form and were coded by two models on a subsample.
Analysis. Each tell is compared as a rate per 1,000 words, or as the share of replies with at least one where the tell is rare (a switch rule fixed on the exploratory data). The model is a Poisson GEE with robust standard errors (modified Poisson for shares; Zou 2004), clustered by prompt, so that each human reply and its model replies stay together, with each model and register weighted equally. For tells too sparse for the GEE (a rule fixed in advance), the ratio uses a Haldane correction, the interval a bootstrap over prompts and the test a permutation of labels within prompts. For those, a verdict follows the permutation test, and the interval can cross 1. We fixed the counting rules on a quarter of the prompts and ran the test once on the other three quarters: 300 prompts per register, 600 human and 4,199 model replies. Multiple testing is corrected by Holm 1979 across the 10 primary tells and Benjamini and Hochberg 1995 across the rest.
Verdicts. A tell holds up if it is significantly more common in model text after correction for multiple tests, at least twice as common, and seeing it at least doubles the odds that a reply is model-written (a positive likelihood ratio, LR+, of 2 or more; the ratio compares rates per word or, for rare tells, shares of replies, and LR+ always counts replies that contain the tell). It is real but weak if significant but under one of those bars, does not hold up if its 90% interval lies inside 0.67 to 1.5, reversed if significantly more common in human text, and inconclusive otherwise. Ratios of 2 for both bars are the smallest differences that would make a tell useful as a flag.
Checking the counters. Two model coders, GLM-4.7-Flash run locally with thinking off and Claude Haiku, coded 275 replies blind to source to check the counters. They reached our agreement bar (Cohen’s κ of 0.6) on only three tells, because GLM could not do the grammatical codes (it found no lists of three in any reply). “Rather than”, em dashes and the flagged words are exact text matches and need no coder. For the two parser-based counters we added a human check: before any coding, we wrote down that a human must confirm at least 70% of each counter’s hits on human and on model text alike, with limits also on the gap between those two shares and on how many true cases a counter misses (the thresholds are in the pre-registration), and that if a counter failed, its ratio would be corrected for its errors and the tell not listed as holding up if the correction took it under 2. One of us coded 80 replies without knowing who wrote them, though knowing the hypotheses, and Claude Fable coded the same replies as a cross-check. We then reviewed our codes wherever they disagreed with the counter or with Fable, plus 10 replies where all three agreed; the reviewed codes decide. The replies checked came from the local models, so applying the corrections to the hosted models assumes the counters err the same way there. No outside reader checked that each claim matches the data.
Deviations. The plan, the data collection and the first test all happened on November 7, 2026. The plan was written and dated before any text was collected but has no independent timestamp. The pre-registration dates 19 changes. Each was written down before it was run, except those found after the fact, which say so. Before the test run, the counting rules were tightened after a code review, the Stack Exchange sample was redrawn so that each question appears once, and Muse’s reasoning level was set by system prompt. The first test, as registered, covered the three local models (November 7). On November 8 and 9, we added the human check and its review, after the main results. On November 9, we added the two OpenAI models, with what would count as a tell replicating in each written down before any text was generated. A Gemini arm was registered the same way and dropped before measurement because the free API was too overloaded to finish. After seeing the OpenAI results we decided to run the registered test, unchanged, over every model. On November 14, we added the two Claude models under a rule registered before any request. The rule also said that a third Claude model would be added only if Sonnet and Opus disagreed on a headline tell, and they did not. On November 14, a test written for the code release found that 27 of the 125 answers in the Llama side test exceeded the sampling rule’s 600-word limit; we kept the test as run and report a rerun without them beside it. The seven-model run is the headline, and the three-model run is reported beside it. Between the two, “it’s not X, it’s Y” and stacked transitions moved from inconclusive to reversed, the 2026 words from inconclusive to holding up, and sentence-length uniformity held at real but weak (it briefly passed in a five-model run). One rule was added after the results and is not in the pre-registration: two verdicts that rest on 3 human replies, below the 5 at which the registered plan already switches to its sparse-tell method, are reported with their counts but not counted among the tells that hold up or are reversed (Result). The registered robustness checks (a negative binomial model, a cluster bootstrap for the primary tells, alternative counting rules for two tells and an 80-word floor on Hacker News) were run on the three-model test; none moved a tell across the bar of 2 or reversed its direction. They are in the release.
Result
← more in human repliesmore in model replies →
- “Y rather than X”4.26 (95% CI 2.70 to 6.71)
- Lists of three3.28 (95% CI 2.52 to 4.26); corrected 3.04
- 2026 words (quietly, load-bearing)2.92 (95% CI 1.38 to 6.18)
- Flagged words (delve, crucial)2.36 (95% CI 1.39 to 4.02)
- Em dashes2.12 (95% CI 1.28 to 4.24)
Doubtful on measurement 1
- Trailing “-ing” clauses2.26 (95% CI 1.65 to 3.10); corrected 0.97
Real but weak 2
- Nominalizations1.72 (95% CI 1.60 to 1.85)
- More uniform sentence lengthsd = −0.55 (−0.65 to −0.46), a standardized difference
- Bold text, Stack Exchange0.63 (95% CI 0.42 to 0.96)
- “It’s not X, it’s Y”0.63 (95% CI 0.45 to 0.87)
- Any list, Stack Exchange0.45 (95% CI 0.29 to 0.69)
- Stacked transitions0.40 (95% CI 0.23 to 1.09)
- “It’s worth noting”, “Note that”0.26 (95% CI 0.13 to 0.64)
- Self-answered questions0.24 (95% CI 0.13 to 0.68)
- Staccato fragments0.24 (95% CI 0.15 to 0.36)
Inconclusive 6
- Sycophantic openers1.26 (95% CI 0.68 to 3.74)
- Hedging closers0.59 (95% CI 0.30 to 1.37)
- Summary closers0.41 (95% CI 0.22 to 1.01)
- Signposting0 of 600 human, 24 of 4,199 model replies
- “Generic” contentModel-coded; the coders disagreed
- “Hype” contentModel-coded; the coders disagreed
Too few to read: 3 human replies each 2
- Bold lead-ins on list items, Stack Exchange2.63 (95% CI 0.98 to 16.96)
- Headings, Stack Exchange0.26 (95% CI 0.09 to 1.72)
Table 1 gives every verdict over seven models, with the registered three-model run beside it.
| Tell | Seven models | Verdict | Three local models (first run) |
|---|---|---|---|
| “Y rather than X” share | 4.26 2.70–6.71 | holds up | 3.38, holds up |
| Lists of three “forms, census tables and medical records”; counter not validated | 3.28 2.52–4.26; corrected 3.04 | holds up | 2.59, corrected 2.40, holds up |
| Em dashes share | 2.12 1.28–4.24 | holds up | 4.11, holds up |
| Flagged words: delve, crucial, robust, landscape and others share | 2.36 1.39–4.02 | holds up | 5.76, holds up |
| 2026 words: quietly, load-bearing and others share | 2.92 1.38–6.18 | holds up | 2.37, inconclusive |
| Trailing “-ing” clauses “…, highlighting the need for…” | 2.26 1.65–3.10; corrected 0.97 | doubtful on measurement | 3.47, corrected 1.49, doubtful on measurement |
| Nominalizations “perform an evaluation of” | 1.72 1.60–1.85 | real but weak | 1.97, real but weak |
| More uniform sentence lengths | d = −0.55 −0.65 to −0.46; LR+ 1.93 | real but weak | d = −0.59, real but weak |
| “It’s not X, it’s Y” share | 0.63 0.45–0.87 | reversed | 1.18, inconclusive |
| Staccato fragments “No setup. No config.” (share) | 0.24 0.15–0.36 | reversed | 0.22, reversed |
| “It’s worth noting”, “Note that” share | 0.26 0.13–0.64 | reversed | 0.11, reversed |
| Self-answered questions share | 0.24 0.13–0.68 | reversed | 0.23, reversed |
| Stacked transitions “Moreover, … Furthermore, …” (share) | 0.40 0.23–1.09 | reversed | 0.95, inconclusive |
| Any list, Stack Exchange share | 0.45 0.29–0.69 | reversed | 0.28, reversed |
| Bold text, Stack Exchange | 0.63 0.42–0.96 | reversed | 0.17, reversed |
| Headings, Stack Exchange share | 0.26 0.09–1.72 | reversed, on 3 human and 5 model replies | 0.27, inconclusive |
| Bold lead-ins on list items, Stack Exchange share | 2.63 0.98–17.0 | holds up, on 3 human replies | 1.31, inconclusive |
| Signposting “Here’s the thing”, “Let’s dive in” (share) | 0 of 600 human, 24 of 4,199 model replies | inconclusive | inconclusive |
| Summary closers, sycophantic openers, hedging closers | 0.41 to 1.26, intervals crossing 1 | inconclusive | inconclusive |
| “Generic” and “hype” content model-coded, three local models | the two coders disagreed on direction | inconclusive (measurement not validated) | inconclusive (measurement not validated) |
The headings and bold lead-in verdicts rest on 3 human replies each (out of 300), too few to read. We give the counts, and by a rule added after the results (Deviations), we don’t count them among the reversed or holding-up tells below. That leaves five tells that hold up over seven models: “rather than”, lists of three, em dashes, the flagged words and the newer words such as “quietly”.
Two tells in every model. “Rather than” and lists of three were more common than in the human replies in all seven models (Table 2). Each of the four hosted models was also tested on its own, under the rule registered for it. In every one, both are at least twice the human rate with every interval above 1, in both registers. In the local models, two of the 12 model-and-register cells have intervals that include 1: Muse’s “rather than” on Stack Exchange (1.72, 0.91 to 3.22) and Gemma’s lists of three on Stack Exchange (1.31, 0.85 to 2.01). Lists of three carry a caveat: the parser-based counter failed the human check. After the review, we confirmed 75% of its hits in human replies and all of them in model replies, a gap over the 0.2 we allowed. Corrected for its errors, the ratio stays above 2 (3.04 over seven models), so by the rule set beforehand it is listed. The correction was measured on local-model text and assumes the counter errs the same way on the hosted models. On the first pass, before the review, the corrected three-model ratio was 1.75; the review mostly moved our codes toward the counter’s, which is what reviewing only disagreements tends to do. It is the less certain of the two.
| Tell | Human | Qwen | Gemma | Muse | gpt-6.1-sol | gpt-6-luna | Sonnet 5.5 | Opus 5.5 |
|---|---|---|---|---|---|---|---|---|
| “Rather than” | 3.2% | 17.0% | 10.3% | 7.0% | 26.0% | 15.8% | 14.3% | 12.5% |
| Lists of three | 9.8% | 21.3% | 12.5% | 39.3% | 37.7% | 40.7% | 29.8% | 24.9% |
| Em dashes | 2.5% | 5.3% | 27.3% | 6.7% | 15.7% | 45.7% | 0.3% | 0.3% |
| Flagged words | 2.2% | 22.5% | 25.2% | 4.3% | 2.8% | 3.0% | 2.8% | 2.8% |
Two tells that depend on the model. Em dashes run from 0.3% of replies in both Claude models (2 of 600 Sonnet replies, 2 of 599 Opus replies) to 45.7% in gpt-6-luna; the human share is 2.5%, so for Claude the tell is reversed. The flagged words are a habit of Qwen and Gemma, in about a quarter of their replies. The other five models are within about two percentage points of the human rate (2.2%). Both hold up on the pooled average, and the average describes no model in particular. The 2026 words (“quietly”, “load-bearing”) are more common in every model, but rare on both sides: 1.3% of human replies, 2.7% to 7.2% of model replies.
How useful the tells are as flags. Even the strongest tell is absent from most model replies. “Rather than” appears in 3.2% of human replies and 14.7% of model replies on average: seeing it multiplies the odds that a reply is model-written by 4.6 (LR+, 95% CI 3.2 to 8.0), and 85% of model replies don’t have it. Lists of three misfire more often, in about one human reply in ten. In the three local models, a classifier given all the counted tells picks the model reply out of a human-model pair 84% of the time (AUC 0.84, 0.83 to 0.86). A classifier on character sequences, trained and tested on the same three models, reaches an AUC of 0.995. That figure is a ceiling for this corpus: the classifier can learn each model’s quirks and leftover formatting, and would not work as a detector elsewhere.
Doubtful on measurement. Trailing “-ing” clauses met the bar on our counter (2.26 times the human rate) and failed the human check: in human replies the counter missed more than half of the clauses we found. Corrected, the ratio falls to 0.97, so by our rule the tell is not listed as holding up. That does not make it false. The correction rests on a few human replies, and counting only the counter’s false hits would move the ratio up. The clause is also hard to pin down: our two passes differed on 11 of 44 human replies.
Reversed. Seven tells were more common in the human replies. Staccato fragments were in 4.2% of human replies and 1.6% of model replies; “Note that” and “it’s worth noting” in 13 of 600 human replies and 23 of 4,199 model replies; self-answered questions in 5 human and 3 model replies. “It’s not X, it’s Y”, the tell most often named online in our review of the claims, came out at 0.63. It is in 6.7% of human replies. Per model the share runs from 1.8% (gpt-6.1-sol) to 14.2% (Qwen), and Qwen used it twice as often as humans on Hacker News while the four hosted models used it a quarter to half as often. Our counter also misses more of the models’ contrast frames than of the humans’, at least in the local models (checked against Haiku: 62% of model frames caught, 87% of human ones). The true ratio is therefore somewhat higher, but not near 2. On Stack Exchange, lists and bold are reversed on the pooled average and split by model: Qwen, Muse and gpt-6-luna wrote almost no formatting when asked for “only the text of the reply”, while the two Claude models put bold in 30% and 50% of their answers, against 17% for humans.
Where the habits come from. In a side test, Llama 3.1 8B continued 125 human answers. The instruction-tuned version used “rather than” about 6 times as often as the base model it was trained from (95% CI 2.1 to 37) and trailing “-ing” clauses about 6 times as often (2.7 to 15), and its sentence lengths were far more uniform. The base model used fewer trailing “-ing” clauses than the humans did (0.30, 0.13 to 0.68). After the results, we found that 27 of the 125 answers were longer than the 600 words the sampling rules allowed (deviation 19 in the pre-registration). Without them, Instruct still used “rather than” 7.6 times as often as the base model (2.3 to 33) and trailing “-ing” clauses 6.9 times as often (2.5 to 19), but the base model’s shortfall against the humans disappears (0.70, 0.24 to 2.0). For this pair of models, the habits appear with instruction tuning, though the two were prompted differently: the base model continued the raw text, and the instruct model was asked in a chat to continue it. In the same test, the instruct model used “it’s worth noting” in 31% of its continuations and summary closers 6.4 times as often as the human answers, the opposite of the 2026 models in the main test.
Discussion
Two claimed tells survive every model we tried, and both are about sentence shape: “rather than” to contrast an option with the one chosen, and lists of exactly three items. The nominalizations Reinhart et al. 2025 found in GPT-4o and Llama 3 also appear here as a model habit, real but weak. Their participial clauses are doubtful on measurement here: our counter for them failed its human check. Our base-versus-instruct test agrees with theirs for one pair of models: in Llama 3.1 8B, the habits appear with instruction tuning.
The tells that get the most attention online are the least general. The flagged vocabulary matches the rise in PubMed abstracts Kobak et al. 2025 for two of seven models. The rest use those words at about the human rate. Use of “delve” in arXiv abstracts fell once the word was named Geng and Trotta 2025; whether model makers moved away from it too, this data can’t show. Em dashes are the sharpest split: common in OpenAI’s models and Gemma, absent from Claude’s. We can’t say why from this data. A tell that one company’s models lack is not a property of LLMs, and any list of tells will go out of date as models change.
For a reader judging a single text, this means no tell is proof. The best one here is missing from 85% of model replies and present in 3% of human ones, and two habits on the circulating lists, short punchy fragments and “note that”, are more common in human writing. This fits the finding that people who rarely use these tools judge at about chance Jakesch et al. 2023 Russell et al. 2025, and suggests that frequent users who do well Russell et al. 2025 rely on more than the named tells: the counted tells recover part of what a character-level classifier finds. The rest may be habits no list names, or quirks of these three models that would not carry over.
Limitations
- Seven models under one bare prompt with no system prompt: three local models at 4-bit, which may itself change style, and four hosted models through their APIs. The study did not test ChatGPT, Claude or Gemini in their apps, which add system prompts, and results may not transfer to them. Each model wrote one reply per prompt at default sampling; the Claude models could not be seeded.
- Two reply registers, forum replies and answers to questions; essays, news and fiction were not tested. Neither register is edited prose. The tells that held up may partly mark formal writing, and some reversed ones may be forum habits. We did not test them against edited human writing. The prompt asked for “only the text of the reply”, which plausibly suppressed formatting, so the formatting reversals may not hold in chat products. The models were given a target length the human writers never had, and the human replies were selected for a score of at least 1 and 40 or more words.
- The human writers are Hacker News commenters and upvoted Stack Exchange answerers from 2019 to 2022, presumably mostly native English speakers; we did not measure this. We could not test whether these tells flag non-native writers, for lack of a free, licensed source of such text.
- How far the counts were checked: “rather than”, em dashes and the flagged words are exact text matches. Lists of three and “-ing” clauses were hand-checked by one coder who knew the hypotheses (80 replies, reviewed once) on local-model text only. The pre-registered check by two model coders failed because one coder could not do the task, so the minor grammatical tells (fragments, closers, self-answered questions) are checked only against Claude Haiku.
- Rare tells were underpowered: for tells in 1% to 3% of human replies, 300 replies per register cannot rule out a doubling, so those results are inconclusive: they show no effect was detected, and cannot show there is none.
- Models were added after the first results. Each addition was registered before its text was generated, and the decision to pool them was made after the OpenAI results were in. The three-model run is kept beside the seven-model one.
- The human text is from 2019 to 2022 and the models are from 2026: language change and source are not separated. A registered check on 200 Hacker News replies from 2025 and 2026 found no tell more common than in the 2019 to 2022 replies, but the recent replies are shorter and Hacker News removes many flagged LLM comments, so recent human text could not serve as a cleaner comparison.
- An early Claude arm, 97 replies written through Claude Code with our house style in context, was replaced by the API arm and has no verdicts; its counts are in the release. Its “rather than” ratio (4.8) is close to the API arm’s, and it had no em dashes, which is also what a model told to avoid them would produce.
- The Claude models and two of the coders share a vendor, and so do the study’s drafting tools. We reviewed our own codes with Claude Fable’s codes visible.
What’s next
What the named tells miss. In the three local models, a classifier given every counted tell picks the model reply out of a pair 84% of the time; one given only character sequences does so 99.5% of the time. That gap is the largest open question here. Part of it is each model’s quirks and leftover formatting, which would not carry over to other models. Part of it may be something no list names and no counter here measures: how predictable each sentence is, how specific a reply is to the comment it answers, how closely it keeps to the question, or how evenly it hedges. Next we will define measures of this kind, register them, test each against the human replies on the same 800 prompts, and check how much of the gap they close when the classifier is trained on some models and tested on others.
Whether readers use the tells. We will ask people to judge matched human and model replies, with and without the tells that held up edited out, to test whether those tells change their judgement, and whether people who use these tools often rely on them more. This needs participants and an ethics review.
Whether the tells survive instructions. The models here wrote under one bare prompt. We will run the same prompts through the chat products, with their own system prompts, and with an instruction to write like a human, to test which tells persist under the instructions people use. Em dashes are almost absent from the Claude models, so one company’s models can lack a tell; this run tests whether a request can remove one too.
Other writing and other writers. Both registers here are replies. We will test essays and news, where the same tells are claimed, against human text from before 2022, and, once a licensed corpus is available, whether the tells flag second-language English writers more often than native ones.
The next generation. The tells change as models do: “delve” faded in arXiv abstracts once it was named. We will run the same 800 prompts on the next model generation, with the same counters and verdict rules, to test which tells persist.
Get the next finding by email.
Ouyang, F. (2026). We checked 23 things people say give away AI writing. Five held up. Mnemosyne Research Labs, MRL:2611.0002v1.
图表:BibTeX
@techreport{ouyang2026tells, author = {Ouyang, Flavia}, title = {We checked 23 things people say give away {AI} writing. Five held up.}, institution = {Mnemosyne Research Labs}, number = {MRL:2611.0002v1}, year = {2026}, month = nov, url = {https://mnemosyneresearch.com/findings/what-gives-away-ai-writing}}
图表:BibTeX, data and code
@software{ouyang2026tellsdata, author = {Ouyang, Flavia}, title = {What gives away {AI} writing: data and code}, publisher = {Zenodo}, version = {v1.0.0}, year = {2026}, month = nov, doi = {10.5281/zenodo.24286168}, url = {https://doi.org/10.5281/zenodo.24286168}}
Data and code
llm-writing-tells on GitHub (Apache 2.0), archived with DOI 10.5281/zenodo.24286168. Hacker News and Stack Exchange texts are released as post IDs and labels, so anyone can fetch the originals. The local models’ replies, all counts, the prompts and the code are included; the OpenAI and Claude replies are released as counts only, because their providers’ terms bar training on them and an open licence would allow it. The statistics for all seven models can be rerun from the released counts, but the counting itself only for the three local models.
Pre-registration
PREREGISTRATION.md, dated November 7, 2026, with 19 dated deviations.
Drafting and the analysis code (Claude). GLM-4.7-Flash, Claude Haiku and Claude Fable coded validation texts as described in the method. Every number was checked against the data by Flavia.
- ↑Benjamini, Y. and Hochberg, Y. (1995). Controlling the false discovery rate: a practical and powerful approach to multiple testing. Journal of the Royal Statistical Society, Series B 57(1), 289-300. doi.org/10.1111/j.2517-6161.1995.tb02031.x
- ↑abGeng, M. and Trotta, R. (2025). Human-LLM coevolution: evidence from academic writing. Findings of ACL 2025, 12689-12696. doi.org/10.18653/v1/2025.findings-acl.657
- ↑Holm, S. (1979). A simple sequentially rejective multiple test procedure. Scandinavian Journal of Statistics 6(2), 65-70. www.jstor.org/stable/4615733
- ↑abJakesch, M., Hancock, J. T. and Naaman, M. (2023). Human heuristics for AI-generated language are flawed. PNAS 120(11), e2208839120. doi.org/10.1073/pnas.2208839120
- ↑Juzek, T. S. and Ward, Z. B. (2025). Why does ChatGPT “delve” so much? Exploring the sources of lexical overrepresentation in large language models. COLING 2025, 6397-6411. aclanthology.org/2025.coling-main.426
- ↑abKobak, D., González-Márquez, R., Horvát, E.-Á. and Lause, J. (2025). Delving into LLM-assisted writing in biomedical publications through excess vocabulary. Science Advances 11(27). doi.org/10.1126/sciadv.adt3813
- ↑abReinhart, A., Markey, B., Laudenbach, M., Pantusen, K., Yurko, R., Weinberg, G. and Brown, D. W. (2025). Do LLMs write like humans? Variation in grammatical and rhetorical styles. PNAS 122(8). doi.org/10.1073/pnas.2422455122
- ↑abcRussell, J., Karpinska, M. and Iyyer, M. (2025). People who frequently use ChatGPT for writing tasks are accurate and robust detectors of AI-generated text. ACL 2025, 5342-5373. doi.org/10.18653/v1/2025.acl-long.267
- ↑Zou, G. (2004). A modified Poisson regression approach to prospective studies with binary data. American Journal of Epidemiology 159(7), 702-706. doi.org/10.1093/aje/kwh090