Can 96.46% AI-text detection travel?
A peer-reviewed benchmark separates human, AI-generated and AI-rephrased reviews and posts with high in-distribution accuracy. But the test reuses the same three generator families and two domains as training, while acknowledged overlap and corpus inconsistencies counsel against real-world verdicts.
Editorial responsibility: The Impact of AI Editorial Desk · Report a factual concern
At a glance
- 1The public corpus contains 161,788 English samples: 46,844 human-written, 57,247 AI-generated and 57,697 AI-rephrased Amazon reviews and Twitter posts.
- 2A stacking ensemble reported 96.46% accuracy on a 32,359-sample test split, ahead of RoBERTa at 95.49% and DeBERTa-v3 at 94.88%.
- 3Training and testing used the same three generator families—GPT/ChatGPT, DeepSeek and Kimi—and the same two domains. Unseen generators, other genres, languages and human–AI co-writing were not tested.
Research topic
Whether hybrid and transformer classifiers can distinguish human-written, AI-generated and AI-rephrased English text, and how far their in-distribution benchmark results can generalise
The benchmark asks a harder three-way question
Most automated text-origin studies reduce authorship to two labels: human or AI. Najma Sadia and colleagues instead created a third class for AI-rephrased material—human text whose wording and syntax were altered by a model while its meaning was preserved. That distinction matters for journalism, education and platform moderation because assisted editing is common and may sit between fully human drafting and fully synthetic generation.
The deposited English-language corpus contains 161,788 samples: 46,844 human-written, 57,247 AI-generated and 57,697 AI-rephrased. Human material came from Amazon product reviews and Twitter posts. GPT/ChatGPT, DeepSeek and Kimi generated new reviews and posts from zero-shot instructions, then rephrased human originals from three few-shot demonstrations. The authors retained texts from 10 to 512 words, cleaned and normalised them, and divided them into 113,249 training, 16,180 validation and 32,359 test samples. This is a sizeable, reproducible semi-synthetic benchmark, not a representative sample of everything people publish.[1][2]
The Impact Brief · Free
Follow the evidence in society & media.
Get a five-minute weekday briefing on what changed, why it matters and where the evidence comes from. Choose the topics you care about.
Transformers and stacking score highly inside the benchmark
For classical models, the researchers built a 168-dimensional representation: 68 stylistic, lexical, linguistic, syntactic and stylometric features plus 100 dimensions from RoBERTa and MiniLM embeddings. Logistic regression and a linear support-vector machine each reached about 74% accuracy, while random forest reached roughly 80%. End-to-end transformer fine-tuning was much stronger: RoBERTa-base reported 95.49% accuracy and a 95.50% weighted F1 score; DeBERTa-v3-base reported 94.88% accuracy and a 94.93% weighted F1.
Averaging transformer probabilities produced 95.66% accuracy. A stacking system then fed model outputs to several meta-learners, with random forest producing the headline 96.46% result. Its reported F1 score for the AI-rephrased class was 0.96. Ablations suggested handcrafted features added information beyond embeddings for the classical models. These are technically useful comparisons under a shared split, but the decimal precision should not be mistaken for confidence about a school essay, newsroom submission or future model that was never represented in the corpus.[1]
The test stays inside the same generators and domains
The central limitation is distribution shift. All three partitions draw from the same GPT/ChatGPT, DeepSeek and Kimi families and from the same Amazon-review and Twitter-post domains. The study did not hold out one generator family, train on one domain and test on the other, or evaluate Claude, Gemini, Llama, Mistral, essays, news, blogs, technical writing, multilingual text or code-switching. The authors explicitly describe the figures as in-distribution performance and identify unseen-generator evaluation as the primary next step.
That boundary is crucial for use. A detector may learn stable artefacts of the selected prompts, decoding settings, cleaning rules or platforms rather than a durable signature of machine authorship. AI-rephrased examples were made with one few-shot procedure—three demonstration pairs, temperature 0.5 and top-p 0.9—so the class does not cover every editing workflow or human–AI collaboration. A score produced outside this design is an extrapolation, not a validated probability that a person used AI.[1]
Acknowledged overlap and corpus corrections narrow confidence
A post-split inspection found 106 test texts, 0.33% of that split, also present in training, mostly short repeated Amazon reviews. The authors also found 18 cases in which an original and a synthetic derivative landed in different partitions. They call the contamination minor and unlikely to change the reported result materially, but group-level splitting before synthetic generation would have been the safer design. Even small leakage is relevant when performance differences between top models are below one percentage point.
The paper also corrects inaccurate corpus statistics from earlier manuscript versions. It says the deposited dataset contains 161,788 samples and notes that short human Amazon reviews were partly retained despite the stated ten-word filter. Yet the data-availability section separately describes a balanced corpus of 180,000 samples. One RoBERTa figure caption also says its displayed human row came from an earlier test split, although the summary metrics were calculated on the corrected split. Transparency about those discrepancies is welcome; it also means independent replication from the deposit and code is necessary before treating 96.46% as a stable benchmark.[1][2]
A benchmark score should not become an accusation
False authorship claims can affect grades, employment, reputation and access to publishing. This study did not test a disciplinary workflow, estimate false-positive rates for particular demographic or language groups, or examine what happens when people revise model output heavily. It does not justify using a detector as sole evidence of misconduct. Even a high average accuracy can hide consequential errors, and the relevant error rate will change as real-world class balance and writing style depart from the benchmark.
The constructive use is narrower. Researchers can use the open corpus to compare three-way methods, probe why rephrased text is difficult and design stronger out-of-distribution tests. Publishers or platforms considering detection should demand calibrated scores, transparent abstention, appeal routes and corroborating provenance evidence. Cooperative signals—document history, disclosed assistance or cryptographic provenance—answer a different question from stylistic classification and may be safer than inferring authorship from text alone.[1]
What would change the assessment
The decisive experiments are leave-one-generator-out and leave-one-domain-out evaluations with deduplicated, group-split data fixed before training. A prospective test should collect naturally occurring human, fully generated, lightly edited and heavily co-written material without revealing labels to developers, then report calibration and class-specific false positives across genres, languages and writer groups. Multiple training seeds, locked external test sets and comparisons with simple baselines would show whether the stacking gain is durable.
The work was supported by the Republic of Korea’s ANCHOR programme through the Jeju ANCHOR centre; the authors declared no competing interests and disclosed using ChatGPT to improve manuscript language and clarity. The present evidence supports the benchmark contribution and the conclusion that rephrased text can be separated well under matched conditions. It does not establish a universal detector, a reliable verdict on an individual document or generalisation to the next generator. Until those external tests arrive, the headline number belongs inside its test enclosure.[1][2]
What this means for people
- A three-way benchmark better reflects assisted editing than a simple human-versus-AI label.
- The reported score must not be used as sole evidence against a student, worker, journalist or author because external validity is unproven.
- Open data can support stronger testing, but operational safeguards and corroborating provenance remain necessary.
Global context
The research team spans Pakistan, the Republic of Korea and Türkiye, while the corpus is English and sourced from two global platforms. The generator families are internationally used, but the benchmark omits many languages, local writing conventions and emerging models. Its main global value is as an open test bed whose boundaries are explicit—not as proof that one detector travels across cultures, platforms and time.
What the evidence does not yet show
- All partitions use the same GPT/ChatGPT, DeepSeek and Kimi generator families; unseen-generator generalisation was not evaluated.
- Only English Amazon reviews and Twitter posts were included; other genres, languages and code-switching are untested.
- The AI-rephrased class reflects one few-shot prompting procedure, not the full range of AI-assisted editing or human–AI co-writing.
- The paper acknowledges 106 train–test duplicate texts and 18 original–derivative pairs split across partitions.
- Corpus counts were corrected from earlier versions, and the article contains a conflicting 180,000-sample description in its data-availability statement.
- Ensemble confidence intervals come from one split and do not account for variation across training seeds.
What to watch next
- Deduplicated leave-one-generator-out and leave-one-domain-out tests.
- Independent replication from the Zenodo deposit with reconciled counts and grouped splits.
- Prospective multilingual evaluation on naturally occurring human–AI co-writing.
- False-positive, calibration, fairness, abstention and appeal evidence before any high-stakes deployment.
Living evidence record
Impact record IAI-1ADG56Z
Evidence stage
Studied
Confidence
Supported
Reporting basis
Source analysis
Independent or research support
Present
Record status
Monitoring
Last checked
5 October 2026
Source trail
2 direct sources across 1 source type.
People impact
Documented in this record.
Uncertainty
Limits and next checks are explicit.
Stages describe the evidence available—not whether a technology is good or bad. See the public method.
Related-source reporting disclosure
This record analyses 2 linked source records around the same underlying development. The extra records add method, date or context, but they do not by themselves constitute independent replication of every performance claim or predicted outcome.
Evidence trail
Sources used for this report
Links checked 5 October 2026
This report is labelled source analysis. We summarise and analyse source material in our own words; company statements remain attributed claims until independently supported. Translated summaries preserve the meaning of the original source and link back to it. Read our editorial standards.
Continue the story
Related reporting
Society & Media
When should newsrooms disclose AI use?
A peer-reviewed case study based on 13 interviews with 12 Financial Times managers and 28 internal documents finds that AI disclosure is treated as a spectrum shaped by oversight, risk and context. It describes one newsroom's practice and does not test whether labels improve audience trust.
7 min · 1 source
Society & Media
Why was an AI review retracted?
New analysis today of a 3 October retraction notice. Springer Nature says contextually incorrect references may indicate undeclared generative-AI use in a 2024 review and that the authors did not provide a satisfactory explanation; the notice does not prove which tool was used, how much text it produced or a field-wide failure rate.
7 min · 2 sources
Society & Media
What should AI companies prove before giving conversational agents to children?
A new World Economic Forum white paper sets out child-focused practices spanning design, privacy, learning, harmful interactions and accountability. It is useful guidance, but not a binding standard or an outcome study, and its public summary reports no sample or systematic-review method.
6 min · 1 source
Reader commentary
Add evidence, experience or a question
No account is required. Reader notes are published after a brief civility, relevance and safety check; disagreement is welcome.
Explore commentary across the portal →Published reader notes
0No published reader notes yet. You can start the evidence-led discussion above.
Prefer a private correction or response? Contact the newsroom.