Pangram AI-text detector misses 80% of rewrites from Meta's Muse-Glimmer
Summary
A new study from Tokyo Metropolitan University reveals that AI-text detectors, specifically Pangram, struggle significantly with rewrites from newer large language models (LLMs). The research highlights that Pangram was able to detect over 99% of rewrites from older models before a generation change, but only managed to catch 3.8% after it. This decline is notably severe with rewrites from Meta's Muse-Glimmer, where Pangram missed 79.8% of scientific abstracts while successfully flagging just 1 in 5,000 human abstracts. The findings underscore the critical need for re-testing detection tools after each major model update to ensure their effectiveness in maintaining the integrity of scientific publishing.
Analysis
Pangram: Pangram is an AI-text detection service designed to identify machine-generated writing. It is central to a recent academic study testing its ability to screen scientific abstracts rewritten by various large language models. The research shows its detection effectiveness can shift markedly with different LLM versions. Rohan Paul: Rohan Paul is an AI-focused commentator and researcher who posts on X as @rohanpaul_ai. He highlighted findings from the Tokyo Metropolitan University paper on social media. His post emphasizes variability in detector performance across LLM generations. Tokyo Metropolitan University: Tokyo Metropolitan University is a public research institution in Japan with active programs in computer science and artificial intelligence. Researchers affiliated with the university authored the paper examining how AI-text detectors respond to content from newer model releases. Their work underscores practical challenges for tools used in academic integrity screening. AI Detection Challenges: Performance of AI-text detectors can degrade substantially when evaluated against content produced by newer large language models. Scientific Publishing Integrity: Tools screening research papers for AI writing require re-testing after each major model update to remain reliable.
Categories
aitechmachine_learning