Newshot

Gemini Decision 1 can't separate AI voices made after 2024

Original text(via HN AI)

Gemini Decision 1 is a multimodal decision model from Google released on November 11, 2026. It is analogous to Jev, the System One model from TypeSafe AI.

While browsing Hacker News, I stumbled upon What music does Gemini Decision 1 like? by Liam Warfield. He ends the article with a few ideas to extend his project. The last of these suggestions, “Does it [Gemini Decision 1] like or hate AI-generated music?”, stood out to me because it implicitly suggests that the decision model may be able to recognize AI-generated audio. As it turns out, Gemini Decision 1 can separate AI voices from human voices well above chance, but only if the AI voice generator was published before 2025.

As a quick clarification, I am saying separates rather than categorizes as the model has a strong bias towards labeling all tested audio as human-made. Out of 1,020 AI clips, only 20 were given a 50% or higher chance of being AI-generated. However, the ROC-AUCchance that a random AI clip is assigned a higher P(AI) than a random human clip was significantly higher in pre-2025 models, suggesting that recent AI voices are less distinguishable than before.

Results

Each point is an AI voice generator, measured by how well Gemini Decision 1 separates 30 of its audio clips from 200 human audiobook clips.

When pooled together, generators from 2023 and 2024 are separated with an AUC of 0.67. Six older generators, from 2017 to 2022, score the same (0.67). However, generators released in 2025 and 2026 have an AUC of only 0.52, effectively chance. All but one of the 2025-2026 generators have a 95% confidence interval that includes 0.5; the only exception, Inworld TTS-2, has a confidence interval of 0.51-0.75.

Each clip was scored with a Simple Noulbinary question with output between 0 and 1, Criteria NoulSimple Noul with provided criteria for context, and 5-Pt Rubricrescaled score from “Definitely authentic human recording” to “Definitely AI-generated or synthetic”.

Question format2017–20242025–2026
Simple Noul0.590.46
Criteria Noul0.670.52
5-Pt Rubric0.660.47

Notably, rather than gently sloping downward, we observe that AUC drops sharply at the end of 2024, or the start of 2025.

Inhuman performance?

The research group behind MLAAD performed a 2026 listening study with 35,532 real-or-fake guesses from 1,768 people. The generators that fooled people also tended to fool Gemini Decision 1 (r = +0.57), but they do not always align. Griffin-Lim, a reference signal-processing method introduced in 1984, was only separated with an AUC of 0.53 (near chance), despite humans recognizing it 72% of the time. Cartesia Sonic-3 was the hardest for people (32% correct) and sits right at chance for Gemini Decision 1 (AUC 0.51).

Humorously, Gemini Decision 1 only achieved an AUC of 0.58 for Gemini 3.1 Flash TTS, suggesting it is only slightly better than chance at recognizing “its own” voice. At this point, neither man nor machine can reliably distinguish recent AI voices.

Dataset and normalization

It’s easy to imagine a few scenarios where leaked metadata could have produced these results. As with all things in AI, I want to make sure I was measuring what I intended to measure.

The AI voice clips are sourced from MLAAD, a dataset of 143 English generators reading from the same set of audiobook sentences. For testing, I took 30 clips of 2.5-10 seconds from 34 generators. Each generator’s clips were tested against the same 200 human clips from 30 speakers in LibriTTS, a dataset of LibriVox audiobook recordings. All clips are clean audiobook narration.

Before sending the audio clips to the decision API, each clip was re-encoded with ffmpeg (-map_metadata -1 -vn -ac 1 -ar 24000 -t 15 -c:a libmp3lame -b:a 128k) to prevent metadata from affecting results. In addition, each clip was band-limited to 8 kHz (aresample=16000,aresample=24000), and loudness-normalized (loudnorm=I=-20:TP=-2:LRA=11).

All 34 generators

GeneratorReleasedTypeAUC95% CIPeople correct†
Griffin-Lim—Reference0.530.40–0.6672%
Tacotron 2Dec 2017Open0.640.52–0.7568%
Glow-TTSMay 2020Open0.800.70–0.8886%
FastPitchJun 2020Open0.620.50–0.7475%
VITSJun 2021Open0.690.58–0.7954%
SpeechT5Oct 2021Open0.720.60–0.8279%
TortoiseApr 2022Open0.550.43–0.6771%
BarkApr 2023Open0.660.54–0.7771%
MMS-TTSMay 2023Open0.690.57–0.7968%
Matcha-TTSSep 2023Open0.580.46–0.7175%
XTTS v2Nov 2023Open0.640.52–0.7568%
OpenAI TTS-1 HDNov 2023Commercial0.630.50–0.7557%
MetaVoice-1BFeb 2024Open0.790.68–0.8773%
OpenVoice V2Apr 2024Open0.760.66–0.8578%
Parler-TTS LargeAug 2024Open0.690.56–0.7958%
F5-TTSOct 2024Open0.690.58–0.7983%
KokoroDec 2024Open0.570.45–0.6857%
Llasa-3BJan 2025Open0.530.41–0.6556%
ZonosFeb 2025Open0.460.34–0.5861%
Spark-TTSFeb 2025Open0.500.37–0.6264%
Sesame CSMMar 2025Open0.500.38–0.6157%
DiaApr 2025Open0.460.34–0.5954%
Resemble AIApr 2025Commercial0.540.41–0.6779%
ChatterboxMay 2025Open0.490.36–0.6158%
ElevenLabs v3Jun 2025Commercial0.570.44–0.69—
IndexTTS-2Jun 2025Open0.550.43–0.6766%
VibeVoiceAug 2025Open0.530.40–0.6657%
Cartesia Sonic-3Oct 2025Commercial0.510.38–0.6332%
MiniMax Speech 2.8Jan 2026Commercial0.510.39–0.63—
Qwen3-TTSJan 2026Open0.510.39–0.6352%

AUC is Criteria Noul, 30 clips per generator against 200 LibriTTS clips; the interval resamples clips and LibriTTS speakers. †Share of correct guesses for that generator in the 2026 listening study (all clips).

Data

AI clips: MLAAD (CC BY-NC 4.0). Human clips: LibriTTS (CC BY 4.0). Human listening results: Human Audio Deepfake Perception 2026 (CC BY-NC 4.0).

The text above was extracted automatically from the source page and may miss images, tables or code blocks. The original is authoritative.

Published by HN AI
This site only summarises and aggregates; copyright stays with the original author. Headlines and summaries follow the source — where they differ, the original wins.
Read the original ↗