Newshot

Nvidia researchers argue huge AI context windows are a dead end

Summary

Nvidia researchers argue that language models often don't need their full context windows—which are slow, expensive, and prone to errors—and propose an alternative. In tests, periodic summaries and targeted keyword search usually matched full-context results at a fraction of the cost. The team even proposes a new class of models that works with short notes instead of long context. The article Nvidia researchers argue huge AI context windows are a dead end appeared first on The Decoder .

Original text(via The Decoder)

Nvidia researchers argue huge AI context windows are a dead end

Image description
Image description

Language models can now process entire novels at once. An Nvidia research team says that's often overkill and expensive. In many tests, periodic summaries and targeted lookups worked about as well.

A language model's context window is its working memory, covering everything it holds in view when generating a reply. Over the past few years, that window has grown rapidly. GPT-4 launched in March 2023 with 8,192 tokens. Today, GPT 5.6 Sol handles around one million tokens, about 120 times as much and enough for Tolstoy's "War and Peace."

THE DECODER+

Continue reading with THE DECODER+

  • Every Plus article and the full archive
  • No ad banners
  • Frontier Radar: 6 deep dives a year

Already a subscriber? Sign in Back to the homepage

The text above was extracted automatically from the source page and may miss images, tables or code blocks. The original is authoritative.

Published by The Decoder
This site only summarises and aggregates; copyright stays with the original author. Headlines and summaries follow the source — where they differ, the original wins.
Read the original ↗