Show HN: Vantage-V-AI – 2-bit ternary KV-cache with attention sinks

Vantage V-AI: Bounded-Memory Streaming Engine, 2-Bit Packed Ternary KV-Cache & HDC Associative Memory

Vantage V-AI (v8.2.0) is an open-source, bounded-memory inference runtime and streaming data processing engine designed to solve KV-cache memory explosion and enable constant-complexity autoregressive generation across CPU, edge devices, and multi-GPU accelerators.

• 📄 Technical Report (Peer-Review Edition): paper/VANTAGE_Paper_v8.2.pdf | LaTeX Source

• 🔬 Task Quality Benchmark: artifacts/benchmarks/task_quality_benchmark.json

• ⚡ Multi-GPU E2E Benchmark: artifacts/benchmarks/benchmark_gpu_kaggle_v8_2.json

🛡️ Authorship, Prior Art & Intellectual Property Notice

• Primary Author & Architect: Nguyen Vo Anh Khoa (@khoadev-vn)

• Official Repository: https://github.com/khoadev-vn/VANTAGE-V-AI

• First Public Disclosure (Prior Art): November 2026 (Git Commit History & Cryptographic Timestamps)

• Legal Protection: Licensed under the MIT License. Under international copyright law and Open Source compliance standards, any distribution, redistribution, embedding, or commercial utilization of this codebase or its derivative works strictly requires preservation of the original copyright notice and authorship attribution: Copyright (c) 2026 Nguyen Vo Anh Khoa. All rights reserved.

        Copyright (c) 2026 Nguyen Vo Anh Khoa. All rights reserved.
      

⚡ Quickstart: 2-Line Drop-in Integration

Vantage integrates transparently with any Hugging Face AutoModelForCausalLM without modifying existing generation loops or model definitions:

        import vantage
from transformers import AutoModelForCausalLM, AutoTokenizer

# Load any open-weight model (Meta Llama-3, Qwen-2.5, DeepSeek-R1, Mistral)
model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-3B-Instruct", torch_dtype="auto", device_map="auto")
tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-3B-Instruct")

# 1-Line Drop-in: bounds active attention window and anchors 4 sink tokens
vantage.enable(model, max_active_tokens=256, sink_tokens=4, quant_mode="fp8")

# Standard generation code runs unchanged with bounded attention memory!
inputs = tokenizer("The future of artificial intelligence depends on", return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=64)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
      

🖥️ Interactive Terminal Demo

Run the visual streaming monitor demo in your terminal:

        python vantage_demo.py
      

🔬 Core Architectural Foundations (v8.2.0)

• Empirical Multi-GPU Hardware Validation (2x NVIDIA Tesla T4, Qwen2.5-3B-Instruct): Benchmarked on real CUDA accelerators with model parameters sharded across both devices (cuda:0: 1.85B params, cuda:1: 1.23B params). Strict multi-device synchronization via torch.cuda.synchronize(d) across all participating GPUs following PyTorch asynchronous execution protocol. 100.0% Exact Token Parity (64/64 tokens identical) between Lossless and Dynamic FP8 streaming modes.

• Benchmarked on real CUDA accelerators with model parameters sharded across both devices (cuda:0: 1.85B params, cuda:1: 1.23B params).

• Strict multi-device synchronization via torch.cuda.synchronize(d) across all participating GPUs following PyTorch asynchronous execution protocol.

• 100.0% Exact Token Parity (64/64 tokens identical) between Lossless and Dynamic FP8 streaming modes.

• Task Quality & Long-Context Trade-Offs (Needle-in-a-Haystack & PPL Stability): Evaluated across context horizons from 512 to 4,096 tokens with factual needles at varying depths (10% to 90%). 0.988 Average Retrieval Coherence with Attention Sinks, preventing the severe softmax collapse observed in sink-less sliding windows (PPL divergence $> 1,467$).

• Evaluated across context horizons from 512 to 4,096 tokens with factual needles at varying depths (10% to 90%).

• 0.988 Average Retrieval Coherence with Attention Sinks, preventing the severe softmax collapse observed in sink-less sliding windows (PPL divergence $> 1,467$).

• 2-Bit Packed Ternary Representation (Inspired by BitNet b1.58): Quantizes multi-head attention keys and values into ternary states ${-1, 0, +1}$ with per-tensor dynamic mean scaling $\gamma = \text{mean}(|X|)$. Employs vectorized SIMD packing (4 elements per uint8 byte = 2.0 bits/element), achieving 7.97x storage reduction over FP16.

• Quantizes multi-head attention keys and values into ternary states ${-1, 0, +1}$ with per-tensor dynamic mean scaling $\gamma = \text{mean}(|X|)$.

• Employs vectorized SIMD packing (4 elements per uint8 byte = 2.0 bits/element), achieving 7.97x storage reduction over FP16.

• Streaming KV-Cache with Attention Sinks (Inspired by StreamingLLM): Integrates initial attention sink retention (sink_tokens=4) within a block-level chunked offloading cache (VantageStreamingCache). Anchors initial softmax mass, preventing autoregressive trajectory collapse during long-sequence generation.

• Integrates initial attention sink retention (sink_tokens=4) within a block-level chunked offloading cache (VantageStreamingCache).

• Anchors initial softmax mass, preventing autoregressive trajectory collapse during long-sequence generation.

• Hyperdimensional Computing (HDC) Associative Memory (core/memory.py): Projects dense embeddings into 4,096-bit binary hypervectors. Hardware-accelerated bitwise XOR + SIMD popcount search (np.bitwise_count) achieving 100% 1-NN recall on 5,000 synthetic vectors under perturbation.

• Projects dense embeddings into 4,096-bit binary hypervectors.

• Hardware-accelerated bitwise XOR + SIMD popcount search (np.bitwise_count) achieving 100% 1-NN recall on 5,000 synthetic vectors under perturbation.

📊 Empirical Multi-GPU Benchmark Results (v8.2.0)

Tested on 2x NVIDIA Tesla T4 GPUs (Turing, Compute Capability 7.5, 32 GB Total VRAM) using Qwen/Qwen2.5-3B-Instruct (FP16, Prompt: 1024 tokens, Generation: 64 tokens, Multi-Device Synchronized):

Raw verifiable JSON artifact: artifacts/benchmarks/benchmark_gpu_kaggle_v8_2.json.

🎯 Task Quality & Perplexity Stability (Long Context)

Evaluated across varying context lengths ($N \in {512, 1024, 2048, 4096}$) to characterize the context trade-off:

Raw Task Quality artifact: artifacts/benchmarks/task_quality_benchmark.json.

📚 Academic Citation (BibTeX)

        @software{nguyen2026vantage,
  author       = {Nguyen, Vo Anh Khoa},
  title        = {{VANTAGE-V-AI: Bounded-Memory Streaming Engine, 2-Bit Packed Ternary KV-Cache and Associative Memory Runtime for Large Language Models}},
  year         = {2026},
  publisher    = {GitHub},
  journal      = {GitHub repository},
  howpublished = {\url{https://github.com/khoadev-vn/VANTAGE-V-AI}},
  version      = {8.2.0}
}
      

📄 License & Compliance

This project is licensed under the MIT License — see the LICENSE file for details.

        Copyright (c) 2026 Nguyen Vo Anh Khoa. All rights reserved.