CAS Launches VTCBench to Evaluate Visual Language Models in Long Text Compression

DeepSeek-OCR's Vision-Text Compression (VTC) technology, which encodes text as visual tokens to achieve up to 10x compression, aims to reduce the cost for large models processing extensive documents. However, the ability of visual language models (VLMs) to comprehend highly dense information after such compression has been a subject of inquiry. To address this, a research team from the Institute of Automation, Chinese Academy of Sciences (CAS), and the Hong Kong Institute of Innovation, CAS, introduced VTCBench, a benchmark designed to assess the cognitive limits of models in visual space.

Visual representation of text compression, showing a document transforming into a dense 2D visual pattern.

The VTC paradigm, exemplified by DeepSeek-OCR, renders long documents into high-density 2D images. A visual encoder then converts these images into a reduced number of visual tokens, achieving a token compression ratio of 2x to 10x. This method significantly lowers the computational and memory overhead associated with processing long texts.

VTCBench: Measuring Understanding in Compressed Visuals

VTCBench is the first benchmark specifically developed for the vision-text compression paradigm. It is open-sourced on GitHub and Huggingface. Its derivative, VTCBench-Wild, provides a comprehensive evaluation of model robustness in real-world scenarios for visual text compression and is integrated into VLMevalkit.

The benchmark evaluates VLM cognitive limits through three primary tasks:

• VTC-Retrieval (Information Retrieval): This task assesses a model's ability to locate specific facts within a visually compressed document, testing its capacity to capture spatially distributed information.

• VTC-Reasoning (Relational Reasoning): This challenges models to perform relational reasoning with minimal text overlap, moving beyond simple lexical retrieval.

• VTC-Memory (Long-Term Memory): This simulates extended conversations to evaluate a model's resilience to temporal and structural information decay within a visual compression framework.

Abstract visualization of diverse document rendering configurations, symbolizing robustness testing for AI models.

Additionally, the team introduced VTCBench-Wild, which incorporates 99 different rendering configurations, including variations in fonts, font sizes, line heights, and backgrounds, to test model robustness in diverse real-world conditions.

Cognitive Bottlenecks and Performance Insights

Test results from VTCBench revealed a "U-shaped curve" in model performance. Similar to text-based models, VLMs demonstrated accuracy in capturing information at the beginning and end of documents. However, their understanding of facts located in the middle of a document declined significantly as the document length increased. This indicates that models exhibit a "spatial attention bias" even in visual space, highlighting an area for future VTC architecture optimization.

Abstract glowing U-shaped curve on a dark digital background, illustrating model performance decline in the middle of documents.

Evaluations of over 10 models, including GPT, Gemini, Claude, QwenVL, InternVL, Gemma, KimiVL, and Seed1.5, provided further insights:

• While VTC improves efficiency, existing VLMs generally performed weaker in complex reasoning and memory tasks compared to pure text large language models (LLMs).

• Ablation experiments confirmed that information density is a critical factor influencing model performance, directly affecting the accuracy of visual encoders.

• Gemini-3-Pro showed strong performance on VTCBench-Wild, with its visual understanding capabilities approaching its pure text baseline. This suggests that VTC is a viable approach for large-scale long text processing.

The VTC paradigm, led by DeepSeek-OCR, aims to enable models to process long texts efficiently. The introduction of VTCBench seeks to ensure that models can still comprehend nuanced meanings within compressed visual information.