New BabyVision Benchmark Reveals Top AI Models Lag Three-Year-Olds in Core Visual Tasks

Leading large language models (LLMs) and vision-language models (VLMs) can excel at complex coding and mathematical problems but struggle with basic visual tasks that even young children master, according to a new benchmark. Simple challenges like tracing lines to connect objects or counting blocks, which humans intuitively understand, expose a fundamental limitation in AI's ability to "describe" visual information using language. This raises questions about the true depth of AI's "understanding."

Over the past year, large models have demonstrated rapid advancements in language processing and textual reasoning, achieving new records in academic and competitive tasks. However, a critical question remains: can these models truly "understand" visual information when a problem cannot be fully articulated in language?

To assess the extent of visual comprehension in AI models, UniPat AI, in collaboration with Sequoia China's xbench team and researchers from various large model companies and universities, introduced BabyVision, a new multimodal understanding benchmark. UniPat AI focuses on developing new paradigms for AI training, evaluation, and real-world application, aiming to promote generalizable and trustworthy AI deployment.

If a visual problem can be entirely described in text without information loss, it essentially becomes a text-based problem. Models can then leverage powerful language reasoning, creating an illusion of visual proficiency. True visual ability, however, requires completing tasks without linguistic aid, encompassing comparison, tracking, spatial imagination, and pattern induction.

Split image showing complex code on one side and a simple child's drawing on the other.

Demis Hassabis, founder of Google DeepMind, has previously noted a similar paradox, observing that large models can win International Mathematical Olympiad gold medals but make errors in elementary geometry, or generate stunning images without understanding basic physics.

UniPat AI and xbench launched BabyVision to address these critical questions, anticipating a new wave of breakthroughs in world models and visual multimodal AI by 2026.

AI Models and Children Take the Same Visual Test

BabyVision conducted a direct comparative experiment using 20 vision-centric tasks from BabyVision-Mini. These tasks were administered to children across different age groups (3, 6, 10, and 12 years old) and to leading multimodal models. The "mini-test" was designed to minimize language dependence, featuring simple questions where answers had to be derived solely from visual input.

The evaluation results indicated that most models performed significantly below the average 3-year-old child in tasks requiring "understanding the world." Gemini-3-Pro-Preview was the only model to consistently surpass the 3-year-old baseline, though it still lagged approximately 20 percentage points behind 6-year-old children.

Child's hand drawing a clear line, while a robotic arm draws a broken, incorrect line.

For instance, one task involved connecting items to the correct colored trash cans for waste sorting. While children easily performed this, top models struggled even to track a single line. In an example provided, the correct answer for connecting three items to their respective trash cans was A-Blue, B-Yellow, C-Green. Gemini3-Pro-Preview, however, responded with A-Green, B-Yellow, C-Blue, indicating a failure in basic visual tracking despite attempting a lengthy "segment-by-segment tracking" reasoning.

Deconstructing Visual Abilities with BabyVision-Full

The research team categorized visual abilities into four core areas, each with several sub-tasks:

• Fine-grained Discrimination: Identifying subtle visual differences (8 sub-tasks).

• Visual Tracking: Following paths, lines, and trajectories (5 sub-tasks).

• Spatial Perception: Understanding 3D structures and their relationships (5 sub-tasks).

• Visual Pattern Recognition: Identifying logical and geometric patterns (4 sub-tasks).

This design aims to quantify "atomic visual abilities" that are intuitive for humans but foundational for intelligence, providing essential lessons for the development of embodied AI.

To ensure a "pure vision" assessment, BabyVision's data construction process was rigorous. The project team referenced children's cognitive textbooks and visual development tests to compile the 4 major categories and 22 basic visual sub-tasks. For each sub-skill, 4–5 seed examples were selected, from which approximately 4,000 similar candidate images were gathered online through reverse image and keyword searches.

During data collection, strict adherence to copyright regulations was maintained, selecting only materials suitable for non-commercial or academic use. Images containing extensive text descriptions or requiring cultural common sense were filtered out. The collected images then underwent manual annotation, with multiple professionals checking, screening, and meticulously designing questions and standard answers. Each question included detailed "solution process" explanations to verify that the answer could be derived through visual reasoning.

Finally, all annotated questions underwent a "double-blind quality inspection" by two independent experts. Questions were only included if both experts agreed on the correctness of the answer and the rigor of the reasoning. Disagreements led to revision, and questions that repeatedly failed to reach consensus were discarded. This process resulted in 388 high-quality visual questions covering 22 sub-tasks.

Final Evaluation Results

On BabyVision-Full, a human baseline was established with 16 testers, all holding at least a bachelor's degree, who completed all 388 questions with an accuracy rate of 94.1%.

In contrast, AI models showed significantly lower performance:

• The strongest closed-source model, Gemini-3-Pro Preview, scored 49.7%.

• Among open-source models, the strongest (Qwen-3-VL-235B-Thinking) achieved an overall score of 22.2%, with most models ranging from 12% to 19%.

This performance gap was not confined to a single category but was observed across all four ability categories, indicating a "systemic lack of fundamental visual abilities." Some sub-tasks, such as "Count 3D Blocks," resulted in near-universal failure among models, highlighting their insufficient ability to structure scenes.

Human brain's neural pathways glowing brightly next to fragmented AI processing unit.

The research team attributes these failures to the "unspeakable" nature of many BabyVision questions. While humans, including children, can solve these tasks by pointing, circling, or tracing, models struggle when attempting to "restate" visual information in text and then apply language reasoning, leading to information loss.

The team identified four typical challenges:

• Observing Non-Verbal Details: In tasks like jigsaw puzzles, options may differ by minute details. Humans use geometric intuition to align boundaries, but models generalize shapes into language, smoothing out details and making options appear "almost identical" in token space.

• Losing Track of Lines (Manifold Understanding): For line-connecting tasks, humans follow a line through intersections. Models often translate lines into discrete steps, leading to branching explosions at intersections and "switching tracks."

• Lack of True Spatial Imagination: When counting 3D blocks or judging structures under occlusion, humans mentally construct and view structures from different angles. Models frequently miss hidden blocks or misinterpret projection relationships, indicating a lack of stable 3D internal representation.

• Difficulty in Visual Pattern Induction: These tasks require abstracting rules from visual examples. Humans perform relational mapping, focusing on "what changed" rather than "what is there." Models often concentrate on surface attributes, misinterpreting "structural rules" as "appearance statistics."

Child's hands solving a jigsaw puzzle, contrasting with a blurred, generalized digital puzzle.

BabyVision-Gen Offers a New Direction

Given the limitations of text reasoning, the question arises whether models can answer visually, similar to children drawing or tracing. This led to the creation of BabyVision-Gen, which re-annotated 280 questions for "generative answering." Models are required to output images or videos to express their solution process or answer. An automatic evaluation tool was developed, showing 96% consistency with human evaluation.

The research team evaluated various generative models, including Nano Banana Pro, Qwen-Image, Veo 3, and Sora 2, on BabyVision-Gen. While the conclusions are cautious, they suggest that generative reasoning exhibits "more human-like" behavior in tasks where VLMs typically fail, such as visual tracking and fine-grained discrimination, by drawing trajectories and making annotations. However, overall, these models still lack the stable ability to achieve completely correct solutions. This indicates that "grounding visual reasoning in visual operations" could be a promising approach to address current shortcomings.

The Importance of BabyVision

As the research team noted in their blog, "It's hard to imagine a robot with visual abilities inferior to a 3-year-old being able to reliably assist humans in the real physical world." While current multimodal models are powerful in language and writing, achieving true general intelligence and embodied intelligence requires strengthening the visual foundation: accurate seeing (fine-grained discrimination), reliable tracking (trajectory/connectivity), imagination (3D structure imagination), and induction (graphic rule transfer).

BabyVision's value lies in deconstructing "understanding the world" into 22 measurable, diagnosable, and iterative atomic abilities. This helps identify current gaps and guides the development of multimodal large models.

Open-source resources for BabyVision are available on the UniPat AI blog, GitHub, and Hugging Face. UniPat AI is dedicated to building new paradigms for AI training, evaluation, and application in real-world scenarios, promoting generalizable and trustworthy AI deployment.