Back to articles

How VLMs Access Concepts: Probing a Global Workspace Analogy

3 min read

A vision-language model can answer a question about an image, but its final response does not reveal how it represents the image’s concepts internally—or whether it selected the relevant information for that particular task. A new study adapts ideas from global workspace theory (GWT) to investigate these questions inside multimodal models. Its method, the Multimodal Concept Lens (MC-Lens), aims to trace concept accessibility, task-dependent selection, and the effects of accessed concepts on later computation. “Conscious” in this context is a theoretical analogy, not a claim that the models have subjective experience.

A shared space for vision and language

GWT describes information from parallel processes being selected, integrated, and made available to specialised processes. The researchers turn this broad idea into testable questions: Is a concept represented in the model? Does the model access it in response to a task? Does that information affect what the model computes next?

MC-Lens is a multimodal Jacobian lens. It reads intermediate states from the vision encoder and language decoder, then maps both into a shared semantic space defined by the decoder’s output vocabulary. This lets the researchers compare concepts across modalities in a common coordinate system, rather than relying only on the model’s final answer. They also expand “readout carriers”—the tokens used to capture and express a concept—from a single name token to a token set. This is intended to better represent concept information and enables counterfactual interventions.

Accessibility is not the same as use

The paper reports several findings:

  • Multimodal concept information can be read from intermediate states, and access is selective: which concepts are used depends on the task. A concept being present in a representation does not mean it will be used every time.
  • Counterfactual experiments indicate that changing accessed concept information can affect downstream computation and reasoning. This supports the view that such information participates in processing, without fully specifying the model’s reasoning mechanism.
  • During controlled training, the authors introduce two J-space metrics for tracking cross-modal concept alignment. They report that these metrics make alignment development clearer than cosine similarity between hidden states. The experiments also suggest that concept accessibility develops before reliable selective access.
  • Across VLM families and model sizes, the researchers find shared patterns of concept accessibility, query-dependent selection, and cross-task intervention effects, despite differences in internal organisation.

What the results do—and do not—show

The study’s central contribution is to distinguish what a model can represent from what it actually draws on for a particular task. Its shared semantic space and intervention method offer tools for inspecting information flow that cannot be inferred from text-image similarity scores or final answers alone. The proposed metrics may also help researchers study cross-modal alignment, although the supplied material does not show that optimising for them necessarily improves task performance.

There is an important boundary to the interpretation. GWT comes from the study of consciousness, but using it to analyse artificial systems does not amount to detecting consciousness. The evidence here concerns readable concept representations, task-dependent access, and downstream computational effects. It provides a way to investigate these functions in multimodal models, not proof of subjective experience or a complete account of how such models reason.

Source: Hugging Face Daily Papers

Comments

Checking sign-in status...

Loading comments...

Related articles

CCTest · Blog
HiUni Explores a Shared Backbone for Visual Understanding and Generation
Multimodal
cctest.ai
Multimodal

HiUni Explores a Shared Backbone for Visual Understanding and Generation

HiUni runs visual understanding, image generation, and editing through a shared vision encoder and dense Transformer backbone, reserving separate components for the output stage. A pixel diffusion head and a lightweight spatial pathway address challenges in image optimization and detail recovery.

Read more