Open Vision Banana Turns Image Generators into Open Vision Systems

Introduction

Image generators may encode more than the ability to synthesize visually plausible images. Their pretraining can also capture information about objects, scenes, spatial relationships, and visual semantics. Open Vision Banana (OVB) explores how to reuse those representations by turning pretrained generators into unified visual perception systems.

The project is also an open response to the limited reproducibility of Vision Banana. That earlier approach framed vision tasks as conditional image editing, but its data, weights, code, and implementation details were not public. OVB releases an end-to-end recipe covering data construction, training, evaluation, and model checkpoints.

Key points

• Perception as image editing. Instead of attaching a separate prediction head to each task, the system uses text conditions to transform an input image into a visual representation of the desired result. This brings semantic segmentation, depth estimation, referring segmentation, and reasoning segmentation into one generative framework.
• Design choices interact. The study examines color encoding, prompt design, parameter adaptation, and task balancing. Its main warning is that these choices cannot always be optimized independently: a setting that helps in isolation can become harmful when combined with other choices.
• An open training recipe. OVB starts from open-weight 4B and 9B parameter base models and uses roughly 2.1 million to 3.8 million training pairs assembled from public datasets. The release includes the data construction and processing pipeline, training and evaluation code, and final, intermediate, and ablation checkpoints.
• Strong zero-shot segmentation. On Cityscapes, OVB reaches 73.9 mIoU in the reported zero-shot setting, compared with 69.9 for Vision Banana. Its depth results are also competitive with state-of-the-art affine-invariant depth estimators.
• Uneven task performance. In non-zero-shot referring and reasoning segmentation, OVB comes within five points of Vision Banana. Instance segmentation remains a larger gap. At the same time, the adapted models largely retain the image-editing capabilities of their base generators.

Test-time scaling with legend ensembling

OVB introduces a practical method called legend ensembling. In segmentation, the model must associate semantic classes with predefined colors, but those bindings can be unstable in a generative setting. The method runs inference with several class-color legends and combines the resulting masks by majority vote.

This approach corrects some inconsistent class-color assignments without additional training. The reported gains reach up to 8.7 mIoU points on Cityscapes and 15.0 points on COCO-Object. It is a useful example of test-time scaling applied to generator-based perception: repeated, differently encoded predictions can provide a more reliable final output.

Why it matters

OVB’s contribution is not merely another model checkpoint. It turns generative perception into a set of explicit and testable engineering decisions. The results suggest that success depends not only on parameter count, but also on how outputs are encoded, how prompts are written, and how multiple tasks are balanced during training.

The open release lowers the barrier for replication and follow-up work. Legend ensembling further shows that some instability may be handled at inference time rather than through another training cycle. Still, the remaining weakness in instance segmentation indicates that a unified generative framework is not yet a universal replacement for specialized vision models.

Source: Hugging Face Daily Papers