CLIP is a neural network OpenAI trained on 400 million image-text pairs to predict which caption matches which image, using a dual-encoder architecture that projects visual and text features into a shared vector space. Once pretrained, natural language is used to reference the visual concepts it learned, enabling zero-shot transfer to new tasks. CLIP matches the original ResNet-50's ImageNet accuracy without using any of that model's 1.28 million training examples, and was benchmarked across more than 30 computer vision datasets spanning OCR, action recognition, geo-localization, and fine-grained object classification.
How does CLIP's dual-encoder design map images and text into a shared vector space?
CLIP uses an image encoder and a text encoder to produce visual features and text features, then projects both into a latent space with the same number of dimensionsCITE:E6. Their dot product yields a similarity score, which is how the model compares a given image against candidate text descriptionsCITE:E6. OpenAI (OpenAI) describes CLIP (Contrastive Language-Image Pre-Training) as a neural network trained on a variety of image-text pairs that can be instructed in natural language to predict the most relevant text snippet for a given image, without directly optimizing for that specific task — a capability OpenAI compares to the zero-shot behavior of GPT-2 and GPT-3CITE:E1.
What pretraining task and dataset scale did OpenAI use to train CLIP?
OpenAI trained CLIP on the task of predicting which caption goes with which image, using a dataset of 400 million image-text pairs collected from the internetCITE:E2. OpenAI states this simple pretraining task is an efficient and scalable way to learn state-of-the-art image representations from scratch, without relying on a pre-built labeled datasetCITE:E2.
How does CLIP achieve zero-shot transfer to new tasks after pretraining?
After pretraining, CLIP uses natural language to reference the visual concepts it has already learned, or to describe new ones, which enables zero-shot transfer of the model to downstream tasksCITE:E3. This means the model can be pointed at a task through a text instruction rather than through additional task-specific trainingCITE:E3.
How does CLIP perform on standard benchmarks and across different domains in zero-shot settings?
CLIP matches the accuracy of the original ResNet-50 on ImageNet in a zero-shot setting, without needing any of the 1.28 million training examples that ResNet-50 was trained onCITE:E4. OpenAI benchmarked this approach on more than 30 existing computer vision datasets, spanning tasks such as OCR, action recognition in videos, geo-localization, and many types of fine-grained object classificationCITE:E5.
| Metric | Value |
|---|
| Image-text pairs used to pretrain CLIP | 400 millionCITE:E2 |
| ResNet-50 training examples not used for CLIP's zero-shot ImageNet match | 1.28 millionCITE:E4 |
| Computer vision datasets used to benchmark CLIP | 30+CITE:E5 |
Taken together, the evidence traces a single line of design logic: a dual-encoder architecture that projects images and text into a shared vector spaceCITE:E6 is trained on 400 million image-text pairs using a caption-matching objectiveCITE:E2, which then lets natural language stand in for task-specific labels during zero-shot transferCITE:E3. The result reported by OpenAI is a model that reaches ResNet-50's ImageNet accuracy without its 1.28 million labeled examplesCITE:E4, and that same zero-shot approach was tested across more than 30 datasets covering OCR, action recognition, geo-localization, and fine-grained classificationCITE:E5.
FAQ
How does CLIP's dual-encoder design map images and text into a shared vector space?
CLIP uses an image encoder and a text encoder to produce visual features and text features, then projects both into a latent space with the same number of dimen…
What pretraining task and dataset scale did OpenAI use to train CLIP?
OpenAI trained CLIP on the task of predicting which caption goes with which image, using a dataset of 400 million image-text pairs collected from the internetCI…
How does CLIP achieve zero-shot transfer to new tasks after pretraining?
After pretraining, CLIP uses natural language to reference the visual concepts it has already learned, or to describe new ones, which enables zero-shot transfer…
How does CLIP perform on standard benchmarks and across different domains in zero-shot settings?
CLIP matches the accuracy of the original ResNet-50 on ImageNet in a zero-shot setting, without needing any of the 1.
Author's Take・EffectStory 編輯部
The technically notable point in this evidence is that CLIP's zero-shot match to ResNet-50 on ImageNet required none of the 1.28 million labeled training examples ResNet-50 depended on — the value is shifted from task-specific labeled data toward the scale of the 400-million-pair pretraining set and the shared image-text embedding space that ties visual features to natural-language descriptions. Because the caption-matching objective and the dual-encoder projection are what let a single natural-language instruction substitute for a labeled dataset, the architecture choice matters as much as the data volume. Given that OpenAI's own benchmark spans over 30 datasets across OCR, action recognition, geo-localization, and fine-grained classification, the metric worth watching next is whether this zero-shot transfer holds consistently across that full spread of domains, rather than judging the approach on any single dataset result.