Vision Transformer (ViT) splits an image into fixed-size patches, embeds them linearly as a token sequence, and feeds that sequence directly into a standard Transformer. Google Research reports that after large-scale pretraining, ViT matches or beats state-of-the-art CNNs on benchmarks while needing substantially fewer training resources — but it generalizes poorly when training data is insufficient.
How Does ViT Convert an Image Into a Sequence a Transformer Can Process?
Google Research's Vision Transformer splits an image into patches and provides the sequence of these patches' linear embeddings as the input to a Transformer CITE:E1. Image patches are treated the same way as tokens, or words, are treated in an NLP application CITE:E2. Position embeddings are then added to the patch embeddings to retain positional information that would otherwise be lost once the image is broken into a sequence CITE:E3.
What Performance Advantages Does ViT Show Compared to Traditional CNNs?
Google Research reports that when pre-trained on large amounts of data and transferred to multiple mid-sized or small image recognition benchmarks — including ImageNet, CIFAR-100, and VTAB — ViT attains excellent results compared to state-of-the-art convolutional networks while requiring substantially fewer computational resources to train CITE:E4. The same research shows that reliance on ConvNets is not necessary: a pure Transformer can perform very well on image classification tasks when applied directly to sequences of image patches CITE:E7.
Why Does ViT's Generalization Suffer When Training Data Is Insufficient?
Google Research states that Transformers lack some of the inductive biases inherent to CNNs, such as translation equivariance and locality, and therefore do not generalize well when trained on insufficient amounts of data CITE:E5. This is presented as the structural counterpart to the benchmark gains described above: the same architecture that performs well after large-scale pretraining depends on that scale precisely because it does not carry CNNs' built-in assumptions about images.
What Is a Typical Deployment Configuration for ViT?
Hugging Face documents google/vit-base-patch16-224 as the base-sized ViT architecture, with a patch resolution of 16x16 and a fine-tuning resolution of 224x224 CITE:E6.
| Spec | Value |
|---|
| Model | google/vit-base-patch16-224 |
| Architecture size | Base |
| Patch resolution | 16x16 |
| Fine-tuning resolution | 224x224 |
| Paper publication year | 2020 |
What This Means
The patch-as-token design with added position embeddings CITE:E1 CITE:E2 CITE:E3 is the mechanism that lets a Transformer process images at all, and it is also the source of the trade-off Google Research documents: strong benchmark results and lower training compute appear only after large-scale pretraining CITE:E4 CITE:E7, while the same lack of CNN-style inductive bias causes weak generalization when training data is insufficient CITE:E5. The concrete deployment spec that Hugging Face publishes — a 16x16 patch grid fine-tuned at 224x224 resolution CITE:E6 — sits inside that same trade-off: the patch size and resolution are fixed architecture choices, and their reported success is tied by Google Research to pretraining on large amounts of data rather than to the configuration alone.
Author's Take・Nathan
The notable design decision here is not the Transformer body but the front end: patches are treated exactly like NLP tokens, with only added position embeddings to recover spatial structure — a much thinner set of built-in assumptions than a CNN's translation equivariance and locality. That is precisely why Google Research ties ViT's strong benchmark results to large-scale pretraining, and its weak generalization to insufficient data — the two findings describe the same trade-off from opposite sides. The published google/vit-base-patch16-224 configuration, a 16x16 patch grid fine-tuned at 224x224 resolution, only makes sense read against that trade-off: a fixed patch/resolution spec on its own says nothing about whether the underlying pretraining data was large enough. The metric worth watching in any ViT deployment is therefore not the patch resolution in isolation, but whether it is reported alongside the scale of pretraining data used with it.