In a paper released 2017-06-12, researchers introduced the Transformer, a network architecture based solely on attention mechanisms that removes recurrence and convolution entirely. Using 8 parallel attention heads and a 512-dimensional model width, it scored 28.4 BLEU on WMT 2014 English-to-German translation, beating prior best results, including ensembles, by more than 2 BLEU.
What Did the 2017 Transformer Paper Propose?
Researchers introduced the Transformer, a network architecture based solely on attention mechanisms that removes recurrence and convolution entirelyCITE:E1. The paper, dated 2017-06-12, describes the design as a departure from the sequence-processing layers that had underpinned prior modelsCITE:E1. Google Research, presenting the same work on its blog on 2017-08-31, described the Transformer as a novel neural network architecture based on a self-attention mechanism that the team believed to be particularly well suited for language understandingCITE:E6.
How Does Self-Attention Change Sequence Modeling?
The Transformer applies self-attention at each step to directly model relationships between all words in a sentence regardless of their positionCITE:E7. This is the mechanism that lets the architecture dispense with recurrence and convolution, since it does not need to process tokens in sequential order or rely on local receptive fields to relate distant wordsCITE:E7CITE:E1.
How Is Multi-Head Attention Configured Inside the Transformer?
The Transformer configures 8 parallel attention heads together with a 512-dimensional width for every sub-layer and embedding outputCITE:E4CITE:E3. The 512-dimension setting was chosen specifically to facilitate the model's residual connections, so that all sub-layers and embedding layers produce outputs of the same dimensionCITE:E3. On top of that shared width, the model runs 8 parallel attention layers, or heads, in each attention layer of the networkCITE:E4. IBM's technical explanation of the mechanism describes this design as computing self-attention operations multiple times in parallel at each attention layer, which lets the model enjoy the efficiency of averaging while still accounting for multifaceted relationships between tokensCITE:E5.
| Parameter | Value | Evidence |
|---|
| Model width (d_model) | 512 | CITE:E3 |
| Parallel attention heads (h) | 8 | CITE:E4 |
| WMT 2014 En-De BLEU score | 28.4 | CITE:E2 |
| Improvement over prior best (incl. ensembles) | over 2 BLEU | CITE:E2 |
| arXiv publication date | 2017-06-12 | CITE:E1 |
| Google Research blog post date | 2017-08-31 | CITE:E6 |
How Did the Transformer Perform on WMT 2014 Translation Benchmarks?
The Transformer scored 28.4 BLEU on the WMT 2014 English-to-German translation task, beating the previous best results, including ensembles, by more than 2 BLEUCITE:E2. That result came from the same architecture described in the 2017-06-12 paper, which relies solely on attention mechanisms rather than the recurrent or convolutional layers used in earlier systemsCITE:E1CITE:E2.
What This Means
The pieces line up into a single account: a June 2017 paper proposed replacing recurrence and convolution with attention aloneCITE:E1, built that proposal around self-attention that relates every word in a sentence to every other word regardless of positionCITE:E7, implemented it with 8 parallel attention heads operating on a shared 512-dimensional widthCITE:E4CITE:E3, and reported a 28.4 BLEU score on WMT 2014 English-to-German translation that beat the prior best, including ensemble systems, by more than 2 BLEUCITE:E2. Google Research's own framing — that the mechanism is particularly well suited for language understandingCITE:E6 — is the stated rationale connecting the architectural choice to the benchmark result.
Author's Take・Nathan
The paper's bet was architectural economy: removing recurrent and convolutional layers entirely and replacing them with attention that relates every word to every other word regardless of position. The 512-dimension, 8-head configuration shows this was a deliberate, constrained design rather than an incidental one — the paper ties the 512-dimension choice directly to enabling residual connections across all sub-layers and embeddings. The clearest evidence the bet paid off is the WMT 2014 result: a 28.4 BLEU score that beat prior ensemble systems — models that typically combine multiple trained networks — by more than 2 BLEU, using a single architecture based solely on attention. The detail worth watching in any assessment of this design is whether the 8-head, 512-dimension pairing holds up as a stable ratio, since the paper presents it as the specific balance chosen between parallel representational capacity and per-head dimensionality, not an arbitrary setting.