BERT: How Google's Bidirectional Transformer Rewired Language Model Pre-training

How Does BERT Achieve Deep Representations Through Bidirectional Pre-training?

BERT pre-trains deep bidirectional representations by jointly conditioning on left and right context in all layersCITE:E1. Google designed the model to build these representations from unlabeled text rather than from labeled task data, with the bidirectional conditioning applied across every layer of the network rather than only at the outputCITE:E1.

Google Research describes this as "deeply bidirectional": BERT represents a word such as "bank" using both its left and right context — as in "I made a ... deposit" — starting from the very bottom of the deep neural networkCITE:E3. This distinguishes the model from architectures that combine left-to-right and right-to-left passes only near the output layer.

The pre-training framework built on top of this bidirectionality is a bidirectional Transformer trained on unlabeled text to predict masked tokens within a sentence and to predict whether one sentence follows anotherCITE:E7.

How Does BERT's Masked Language Model Training Mechanism Work?

BERT's masked language model hides 15% of the words in the input, runs the full sequence through a deep bidirectional Transformer encoder, and then predicts only the masked wordsCITE:E4. Because the entire sequence — not just a left-to-right or right-to-left slice — passes through the encoder before prediction, the model is forced to draw on context from both directions to fill in each masked positionCITE:E4. This masking objective is paired with the next-sentence prediction task, in which BERT predicts whether one sentence follows anotherCITE:E7.

What Are the Architectural Differences Between BERT-Base and BERT-Large?

BERT-Base uses 12 layers, 768 hidden dimensions, and 12 attention heads for 110 million parameters, while BERT-Large scales to 24 layers, 1024 hidden dimensions, and 16 attention heads for 340 million parametersCITE:E5CITE:E6.

Both configurations share the same pre-training objective — masking 15% of input words and predicting sentence order — with BERT-Large applying roughly double the layer count and hidden dimensionality of BERT-BaseCITE:E5CITE:E6.

What Results Did BERT Achieve on NLP Tasks?

BERT set new state-of-the-art results on eleven natural language processing tasks and pushed the GLUE score to 80.5%, a 7.7 percentage point absolute improvementCITE:E2. Google reported this figure as part of the same paper that introduced the bidirectional pre-training method and the masked-language-model objectiveCITE:E2.

What This Means

The eleven-task, 80.5% GLUE result was produced by the same architecture described above: bidirectional conditioning applied in every layerCITE:E1, a 15%-masking training objectiveCITE:E4, and a next-sentence-prediction task run alongside itCITE:E7. The two published configurations — BERT-Base at 110M parameters and BERT-Large at 340M parametersCITE:E5CITE:E6 — represent two scales of the identical bidirectional pre-training recipe rather than two different methods.