Mixed-precision training combines multiple numerical formats in one computation. Moving from 32-bit to 16-bit formats such as BF16, and now to 8-bit FP8, tracks a hardware trend already visible in NVIDIA's Volta Tensor Cores, which deliver 8x the throughput of single-precision pipelines, while half-precision training cuts memory consumption by nearly 2x and a full-precision weight copy protects accuracy.
What Is Mixed-Precision Training?
Mixed precision is the combined use of different numerical precisions within a single computational method, according to NVIDIA's technical documentationCITE:E1. Rather than running every operation at one fixed bit-width, a training method can mix formats — for example carrying out arithmetic at lower precision while keeping certain values at higher precision — within the same computationCITE:E1.
How Do FP16 and BF16 Compare as 16-Bit Formats?
Bfloat16 (BF16) is a 16-bit floating-point format built from one sign bit, eight exponent bits, and seven mantissa bits, Google Cloud explained in an August 24, 2019 blog postCITE:E5. Google Cloud also stated that bfloat16 has a greater dynamic range — that is, more exponent bits — than FP16CITE:E6. Both formats total 16 bits, but the split between exponent and mantissa bits differs, which is why BF16 trades some numeric precision for a wider representable range compared with FP16CITE:E5CITE:E6.
What Is FP8 and How Are E4M3 and E5M2 Encoded?
FP8 is an 8-bit floating-point binary interchange format proposed in a paper published on arXiv on September 12, 2022, by Micikevicius et al., consisting of two encodingsCITE:E7. The first, E4M3, uses a 4-bit exponent and a 3-bit mantissa; the second, E5M2, uses a 5-bit exponent and a 2-bit mantissaCITE:E7. The two encodings mirror the FP16/BF16 trade-off one bit-width down: E4M3 keeps more mantissa bits for precision, while E5M2 keeps more exponent bits for rangeCITE:E7.
| Format | Sign Bits | Exponent Bits | Mantissa Bits | Total Bits |
|---|
| BF16 | 1 | 8 | 7 | 16 |
| FP8 E4M3 | shared | 4 | 3 | 8 |
| FP8 E5M2 | shared | 5 | 2 | 8 |
How Do Tensor Cores Accelerate Mixed-Precision Math?
NVIDIA's Volta generation of GPUs introduced Tensor Cores that provide 8x more throughput than single-precision math pipelines, according to NVIDIA's documentationCITE:E4. This hardware-level throughput multiplier is what makes reduced-precision formats like FP16, BF16, and FP8 attractive for training beyond their memory benefits aloneCITE:E4.
How Much Memory Does Lower Precision Save?
Training with half precision reduces the memory consumption of deep learning models by nearly 2x, Micikevicius et al. reported in a paper published on arXiv on October 10, 2017CITE:E3. That memory reduction comes from storing activations, gradients, and weights in a 16-bit format instead of 32-bit during the bulk of training computationCITE:E3.
How Is Accuracy Preserved in Mixed-Precision Training?
Micikevicius et al. recommend maintaining a single-precision (FP32) copy of the weights that accumulates gradients after each optimizer stepCITE:E2. This FP32 master copy acts as the accuracy safeguard underneath the lower-precision forward and backward computation, so the memory and throughput gains from reduced precision do not come at the cost of losing small gradient updatesCITE:E2.
What This Means
Across the evidence, the same pattern repeats at two different bit-widths: NVIDIA's Volta Tensor Cores already delivered an 8x throughput gain for reduced-precision mathCITE:E4, the 2017 paper measured a nearly 2x memory reduction from moving to half precisionCITE:E3, and that same paper's fix for the resulting precision loss — a persistent FP32 weight copy accumulating gradientsCITE:E2 — was published alongside the format comparison between FP16 and BF16 that Google Cloud later detailedCITE:E5CITE:E6. The 2022 FP8 proposal splits precision and range across two distinct encodings, E4M3 and E5M2CITE:E7, following the same exponent-versus-mantissa trade-off already visible in the 16-bit formatsCITE:E5CITE:E6.