OpenAI's Whisper is a general-purpose, multitasking speech model trained simply to predict transcripts of internet audio. Scaled to 680,000 hours of multilingual and multitask supervision, it generalizes in zero-shot settings without fine-tuning and approaches human-level accuracy and robustness.
What kind of speech recognition model is Whisper, and what functions does it support?
Whisper is a general-purpose speech recognition modelCITE:E4. It is trained on a large dataset of diverse audio and is also a multitasking model that can perform multilingual speech recognition, speech translation, and language identificationCITE:E5. That combination — one model covering recognition, translation, and language identification rather than separate specialized systems — defines its stated scope.
How does Whisper use large-scale weak supervision to build its speech recognition system?
OpenAI trains Whisper to predict transcripts of large amounts of internet audio, the core method behind its weak-supervision approachCITE:E1. Structurally, Whisper is built as an encoder-decoder (sequence-to-sequence) transformer, pretrained on 680,000 hours of labeled audio dataCITE:E6. In other words, the training signal comes from transcripts already present on the internet rather than from a purpose-built, manually annotated corpus, and that signal is fed into a standard encoder-decoder transformer architecture.
How does 680,000 hours of pretraining data drive multilingual and multitask capability?
When scaled to 680,000 hours of multilingual and multitask supervision, Whisper generalizes well and is often competitive with fully supervised results in a zero-shot transfer setting, without any fine-tuningCITE:E2. This amount of pretraining data enables zero-shot performance on audio tasks in English and many other languagesCITE:E7. The same 680,000-hour figure appears consistently across the original research paper and separate technical documentation, describing both the scale of the training data and the resulting zero-shot capability.
How does Whisper's accuracy and robustness compare to human-level performance?
Whisper's models approach human-level accuracy and robustness when directly compared against human transcribersCITE:E3. This comparison is presented alongside the zero-shot, no-fine-tuning results described above, tying the training method to the reported outcome.
What does this mean?
Across the research paper, the model repository, and independent technical documentation, the same figures recur: transcripts of internet audio as the training signalCITE:E1, 680,000 hours of multilingual and multitask supervision as the scaleCITE:E2CITE:E6, and zero-shot performance across English and many other languages as the outcomeCITE:E7. Whisper's general-purpose, multitasking framingCITE:E4CITE:E5 and its near-human accuracy and robustness claimCITE:E3 are presented as the product of that same 680,000-hour weak-supervision process, not as separate achievements.
Author's Take・EffectStory 編輯部
Whisper's core bet is architectural conservatism paired with data scale: the underlying model is a standard encoder-decoder transformer, and its stated advantage — zero-shot performance competitive with fully supervised systems — is attributed to the 680,000 hours of weakly supervised transcript data rather than to a novel architecture. That is a notable claim because it implies transcription quality and multilingual/multitask coverage track data volume more than model design. The detail worth watching is one the source material itself leaves open: the zero-shot claim is stated only as covering 'English and many other languages,' with no per-language breakdown or accuracy figure attached. Until that gap is filled, the 'near-human accuracy and robustness' comparison should be read as a benchmark-level claim rather than a guarantee across every language Whisper's documentation says it supports.