RS Glossary
Pretraining
Pretraining is the broad first stage of training where a model learns general patterns from a very large dataset before any narrower tuning.
- Review state
- Reviewed in POC
- Freshness
- Current
- Version
- 2
- Pretraining
- Pre-training
Remember it this wayLearn broad patterns first; specialize later.
In plain language
Pretraining is the broad initial training stage that teaches a model general patterns from a large dataset before the model is adapted to a narrower task or use.
Why it matters
Pretraining shapes the capabilities and limitations that later prompting or fine-tuning builds on. Its data, objective, and scale influence every downstream use.
Example
A language model first learns to predict text across a large corpus, then receives supervised fine-tuning for instruction following.
Common misconception
Pretraining does not make a model ready for every use. It creates a base that may still need adaptation, evaluation, safeguards, and application-specific context.
Related terms
Sources
- Jacob Devlin and colleagues, “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding”demonstrates broad language-model pretraining followed by task-specific fine-tuning.
Learn more
- Compare [[training|Training]], [[foundation-model|Foundation Model]], [[fine-tuning|Fine-Tuning]], and [[training-data|Training Data]] to place Pretraining in the surrounding system.
- Follow the cited source material above for the method, standard, or research context behind this draft.
Public history
23 recorded events · 23 public summaries
A potential shared concept was discussed and selected for initial glossary drafting.
The captured candidate received a complete initial draft and entered section-level human review.
Radar reviewed the initial definition draft for human consideration.
Radar reviewed the initial plain-language-explanation draft for human consideration.
Radar reviewed the initial why-it-matters draft for human consideration.
Radar reviewed the initial example draft for human consideration.
Radar reviewed the initial common-misconception draft for human consideration.
Radar reviewed the initial remember-it-this-way draft for human consideration.
Radar reviewed the initial related-terms draft for human consideration.
Radar reviewed the initial sources draft for human consideration.
Radar reviewed the initial learn-more draft for human consideration.
Definition review recorded as approve.
Plain-Language Explanation review recorded as approve.
Why It Matters review recorded as approve.
Example review recorded as approve.
Common Misconception review recorded as approve.
Remember It This Way review recorded as approve.
Related Terms review recorded as approve.
Sources review recorded as approve.
Learn More review recorded as approve.
Sources review recorded as replace.
Radar verified the learner-facing sources after personal origin provenance was removed.
All nine current wording approvals were reconciled to Approved term status.
Source: vault/Glossary/terms/pretraining/term.md · sha256:63aa1e9543a1628e2360121902cfc5e91b48b1047594fe7eb24b399e8ce45b2b · Release: rs-glossary-glossary-team-poc-0.5.5-2026-09-02