Review Scaling Laws

Scaling laws are empirical relationships that describe how selected measures of model performance or training behavior change as factors such as compute, data, and model size increase under specified conditions.

Researchers fit patterns to results from multiple training runs and use them to estimate tradeoffs or expected improvement. The relationship depends on the metric, architecture, data, and range studied.

Scaling laws influence training plans and investment, but extrapolation can fail when methods, data quality, bottlenecks, or evaluation targets change.

A team estimates how validation loss may change when training compute and dataset size increase together according to an observed relationship.

A scaling law is not a universal law guaranteeing that every useful capability improves predictably with more parameters or money.

Scaling laws describe observed trends under stated conditions.

* [[compute|Compute]] * [[parameters|Parameters]] * [[training-data|Training Data]] * [[emergent-capabilities|Emergent Capabilities]] * [[evidence-selection|Evidence Selection]]

* [Kaplan et al., Scaling Laws for Neural Language Models](https://arxiv.org/abs/2001.08361) — primary empirical paper describing language-model scaling relationships.

* Read any scaling claim with its metric, range, assumptions, and supporting evidence.