Review Self-Supervised Learning

Self-supervised training is a method where the training signal is derived from the data itself, rather than from human-provided labels.

Self-supervised learning creates a training signal from the structure of the data itself. The system may hide, transform, or relate parts of an example and learn by predicting the missing or connected information.

It can use large amounts of unlabeled data, reducing the need for manual annotation. The task design still determines what patterns the model is encouraged to learn.

A language model learns by predicting masked or next tokens drawn from the surrounding text rather than from labels written by a human for each sentence.

Self-supervised does not mean the model sets its own goals or evaluates itself. People still choose the data, objective, architecture, and training process.

The data supplies the training signal.

* [[pretraining|Pretraining]] * [[contrastive-learning|Contrastive Learning]] * [[unsupervised-learning|Unsupervised Learning]] * [[training-data|Training Data]]

* [Linus Ericsson and colleagues, “Self-Supervised Representation Learning: Introduction, Advances and Challenges”](https://arxiv.org/abs/2110.09327) — surveys methods that learn representations without manually annotated labels across several modalities.

* Compare [[pretraining|Pretraining]], [[contrastive-learning|Contrastive Learning]], [[unsupervised-learning|Unsupervised Learning]], and [[training-data|Training Data]] to place Self-Supervised Learning in the surrounding system. * Follow the cited source material above for the method, standard, or research context behind this draft.