Good representations reduce sample complexity of downstream tasks
Good, useful representations reduce sample complexity on downstream supervised tasks. sample complexity is the number of training examples a machine learning model needs to successfully learn a target function. Therefore, good representations reduce the amount of labeled training data needed for downstream tasks to achieve a given level of performance.
Solving a downstream supervised task with good pre-trained representations only requires samples on the order of the dimensionality of output. So binary classification tasks will only require ~2 examples during fine-tuning to “solve” the task.