{
  "source_file": "0930",
  "source_platform": "个人博客",
  "author": "Lilian Weng / Lil'Log",
  "title_listed": "Contrastive Representation Learning",
  "keywords": "对比学习,表示学习,自监督",
  "domain": "机器学习",
  "content_type": "深度技术解析",
  "is_paid_flag": "否",
  "url": "https://lilianweng.github.io/posts/2021-05-31-contrastive/",
  "publish_date": "2021-05",
  "corpus_stage": "预训练/SFT",
  "notes": "对比表示学习综述（SimCLR、MoCo等）",
  "id": "c8f3f2a59bb8",
  "collected_at": "2026-10-02T14:12:57",
  "title": "Contrastive Representation Learning | Lil'Log",
  "body_markdown": "The goal of contrastive representation learning is to learn such an embedding space in which similar sample pairs stay close to each other while dissimilar ones are far apart. Contrastive learning can be applied to both supervised and unsupervised settings. When working with unsupervised data, contrastive learning is one of the most powerful approaches in\nself-supervised learning\n.\nContrastive Training Objectives\n#\nIn early versions of loss functions for contrastive learning, only one positive and one negative sample are involved. The trend in recent training objectives is to include multiple positive and negative pairs in one batch.\nContrastive Loss\n#\nContrastive loss\n(\nChopra et al. 2005\n) is one of the earliest training objectives used for deep metric learning in a contrastive fashion.\nGiven a list of input samples $\\{ \\mathbf{x}_i \\}$, each has a corresponding label $y_i \\in \\{1, \\dots, L\\}$ among $L$ classes. We would like to learn a function $f_\\theta(.): \\mathcal{X}\\to\\mathbb{R}^d$ that encodes $x_i$ into an embedding vector such that examples from the same class have similar embeddings and samples from different classes have very different ones. Thus, contrastive loss takes a pair of inputs $(x_i, x_j)$ and minimizes the embedding distance when they are from the same class but maximizes the distance otherwise.\n$$\n\\mathcal{L}_\\text{cont}(\\mathbf{x}_i, \\mathbf{x}_j, \\theta) = \\mathbb{1}[y_i=y_j] \\| f_\\theta(\\mathbf{x}_i) - f_\\theta(\\mathbf{x}_j) \\|^2_2 + \\mathbb{1}[y_i\\neq y_j]\\max(0, \\epsilon - \\|f_\\theta(\\mathbf{x}_i) - f_\\theta(\\mathbf{x}_j)\\|_2)^2\n$$\nwhere $\\epsilon$ is a hyperparameter, defining the lower bound distance between samples of different classes.\nTriplet Loss\n#\nTriplet loss\nwas originally proposed in the FaceNet (\nSchroff et al. 2015\n) paper and was used to learn face recognition of the same person at different poses and angles.\nIllustration of triplet loss given one positive and one negative per anchor. (Image source:\nSchroff et al. 2015\n)\nGiven one anchor input $\\mathbf{x}$, we select one positive sample $\\mathbf{x}^+$ and one negative $\\mathbf{x}^-$, meaning that $\\mathbf{x}^+$ and $\\mathbf{x}$ belong to the same class and $\\mathbf{x}^-$ is sampled from another different class. Triplet loss learns to minimize the distance between the anchor $\\mathbf{x}$ and positive $\\mathbf{x}^+$ and maximize the distance between the anchor $\\mathbf{x}$ and negative $\\mathbf{x}^-$ at the same time with the following equation:\n$$\n\\mathcal{L}_\\text{triplet}(\\mathbf{x}, \\mathbf{x}^+, \\mathbf{x}^-) = \\sum_{\\mathbf{x} \\in \\mathcal{X}} \\max\\big( 0, \\|f(\\mathbf{x}) - f(\\mathbf{x}^+)\\|^2_2 - \\|f(\\mathbf{x}) - f(\\mathbf{x}^-)\\|^2_2 + \\epsilon \\big)\n$$\nwhere the margin parameter $\\epsilon$ is configured as the minimum offset between distances of similar vs dissimilar pairs.\nIt is crucial to select challenging $\\mathbf{x}^-$ to truly improve the model.\nLifted Structured Loss\n#\nLifted Structured Loss\n(\nSong et al. 2015\n) utilizes all the pairwise edges within one training batch for better computational efficiency.\nIllustration compares contrastive loss, triplet loss and lifted structured loss. Red and blue edges connect similar and dissimilar sample pairs respectively. (Image source:\nSong et al. 2015\n)\nLet $D_{ij} = | f(\\mathbf{x}_i) - f(\\mathbf{x}_j) |_2$, a structured loss function is defined as\n$$\n\\begin{aligned}\n\\mathcal{L}_\\text{struct} &= \\frac{1}{2\\vert \\mathcal{P} \\vert} \\sum_{(i,j) \\in \\mathcal{P}} \\max(0, \\mathcal{L}_\\text{struct}^{(ij)})^2 \\\\\n\\text{where } \\mathcal{L}_\\text{struct}^{(ij)} &= D_{ij} + \\color{red}{\\max \\big( \\max_{(i,k)\\in \\mathcal{N}} \\epsilon - D_{ik}, \\max_{(j,l)\\in \\mathcal{N}} \\epsilon - D_{jl} \\big)}\n\\end{aligned}\n$$\nwhere $\\mathcal{P}$ contains the set of positive pairs and $\\mathcal{N}$ is the set of negative pairs. Note that the dense pairwise squared distance matrix can be easily computed per training batch.\nThe\nred\npart in $\\mathcal{L}_\\text{struct}^{(ij)}$ is used for mining hard negatives. However, it is not smooth and may cause the convergence to a bad local optimum in practice. Thus, it is relaxed to be:\n$$\n\\mathcal{L}_\\text{struct}^{(ij)} = D_{ij} + \\log \\Big( \\sum_{(i,k)\\in\\mathcal{N}} \\exp(\\epsilon - D_{ik}) + \\sum_{(j,l)\\in\\mathcal{N}} \\exp(\\epsilon - D_{jl}) \\Big)\n$$\nIn the paper, they also proposed to enhance the quality of negative samples in each batch by actively incorporating difficult negative samples given a few random positive pairs.\nN-pair Loss\n#\nMulti-Class N-pair loss\n(\nSohn 2016\n) generalizes triplet loss to include comparison with multiple negative samples.\nGiven a $(N + 1)$-tuplet of training samples, $\\{ \\mathbf{x}, \\mathbf{x}^+, \\mathbf{x}^-_1, \\dots, \\mathbf{x}^-_{N-1} \\}$, including one positive and $N-1$ negative ones, N-pair loss is defined as:\n$$\n\\begin{aligned}\n\\mathcal{L}_\\text{N-pair}(\\mathbf{x}, \\mathbf{x}^+, \\{\\mathbf{x}^-_i\\}^{N-1}_{i=1}) \n&= \\log\\big(1 + \\sum_{i=1}^{N-1} \\exp(f(\\mathbf{x})^\\top f(\\mathbf{x}^-_i) - f(\\mathbf{x})^\\top f(\\mathbf{x}^+))\\big) \\\\\n&= -\\log\\frac{\\exp(f(\\mathbf{x})^\\top f(\\mathbf{x}^+))}{\\exp(f(\\mathbf{x})^\\top f(\\mathbf{x}^+)) + \\sum_{i=1}^{N-1} \\exp(f(\\mathbf{x})^\\top f(\\mathbf{x}^-_i))}\n\\end{aligned}\n$$\nIf we only sample one negative sample per class, it is equivalent to the softmax loss for multi-class classification.\nNCE\n#\nNoise Contrastive Estimation\n, short for\nNCE\n, is a method for estimating parameters of a statistical model, proposed by\nGutmann & Hyvarinen\nin 2010. The idea is to run logistic regression to tell apart the target data from noise. Read more on how NCE is used for learning word embedding\nhere\n.\nLet $\\mathbf{x}$ be the target sample $\\sim P(\\mathbf{x} \\vert C=1; \\theta) = p_\\theta(\\mathbf{x})$ and $\\tilde{\\mathbf{x}}$ be the noise sample $\\sim  P(\\tilde{\\mathbf{x}} \\vert C=0) = q(\\tilde{\\mathbf{x}})$. Note that the logistic regression models the logit (i.e. log-odds) and in this case we would like to model the logit of a sample $u$ from the target data distribution instead of the noise distribution:\n$$\n\\ell_\\theta(\\mathbf{u}) = \\log \\frac{p_\\theta(\\mathbf{u})}{q(\\mathbf{u})} = \\log p_\\theta(\\mathbf{u}) - \\log q(\\mathbf{u})\n$$\nAfter converting logits into probabilities with sigmoid $\\sigma(.)$, we can apply cross entropy loss:\n$$\n\\begin{aligned}\n\\mathcal{L}_\\text{NCE} &= - \\frac{1}{N} \\sum_{i=1}^N \\big[ \\log \\sigma (\\ell_\\theta(\\mathbf{x}_i)) + \\log (1 - \\sigma (\\ell_\\theta(\\tilde{\\mathbf{x}}_i))) \\big] \\\\\n\\text{ where }\\sigma(\\ell) &= \\frac{1}{1 + \\exp(-\\ell)} = \\frac{p_\\theta}{p_\\theta + q}\n\\end{aligned}\n$$\nHere I listed the original form of NCE loss which works with only one positive and one noise sample. In many follow-up works, contrastive loss incorporating multiple negative samples is also broadly referred to as NCE.\nInfoNCE\n#\nThe\nInfoNCE loss\nin CPC (\nContrastive Predictive Coding\n;\nvan den Oord, et al. 2018\n), inspired by\nNCE\n, uses categorical cross-entropy loss to identify the positive sample amongst a set of unrelated noise samples.\nGiven a context vector $\\mathbf{c}$, the positive sample should be drawn from the conditional distribution $p(\\mathbf{x} \\vert \\mathbf{c})$, while $N-1$ negative samples are drawn from the proposal distribution $p(\\mathbf{x})$, independent from the context $\\mathbf{c}$. For brevity, let us label all the samples as $X=\\{ \\mathbf{x}_i \\}^N_{i=1}$ among which only one of them $\\mathbf{x}_\\texttt{pos}$ is a positive sample. The probability of we detecting the positive sample correctly is:\n$$\np(C=\\texttt{pos} \\vert X, \\mathbf{c}) \n= \\frac{p(x_\\texttt{pos} \\vert \\mathbf{c}) \\prod_{i=1,\\dots,N; i \\neq \\texttt{pos}} p(\\mathbf{x}_i)}{\\sum_{j=1}^N \\big[ p(\\mathbf{x}_j \\vert \\mathbf{c}) \\prod_{i=1,\\dots,N; i \\neq j} p(\\mathbf{x}_i) \\big]}\n= \\frac{ \\frac{p(\\mathbf{x}_\\texttt{pos}\\vert c)}{p(\\mathbf{x}_\\texttt{pos})} }{ \\sum_{j=1}^N \\frac{p(\\mathbf{x}_j\\vert \\mathbf{c})}{p(\\mathbf{x}_j)} }\n= \\frac{f(\\mathbf{x}_\\texttt{pos}, \\mathbf{c})}{ \\sum_{j=1}^N f(\\mathbf{x}_j, \\mathbf{c}) }\n$$\nwhere the scoring function is $f(\\mathbf{x}, \\mathbf{c}) \\propto \\frac{p(\\mathbf{x}\\vert\\mathbf{c})}{p(\\mathbf{x})}$.\nThe InfoNCE loss optimizes the negative log probability of classifying the positive sample correctly:\n$$\n\\mathcal{L}_\\text{InfoNCE} = - \\mathbb{E} \\Big[\\log \\frac{f(\\mathbf{x}, \\mathbf{c})}{\\sum_{\\mathbf{x}' \\in X} f(\\mathbf{x}', \\mathbf{c})} \\Big]\n$$\nThe fact that $f(x, c)$ estimates the density ratio $\\frac{p(x\\vert c)}{p(x)}$ has a connection with mutual information optimization. To maximize the the mutual information between input $x$ and context vector $c$, we have:\n$$\nI(\\mathbf{x}; \\mathbf{c}) = \\sum_{\\mathbf{x}, \\mathbf{c}} p(\\mathbf{x}, \\mathbf{c}) \\log\\frac{p(\\mathbf{x}, \\mathbf{c})}{p(\\mathbf{x})p(\\mathbf{c})} = \\sum_{\\mathbf{x}, \\mathbf{c}} p(\\mathbf{x}, \\mathbf{c})\\log\\color{blue}{\\frac{p(\\mathbf{x}|\\mathbf{c})}{p(\\mathbf{x})}}\n$$\nwhere the logarithmic term in\nblue\nis estimated by $f$.\nFor sequence prediction tasks, rather than modeling the future observations $p_k(\\mathbf{x}_{t+k} \\vert \\mathbf{c}_t)$ directly (which could be fairly expensive), CPC models a density function to preserve the mutual information between $\\mathbf{x}_{t+k}$ and $\\mathbf{c}_t$:\n$$\nf_k(\\mathbf{x}_{t+k}, \\mathbf{c}_t) = \\exp(\\mathbf{z}_{t+k}^\\top \\mathbf{W}_k \\mathbf{c}_t) \\propto \\frac{p(\\mathbf{x}_{t+k}\\vert\\mathbf{c}_t)}{p(\\mathbf{x}_{t+k})}\n$$\nwhere $\\mathbf{z}_{t+k}$ is the encoded input and $\\mathbf{W}_k$ is a trainable weight matrix.\nSoft-Nearest Neighbors Loss\n#\nSoft-Nearest Neighbors Loss\n(\nSalakhutdinov & Hinton 2007\n,\nFrosst et al. 2019\n) extends it to include multiple positive samples.\nGiven a batch of samples, $\\{\\mathbf{x}_i, y_i)\\}^B_{i=1}$ where $y_i$ is the class label of $\\mathbf{x}_i$ and a function $f(.,.)$ for measuring similarity between two inputs, the soft nearest neighbor loss at temperature $\\tau$ is defined as:\n$$\n\\mathcal{L}_\\text{snn} = -\\frac{1}{B}\\sum_{i=1}^B \\log \\frac{\\sum_{i\\neq j, y_i = y_j, j=1,\\dots,B} \\exp(- f(\\mathbf{x}_i, \\mathbf{x}_j) / \\tau)}{\\sum_{i\\neq k, k=1,\\dots,B} \\exp(- f(\\mathbf{x}_i, \\mathbf{x}_k) /\\tau)}\n$$\nThe temperature $\\tau$ is used for tuning how concentrated the features are in the representation space. For example, when at low temperature, the loss is dominated by the small distances and widely separated representations cannot contribute much and become irrelevant.\nCommon Setup\n#\nWe can loosen the definition of “classes” and “labels” in soft nearest-neighbor loss to create positive and negative sample pairs out of unsupervised data by, for example, applying data augmentation to create noise versions of original samples.\nMost recent studies follow the following definition of contrastive learning objective to incorporate multiple positive and negative samples. According to the setup in (\nWang & Isola 2020\n), let $p_\\texttt{data}(.)$ be the data distribution over $\\mathbb{R}^n$ and $p_\\texttt{pos}(., .)$ be the distribution of positive pairs over $\\mathbb{R}^{n \\times n}$. These two distributions should satisfy:\nSymmetry: $\\forall \\mathbf{x}, \\mathbf{x}^+, p_\\texttt{pos}(\\mathbf{x}, \\mathbf{x}^+) = p_\\texttt{pos}(\\mathbf{x}^+, \\mathbf{x})$\nMatching marginal: $\\forall \\mathbf{x}, \\int p_\\texttt{pos}(\\mathbf{x}, \\mathbf{x}^+) d\\mathbf{x}^+ = p_\\texttt{data}(\\mathbf{x})$\nTo learn an encoder $f(\\mathbf{x})$ to learn a\nL2-normalized feature vector\n, the contrastive learning objective is:\n$$\n\\begin{aligned}\n\\mathcal{L}_\\text{contrastive} \n&= \\mathbb{E}_{(\\mathbf{x},\\mathbf{x}^+)\\sim p_\\texttt{pos}, \\{\\mathbf{x}^-_i\\}^M_{i=1} \\overset{\\text{i.i.d}}{\\sim} p_\\texttt{data} } \\Big[ -\\log\\frac{\\exp(f(\\mathbf{x})^\\top f(\\mathbf{x}^+) / \\tau)}{ \\exp(f(\\mathbf{x})^\\top f(\\mathbf{x}^+) / \\tau) + \\sum_{i=1}^M \\exp(f(\\mathbf{x})^\\top f(\\mathbf{x}_i^-) / \\tau)} \\Big] & \\\\\n&\\approx \\mathbb{E}_{(\\mathbf{x},\\mathbf{x}^+)\\sim p_\\texttt{pos}, \\{\\mathbf{x}^-_i\\}^M_{i=1} \\overset{\\text{i.i.d}}{\\sim} p_\\texttt{data} }\\Big[ - f(\\mathbf{x})^\\top f(\\mathbf{x}^+) / \\tau + \\log\\big(\\sum_{i=1}^M \\exp(f(\\mathbf{x})^\\top f(\\mathbf{x}_i^-) / \\tau)\\big) \\Big] & \\scriptstyle{\\text{; Assuming infinite negatives}} \\\\\n&= -\\frac{1}{\\tau}\\mathbb{E}_{(\\mathbf{x},\\mathbf{x}^+)\\sim p_\\texttt{pos}}f(\\mathbf{x})^\\top f(\\mathbf{x}^+) + \\mathbb{E}_{ \\mathbf{x} \\sim p_\\texttt{data}} \\Big[ \\log \\mathbb{E}_{\\mathbf{x}^- \\sim p_\\texttt{data}} \\big[ \\sum_{i=1}^M \\exp(f(\\mathbf{x})^\\top f(\\mathbf{x}_i^-) / \\tau)\\big] \\Big] &\n\\end{aligned}\n$$\nKey Ingredients\n#\nHeavy Data Augmentation\n#\nGiven a training sample, data augmentation techniques are needed for creating noise versions of itself to feed into the loss as positive samples. Proper data augmentation setup is critical for learning good and generalizable embedding features. It introduces the non-essential variations into examples without modifying semantic meanings and thus encourages the model to learn the essential part of the representation. For example, experiments in\nSimCLR\nshowed that the composition of random cropping and random color distortion is crucial for good performance on learning visual representation of images.\nLarge Batch Size\n#\nUsing a large batch size during training is another key ingredient in the success of many contrastive learning methods (e.g.\nSimCLR\n,\nCLIP\n), especially when it relies on in-batch negatives. Only when the batch size is big enough, the loss function can cover a diverse enough collection of negative samples, challenging enough for the model to learn meaningful representation to distinguish different examples.\nHard Negative Mining\n#\nHard negative samples should have different labels from the anchor sample, but have embedding features very close to the anchor embedding. With access to ground truth labels in supervised datasets, it is easy to identify task-specific hard negatives. For example when learning sentence embedding, we can treat sentence pairs labelled as “contradiction” in NLI datasets as hard negative pairs (e.g.\nSimCSE\n, or use top incorrect candidates returned by BM25 with most keywords matched as hard negative samples (\nDPR\n;\nKarpukhin et al., 2020\n).\nHowever, it becomes tricky to do hard negative mining when we want to remain unsupervised. Increasing training batch size or\nmemory bank\nsize implicitly introduces more hard negative samples, but it leads to a heavy burden of large memory usage as a side effect.\nChuang et al. (2020)\nstudied the sampling bias in contrastive learning and proposed debiased loss. In the unsupervised setting, since we do not know the ground truth labels, we may accidentally sample false negative samples. Sampling bias can lead to significant performance drop.\nSampling bias which refers to false negative samples in contrastive learning can lead to a big performance drop. (Image source:\nChuang et al., 2020\n)\nLet us assume the probability of anchor class $c$ is uniform $\\rho(c)=\\eta^+$ and the probability of observing a different class is $\\eta^- = 1-\\eta^+$.\nThe probability of observing a positive example for $\\mathbf{x}$ is $p^+_x(\\mathbf{x}’)=p(\\mathbf{x}’\\vert \\mathbf{h}_{x’}=\\mathbf{h}_x)$;\nThe probability of getting a negative sample for $\\mathbf{x}$ is $p^-_x(\\mathbf{x}’)=p(\\mathbf{x}’\\vert \\mathbf{h}_{x’}\\neq\\mathbf{h}_x)$.\nWhen we are sampling $\\mathbf{x}^-$ , we cannot access the true $p^-_x(\\mathbf{x}^-)$ and thus $\\mathbf{x}^-$ may be sampled from the (undesired) anchor class $c$ with probability $\\eta^+$. The actual sampling data distribution becomes:\n$$\np(\\mathbf{x}') = \\eta^+ p^+_x(\\mathbf{x}') + \\eta^- p_x^-(\\mathbf{x}')\n$$\nThus we can use $p^-_x(\\mathbf{x}’) = (p(\\mathbf{x}’) - \\eta^+ p^+_x(\\mathbf{x}’))/\\eta^-$ for sampling $\\mathbf{x}^-$ to debias the loss. With $N$ samples $\\{\\mathbf{u}_i\\}^N_{i=1}$ from $p$ and $M$ samples $\\{ \\mathbf{v}_i \\}_{i=1}^M$ from $p^+_x$ , we can estimate the expectation of the second term $\\mathbb{E}_{\\mathbf{x}^-\\sim p^-_x}[\\exp(f(\\mathbf{x})^\\top f(\\mathbf{x}^-))]$ in the denominator of contrastive learning loss:\n$$\ng(\\mathbf{x}, \\{\\mathbf{u}_i\\}^N_{i=1}, \\{\\mathbf{v}_i\\}_{i=1}^M) = \\max\\Big\\{ \\frac{1}{\\eta^-}\\Big( \\frac{1}{N}\\sum_{i=1}^N \\exp(f(\\mathbf{x})^\\top f(\\mathbf{u}_i)) - \\frac{\\eta^+}{M}\\sum_{i=1}^M \\exp(f(\\mathbf{x})^\\top f(\\mathbf{v}_i)) \\Big), \\exp(-1/\\tau) \\Big\\}\n$$\nwhere $\\tau$ is the temperature and $\\exp(-1/\\tau)$ is the theoretical lower bound of $\\mathbb{E}_{\\mathbf{x}^-\\sim p^-_x}[\\exp(f(\\mathbf{x})^\\top f(\\mathbf{x}^-))]$.\nThe final debiased contrastive loss looks like:\n$$\n\\mathcal{L}^{N,M}_\\text{debias}(f) = \\mathbb{E}_{\\mathbf{x},\\{\\mathbf{u}_i\\}^N_{i=1}\\sim p;\\;\\mathbf{x}^+, \\{\\mathbf{v}_i\\}_{i=1}^M\\sim p^+} \\Big[ -\\log\\frac{\\exp(f(\\mathbf{x})^\\top f(\\mathbf{x}^+)}{\\exp(f(\\mathbf{x})^\\top f(\\mathbf{x}^+) + N g(x,\\{\\mathbf{u}_i\\}^N_{i=1}, \\{\\mathbf{v}_i\\}_{i=1}^M)} \\Big]\n$$\nt-SNE visualization of learned representation with debiased contrastive learning. (Image source:\nChuang et al., 2020\n)\nFollowing the above annotation,\nRobinson et al. (2021)\nmodified the sampling probabilities to target at hard negatives by up-weighting the probability $p^-_x(x’)$ to be proportional to its similarity to the anchor sample. The new sampling probability $q_\\beta(x^-)$ is:\n$$\nq_\\beta(\\mathbf{x}^-) \\propto \\exp(\\beta f(\\mathbf{x})^\\top f(\\mathbf{x}^-)) \\cdot p(\\mathbf{x}^-)\n$$\nwhere $\\beta$ is a hyperparameter to tune.\nWe can estimate the second term in the denominator $\\mathbb{E}_{\\mathbf{x}^- \\sim q_\\beta} [\\exp(f(\\mathbf{x})^\\top f(\\mathbf{x}^-))]$ using importance sampling where both the partition functions $Z_\\beta, Z^+_\\beta$ can be estimated empirically.\n$$\n\\begin{aligned}\n\\mathbb{E}_{\\mathbf{u} \\sim q_\\beta} [\\exp(f(\\mathbf{x})^\\top f(\\mathbf{u}))] &= \\mathbb{E}_{\\mathbf{u} \\sim p} [\\frac{q_\\beta}{p}\\exp(f(\\mathbf{x})^\\top f(\\mathbf{u}))] = \\mathbb{E}_{\\mathbf{u} \\sim p} [\\frac{1}{Z_\\beta}\\exp((\\beta + 1)f(\\mathbf{x})^\\top f(\\mathbf{u}))] \\\\\n\\mathbb{E}_{\\mathbf{v} \\sim q^+_\\beta} [\\exp(f(\\mathbf{x})^\\top f(\\mathbf{v}))] &= \\mathbb{E}_{\\mathbf{v} \\sim p^+} [\\frac{q^+_\\beta}{p}\\exp(f(\\mathbf{x})^\\top f(\\mathbf{v}))] = \\mathbb{E}_{\\mathbf{v} \\sim p} [\\frac{1}{Z^+_\\beta}\\exp((\\beta + 1)f(\\mathbf{x})^\\top f(\\mathbf{v}))]\n\\end{aligned}\n$$\nPseudo code for computing NCE loss, debiased contrastive loss, and hard negative sample objective when setting $M=1$. (Image source:\nRobinson et al., 2021\n)\nVision: Image Embedding\n#\nImage Augmentations\n#\nMost approaches for contrastive representation learning in the vision domain rely on creating a noise version of a sample by applying a sequence of data augmentation techniques. The augmentation should significantly change its visual appearance but keep the semantic meaning unchanged.\nBasic Image Augmentation\n#\nThere are many ways to modify an image while retaining its semantic meaning. We can use any one of the following augmentation or a composition of multiple operations.\nRandom cropping and then resize back to the original size.\nRandom color distortions\nRandom Gaussian blur\nRandom color jittering\nRandom horizontal flip\nRandom grayscale conversion\nMulti-crop augmentation: Use two standard resolution crops and sample a set of additional low resolution crops that cover only small parts of the image. Using low resolution crops reduces the compute cost. (\nSwAV\n)\nAnd many more …\nAugmentation Strategies\n#\nMany frameworks are designed for learning good data augmentation strategies (i.e. a composition of multiple transforms). Here are a few common ones.\nAutoAugment\n(\nCubuk, et al. 2018\n): Inspired by\nNAS\n, AutoAugment frames the problem of learning best data augmentation operations (i.e. shearing, rotation, invert, etc.) for image classification as an RL problem and looks for the combination that leads to the highest accuracy on the evaluation set.\nRandAugment (\nCubuk et al., 2019\n): RandAugment greatly reduces the search space of AutoAugment by controlling the magnitudes of different transformation operations with a single magnitude parameter.\nPBA (Population based augmentation;\nHo et al., 2019\n): PBA combined PBT (\nJaderberg et al, 2017\n) with AutoAugment, using the evolutionary algorithm to train a population of children models in parallel to evolve the best augmentation strategies.\nUDA (Unsupervised Data Augmentation;\nXie et al., 2019\n): Among a set of possible augmentation strategies, UDA selects those to minimize the KL divergence between the predicted distribution over an unlabelled example and its unlabelled augmented version.\nImage Mixture\n#\nImage mixture methods can construct new training examples from existing data points.\nMixup (\nZhang et al., 2018\n): It runs global-level mixture by creating a weighted pixel-wise combination of two existing images $I_1$ and $I_2$: $I_\\text{mixup} \\gets \\alpha I_1 + (1-\\alpha) I_2$ and $\\alpha \\in [0, 1]$.\nCutmix (\nYun et al., 2019\n): Cutmix does region-level mixture by generating a new example by combining a local region of one image with the rest of the other image. $I_\\text{cutmix} \\gets \\mathbf{M}_b \\odot I_1 + (1-\\mathbf{M}_b) \\odot I_2$, where $\\mathbf{M}_b \\in \\{0, 1\\}^I$ is a binary mask and $\\odot$ is element-wise multiplication. It is equivalent to filling the cutout (\nDeVries & Taylor 2017\n) region with the same region from another image.\nMoCHi (“Mixing of Contrastive Hard Negatives”;\nKalantidis et al. 2020\n): Given a query $\\mathbf{q}$, MoCHi maintains a queue of $K$ negative features $Q=\\{\\mathbf{n}_1, \\dots, \\mathbf{n}_K \\}$ and sorts these negative features by similarity to the query, $\\mathbf{q}^\\top \\mathbf{n}$, in descending order. The first $N$ items in the queue are considered as the hardest negatives, $Q^N$. Then synthetic hard examples can be generated by $\\mathbf{h} = \\tilde{\\mathbf{h}} / |\\tilde{\\mathbf{h}}|$ where $\\tilde{\\mathbf{h}} = \\alpha\\mathbf{n}_i + (1-\\alpha) \\mathbf{n}_j$ and $\\alpha \\in (0, 1)$. Even harder examples can be created by mixing with the query feature, $\\mathbf{h}’ = \\tilde{\\mathbf{h}’} / |\\tilde{\\mathbf{h}’}|_2$ where $\\tilde{\\mathbf{h}’} = \\beta\\mathbf{q} + (1-\\beta) \\mathbf{n}_j$ and $\\beta \\in (0, 0.5)$.\nParallel Augmentation\n#\nThis category of approaches produce two noise versions of one anchor image and aim to learn representation such that these two augmented samples share the same embedding.\nSimCLR\n#\nSimCLR\n(\nChen et al, 2020\n) proposed a simple framework for contrastive learning of visual representations. It learns representations for visual inputs by maximizing agreement between differently augmented views of the same sample via a contrastive loss in the latent space.\nA simple framework for contrastive learning of visual representations. (Image source:\nChen et al, 2020\n)\nRandomly sample a minibatch of $N$ samples and each sample is applied with two different data augmentation operations, resulting in $2N$ augmented samples in total.\n$$\n\\tilde{\\mathbf{x}}_i = t(\\mathbf{x}),\\quad\\tilde{\\mathbf{x}}_j = t'(\\mathbf{x}),\\quad t, t' \\sim \\mathcal{T}\n$$\nwhere two separate data augmentation operators, $t$ and $t’$, are sampled from the same family of augmentations $\\mathcal{T}$. Data augmentation includes random crop, resize with random flip, color distortions, and Gaussian blur.\nGiven one positive pair, other $2(N-1)$ data points are treated as negative samples. The representation is produced by a base encoder $f(.)$:\n$$\n\\mathbf{h}_i = f(\\tilde{\\mathbf{x}}_i),\\quad \\mathbf{h}_j = f(\\tilde{\\mathbf{x}}_j)\n$$\nThe contrastive learning loss is defined using cosine similarity $\\text{sim}(.,.)$. Note that the loss operates on an extra projection layer of the representation $g(.)$ rather than on the representation space directly. But only the representation $\\mathbf{h}$ is used for downstream tasks.\n$$\n\\begin{aligned}\n\\mathbf{z}_i &= g(\\mathbf{h}_i),\\quad\n\\mathbf{z}_j = g(\\mathbf{h}_j) \\\\\n\\mathcal{L}_\\text{SimCLR}^{(i,j)} &= - \\log\\frac{\\exp(\\text{sim}(\\mathbf{z}_i, \\mathbf{z}_j) / \\tau)}{\\sum_{k=1}^{2N} \\mathbb{1}_{[k \\neq i]} \\exp(\\text{sim}(\\mathbf{z}_i, \\mathbf{z}_k) / \\tau)}\n\\end{aligned}\n$$\nwhere $\\mathbb{1}_{[k \\neq i]}$ is an indicator function: 1 if $k\\neq i$ 0 otherwise.\nSimCLR needs a large batch size to incorporate enough negative samples to achieve good performance.\nThe algorithm for SimCLR. (Image source:\nChen et al, 2020\n).\nBarlow Twins\n#\nBarlow Twins\n(\nZbontar et al. 2021\n) feeds two distorted versions of samples into the same network to extract features and learns to make the\ncross-correlation matrix\nbetween these two groups of output features close to the identity. The goal is to keep the representation vectors of different distorted versions of one sample similar, while minimizing the redundancy between these vectors.\nIllustration of Barlow Twins learning pipeline. (Image source:\nZbontar et al. 2021\n).\nLet $\\mathcal{C}$ be a cross-correlation matrix computed between outputs from two identical networks along the batch dimension. $\\mathcal{C}$ is a square matrix with the size same as the feature network’s output dimensionality. Each entry in the matrix $\\mathcal{C}_{ij}$ is the cosine similarity between network output vector dimension at index $i, j$ and batch index $b$, $\\mathbf{z}_{b,i}^A$ and $\\mathbf{z}_{b,j}^B$, with a value between -1 (i.e. perfect anti-correlation) and 1 (i.e. perfect correlation).\n$$\n\\begin{aligned}\n\\mathcal{L}_\\text{BT} &= \\underbrace{\\sum_i (1-\\mathcal{C}_{ii})^2}_\\text{invariance term} + \\lambda \\underbrace{\\sum_i\\sum_{i\\neq j} \\mathcal{C}_{ij}^2}_\\text{redundancy reduction term} \\\\ \\text{where } \\mathcal{C}_{ij} &= \\frac{\\sum_b \\mathbf{z}^A_{b,i} \\mathbf{z}^B_{b,j}}{\\sqrt{\\sum_b (\\mathbf{z}^A_{b,i})^2}\\sqrt{\\sum_b (\\mathbf{z}^B_{b,j})^2}}\n\\end{aligned}\n$$\nBarlow Twins is competitive with SOTA methods for self-supervised learning. It naturally avoids trivial constants (i.e. collapsed representations), and is robust to different training batch sizes.\nAlgorithm of Barlow Twins in Pytorch style pseudo code. (Image source:\nZbontar et al. 2021\n).\nBYOL\n#\nDifferent from the above approaches, interestingly,\nBYOL\n(Bootstrap Your Own Latent;\nGrill, et al 2020\n) claims to achieve a new state-of-the-art results\nwithout using negative samples\n. It relies on two neural networks, referred to as\nonline\nand\ntarget\nnetworks that interact and learn from each other. The target network (parameterized by $\\xi$) has the same architecture as the online one (parameterized by $\\theta$), but with polyak averaged weights, $\\xi \\leftarrow \\tau \\xi + (1-\\tau) \\theta$.\nThe goal is to learn a presentation $y$ that can be used in downstream tasks. The online network parameterized by $\\theta$ contains:\nAn encoder $f_\\theta$;\nA projector $g_\\theta$;\nA predictor $q_\\theta$.\nThe target network has the same network architecture, but with different parameter $\\xi$, updated by polyak averaging $\\theta$: $\\xi \\leftarrow \\tau \\xi + (1-\\tau) \\theta$.\nThe model architecture of BYOL. After training, we only care about $f\\_\\theta$ for producing representation, $y=f\\_\\theta(x)$, and everything else is discarded. $\\text{sg}$ means stop gradient. (Image source:\nGrill, et al 2020\n)\nGiven an image $\\mathbf{x}$, the BYOL loss is constructed as follows:\nCreate two augmented views: $\\mathbf{v}=t(\\mathbf{x}); \\mathbf{v}’=t’(\\mathbf{x})$ with augmentations sampled $t \\sim \\mathcal{T}, t’ \\sim \\mathcal{T}’$;\nThen they are encoded into representations, $\\mathbf{y}_\\theta=f_\\theta(\\mathbf{v}), \\mathbf{y}’=f_\\xi(\\mathbf{v}’)$;\nThen they are projected into latent variables, $\\mathbf{z}_\\theta=g_\\theta(\\mathbf{y}_\\theta), \\mathbf{z}’=g_\\xi(\\mathbf{y}’)$;\nThe online network outputs a prediction $q_\\theta(\\mathbf{z}_\\theta)$;\nBoth $q_\\theta(\\mathbf{z}_\\theta)$ and $\\mathbf{z}’$ are L2-normalized, giving us $\\bar{q}_\\theta(\\mathbf{z}_\\theta) = q_\\theta(\\mathbf{z}_\\theta) / | q_\\theta(\\mathbf{z}_\\theta) |$ and $\\bar{\\mathbf{z}’} = \\mathbf{z}’ / |\\mathbf{z}’|$;\nThe loss $\\mathcal{L}^\\text{BYOL}_\\theta$ is MSE between L2-normalized prediction $\\bar{q}_\\theta(\\mathbf{z})$ and $\\bar{\\mathbf{z}’}$;\nThe other symmetric loss $\\tilde{\\mathcal{L}}^\\text{BYOL}_\\theta$ can be generated by switching $\\mathbf{v}’$ and $\\mathbf{v}$; that is, feeding $\\mathbf{v}’$ to online network and $\\mathbf{v}$ to target network.\nThe final loss is $\\mathcal{L}^\\text{BYOL}_\\theta + \\tilde{\\mathcal{L}}^\\text{BYOL}_\\theta$ and only  parameters $\\theta$ are optimized.\nUnlike most popular contrastive learning based approaches, BYOL does not use negative pairs. Most bootstrapping approaches rely on pseudo-labels or cluster indices, but BYOL directly boostrapps the latent representation.\nIt is quite interesting and surprising that\nwithout\nnegative samples, BYOL still works well. Later I ran into this\npost\nby Abe Fetterman & Josh Albrecht, they highlighted two surprising findings while they were trying to reproduce BYOL:\nBYOL generally performs no better than random when\nbatch normalization is removed\n.\nThe presence of batch normalization implicitly causes a form of contrastive learning.\nThey believe that using negative samples is important for avoiding model collapse (i.e. what if you use all-zeros representation for every data point?). Batch normalization injects dependency on negative samples\ninexplicitly\nbecause no matter how similar a batch of inputs are, the values are re-distributed (spread out $\\sim \\mathcal{N}(0, 1$) and therefore batch normalization prevents model collapse. Strongly recommend you to read the\nfull article\nif you are working in this area.\nMemory Bank\n#\nComputing embeddings for a large number of negative samples in every batch is extremely expensive. One common approach is to store the representation in memory to trade off data staleness for cheaper compute.\nInstance Discrimination with Memoy Bank\n#\nInstance contrastive learning\n(\nWu et al, 2018\n) pushes the class-wise supervision to the extreme by considering each instance as\na distinct class of its own\n. It implies that the number of “classes” will be the same as the number of samples in the training dataset. Hence, it is unfeasible to train a softmax layer with these many heads, but instead it can be approximated by\nNCE\n.\nThe training pipeline of instance-level contrastive learning. The learned embedding is L2-normalized. (Image source:\nWu et al, 2018\n)\nLet $\\mathbf{v} = f_\\theta(x)$ be an embedding function to learn and the vector is normalized to have $|\\mathbf{v}|=1$. A non-parametric classifier predicts the probability of a sample $\\mathbf{v}$ belonging to class $i$ with a temperature parameter $\\tau$:\n$$\nP(C=i\\vert \\mathbf{v}) = \\frac{\\exp(\\mathbf{v}_i^\\top \\mathbf{v} / \\tau)}{\\sum_{j=1}^n \\exp(\\mathbf{v}_j^\\top \\mathbf{v} / \\tau)}\n$$\nInstead of computing the representations for all the samples every time, they implement an\nMemory Bank\nfor storing sample representation in the database from past iterations. Let $V=\\{ \\mathbf{v}_i \\}$ be the memory bank and $\\mathbf{f}_i = f_\\theta(\\mathbf{x}_i)$ be the feature generated by forwarding the network. We can use the representation from the memory bank $\\mathbf{v}_i$ instead of the feature forwarded from the network $\\mathbf{f}_i$ when comparing pairwise similarity.\nThe denominator theoretically requires access to the representations of all the samples, but that is too expensive in practice. Instead we can estimate it via Monte Carlo approximation using a random subset of $M$ indices $\\{j_k\\}_{k=1}^M$.\n$$\nP(i\\vert \\mathbf{v}) \n= \\frac{\\exp(\\mathbf{v}^\\top \\mathbf{f}_i / \\tau)}{\\sum_{j=1}^N \\exp(\\mathbf{v}_j^\\top \\mathbf{f}_i / \\tau)}\n\\simeq \\frac{\\exp(\\mathbf{v}^\\top \\mathbf{f}_i / \\tau)}{\\frac{N}{M} \\sum_{k=1}^M \\exp(\\mathbf{v}_{j_k}^\\top \\mathbf{f}_i / \\tau)}\n$$\nBecause there is only one instance per class, the training is unstable and fluctuates a lot. To improve the training smoothness, they introduced an extra term for positive samples in the loss function based on the\nproximal optimization method\n. The final NCE loss objective looks like:\n$$\n\\begin{aligned}\n\\mathcal{L}_\\text{instance} &= - \\mathbb{E}_{P_d}\\big[\\log h(i, \\mathbf{v}^{(t-1)}_i) - \\lambda \\|\\mathbf{v}^{(t)}_i - \\mathbf{v}^{(t-1)}_i\\|^2_2\\big] - M\\mathbb{E}_{P_n}\\big[\\log(1 - h(i, \\mathbf{v}'^{(t-1)})\\big] \\\\\nh(i, \\mathbf{v}) &= \\frac{P(i\\vert\\mathbf{v})}{P(i\\vert\\mathbf{v}) + MP_n(i)} \\text{ where the noise distribution is uniform }P_n = 1/N\n\\end{aligned}\n$$\nwhere $\\{ \\mathbf{v}^{(t-1)} \\}$ are embeddings stored in the memory bank from the previous iteration. The difference between iterations $|\\mathbf{v}^{(t)}_i - \\mathbf{v}^{(t-1)}_i|^2_2$ will gradually vanish as the learned embedding converges.\nMoCo & MoCo-V2\n#\nMomentum Contrast\n(\nMoCo\n;\nHe et al, 2019\n) provides a framework of unsupervised learning visual representation as a\ndynamic dictionary look-up\n. The dictionary is structured as a large FIFO queue of encoded representations of data samples.\nGiven a query sample $\\mathbf{x}_q$, we get a query representation through an encoder $\\mathbf{q} = f_q(\\mathbf{x}_q)$. A list of key representations $\\{\\mathbf{k}_1, \\mathbf{k}_2, \\dots \\}$ in the dictionary are encoded by a momentum encoder $\\mathbf{k}_i = f_k (\\mathbf{x}^k_i)$. Let’s assume among them there is a single\npositive\nkey $\\mathbf{k}^+$ in the dictionary that matches $\\mathbf{q}$. In the paper, they create $\\mathbf{k}^+$ using a noise copy of $\\mathbf{x}_q$ with different\naugmentation\n. Then the\nInfoNCE\ncontrastive loss with temperature $\\tau$ is used over one positive and $N-1$ negative samples:\n$$\n\\mathcal{L}_\\text{MoCo} = - \\log \\frac{\\exp(\\mathbf{q} \\cdot \\mathbf{k}^+ / \\tau)}{\\sum_{i=1}^N \\exp(\\mathbf{q} \\cdot \\mathbf{k}_i / \\tau)}\n$$\nCompared to the\nmemory bank\n, a queue-based dictionary in MoCo enables us to reuse representations of immediately preceding mini-batches of data.\nThe MoCo dictionary is not differentiable as a queue, so we cannot rely on back-propagation to update the key encoder $f_k$. One naive way might be to use the same encoder for both $f_q$ and $f_k$. Differently, MoCo proposed to use a momentum-based update with a momentum coefficient $m \\in [0, 1)$. Say, the parameters of $f_q$ and $f_k$ are labeled as $\\theta_q$ and $\\theta_k$, respectively.\n$$\n\\theta_k \\leftarrow m \\theta_k + (1-m) \\theta_q\n$$\nIllustration of how Momentum Contrast (MoCo) learns visual representations. (Image source:\nHe et al, 2019\n)\nThe advantage of MoCo compared to\nSimCLR\nis that MoCo decouples the batch size from the number of negatives, but SimCLR requires a large batch size in order to have enough negative samples and suffers performance drops when their batch size is reduced.\nTwo designs in SimCLR, namely, (1) an MLP projection head and (2) stronger data augmentation, are proved to be very efficient.\nMoCo V2\n(\nChen et al, 2020\n) combined these two designs, achieving even better transfer performance with no dependency on a very large batch size.\nCURL\n#\nCURL\n(\nSrinivas, et al. 2020\n) applies the above ideas in\nReinforcement Learning\n. It learns a visual representation for RL tasks by matching embeddings of two data-augmented versions, $o_q$ and $o_k$, of the raw observation $o$ via contrastive loss. CURL primarily relies on random crop data augmentation. The key encoder is implemented as a momentum encoder with weights as EMA of the query encoder weights, same as in\nMoCo\n.\nOne significant difference between RL and supervised visual tasks is that RL depends on\ntemporal consistency\nbetween consecutive frames. Therefore, CURL applies augmentation consistently on each stack of frames to retain information about the temporal structure of the observation.\nThe architecture of CURL. (Image source:\nSrinivas, et al. 2020\n)\nFeature Clustering\n#\nDeepCluster\n#\nDeepCluster\n(\nCaron et al. 2018\n) iteratively clusters features via k-means and uses cluster assignments as pseudo labels to provide supervised signals.\nIllustration of DeepCluster method which iteratively clusters deep features and uses the cluster assignments as pseudo-labels. (Image source:\nCaron et al. 2018\n)\nIn each iteration, DeepCluster clusters data points using the prior representation and then produces the new cluster assignments as the classification targets for the new representation. However this iterative process is prone to trivial solutions. While avoiding the use of negative pairs, it requires a costly clustering phase and specific precautions to avoid collapsing to trivial solutions.\nSwAV\n#\nSwAV\n(\nSwapping Assignments between multiple Views\n;\nCaron et al. 2020\n) is an online contrastive learning algorithm. It computes a code from an augmented version of the image and tries to predict this code using another augmented version of the same image.\nComparison of SwAV and [contrastive instance learning](#instance-discrimination-with-memoy-bank). (Image source:\nCaron et al. 2020\n)\nGiven features of images with two different augmentations, $\\mathbf{z}_t$ and $\\mathbf{z}_s$, SwAV computes corresponding codes $\\mathbf{q}_t$ and $\\mathbf{q}_s$ and the loss quantifies the fit by swapping two codes using $\\ell(.)$ to measure the fit between a feature and a code.\n$$\n\\mathcal{L}_\\text{SwAV}(\\mathbf{z}_t, \\mathbf{z}_s) = \\ell(\\mathbf{z}_t, \\mathbf{q}_s) + \\ell(\\mathbf{z}_s, \\mathbf{q}_t)\n$$\nThe swapped fit prediction depends on the cross entropy between the predicted code and a set of $K$ trainable prototype vectors $\\mathbf{C} = \\{\\mathbf{c}_1, \\dots, \\mathbf{c}_K\\}$. The prototype vector matrix is shared across different batches and represents\nanchor clusters\nthat each instance should be clustered to.\n$$\n\\ell(\\mathbf{z}_t, \\mathbf{q}_s) = - \\sum_k \\mathbf{q}^{(k)}_s\\log\\mathbf{p}^{(k)}_t \\text{ where } \\mathbf{p}^{(k)}_t = \\frac{\\exp(\\mathbf{z}_t^\\top\\mathbf{c}_k  / \\tau)}{\\sum_{k'}\\exp(\\mathbf{z}_t^\\top \\mathbf{c}_{k'} / \\tau)}\n$$\nIn a mini-batch containing $B$ feature vectors $\\mathbf{Z} = [\\mathbf{z}_1, \\dots, \\mathbf{z}_B]$, the mapping matrix between features and prototype vectors is defined as $\\mathbf{Q} = [\\mathbf{q}_1, \\dots, \\mathbf{q}_B] \\in \\mathbb{R}_+^{K\\times B}$. We would like to maximize the similarity between the features and the prototypes:\n$$\n\\begin{aligned}\n\\max_{\\mathbf{Q}\\in\\mathcal{Q}} &\\text{Tr}(\\mathbf{Q}^\\top \\mathbf{C}^\\top \\mathbf{Z}) + \\varepsilon \\mathcal{H}(\\mathbf{Q}) \\\\\n\\text{where }\\mathcal{Q} &= \\big\\{ \\mathbf{Q} \\in \\mathbb{R}_{+}^{K \\times B} \\mid \\mathbf{Q}\\mathbf{1}_B = \\frac{1}{K}\\mathbf{1}_K, \\mathbf{Q}^\\top\\mathbf{1}_K = \\frac{1}{B}\\mathbf{1}_B \\big\\}\n\\end{aligned}\n$$\nwhere $\\mathcal{H}$ is the entropy, $\\mathcal{H}(\\mathbf{Q}) = - \\sum_{ij} \\mathbf{Q}_{ij} \\log \\mathbf{Q}_{ij}$, controlling the smoothness of the code. The coefficient $\\epsilon$ should not be too large; otherwise, all the samples will be assigned uniformly to all the clusters. The candidate set of solutions for $\\mathbf{Q}$ requires every mapping matrix to have each row sum up to $1/K$ and each column to sum up to $1/B$, enforcing that each prototype gets selected at least $B/K$ times on average.\nSwAV relies on the iterative Sinkhorn-Knopp algorithm (\nCuturi 2013\n) to find the solution for $\\mathbf{Q}$.\nWorking with Supervised Datasets\n#\nCLIP\n#\nCLIP\n(\nContrastive Language-Image Pre-training\n;\nRadford et al. 2021\n) jointly trains a text encoder and an image feature extractor over the pretraining task that predicts which caption goes with which image.\nIllustration of CLIP contrastive pre-training over text-image pairs. (Image source:\nRadford et al. 2021\n)\nGiven a batch of $N$ (image, text) pairs, CLIP computes the dense cosine similarity matrix between all $N\\times N$ possible (image, text) candidates within this batch. The text and image encoders are jointly trained to maximize the similarity between $N$ correct pairs of (image, text) associations while minimizing the similarity for $N(N-1)$ incorrect pairs via a symmetric cross entropy loss over the dense matrix.\nSee the numy-like pseudo code for CLIP in\nCLIP algorithm in Numpy style pseudo code. (Image source:\nRadford et al. 2021\n)\nCompared to other methods above for learning good visual representation, what makes CLIP really special is\n“the appreciation of using natural language as a training signal”\n. It does demand access to supervised dataset in which we know which text matches which image. It is trained on 400 million (text, image) pairs, collected from the Internet. The query list contains all the words occurring at least 100 times in the English version of Wikipedia. Interestingly, they found that Transformer-based language models are 3x slower than a bag-of-words (BoW) text encoder at zero-shot ImageNet classification. Using contrastive objective instead of trying to predict the exact words associated with images (i.e. a method commonly adopted by image caption prediction tasks) can further improve the data efficiency another 4x.\nUsing bag-of-words text encoding and contrastive training objectives can bring in multiple folds of data efficiency improvement. (Image source:\nRadford et al. 2021\n)\nCLIP produces good visual representation that can non-trivially transfer to many CV benchmark datasets, achieving results competitive with supervised baseline. Among tested transfer tasks, CLIP struggles with very fine-grained classification, as well as abstract or systematic tasks such as counting the number of objects. The transfer performance of CLIP models is smoothly correlated with the amount of model compute.\nSupervised Contrastive Learning\n#\nThere are several known issues with cross entropy loss, such as the lack of robustness to noisy labels and the possibility of poor margins. Existing improvement for cross entropy loss involves the curation of better training data, such as label smoothing and data augmentation.\nSupervised Contrastive Loss\n(\nKhosla et al. 2021\n) aims to leverage label information more effectively than cross entropy, imposing that normalized embeddings from the same class are closer together than embeddings from different classes.\nSupervised vs self-supervised contrastive losses. Supervised contrastive learning considers different samples from the same class as positive examples, in addition to augmented versions. (Image source:\nKhosla et al. 2021\n)\nGiven a set of randomly sampled $n$ (image, label) pairs, $\\{\\mathbf{x}_i, y_i\\}_{i=1}^n$, $2n$ training pairs can be created by applying two random augmentations of every sample, $\\{\\tilde{\\mathbf{x}}_i, \\tilde{y}_i\\}_{i=1}^{2n}$.\nSupervised contrastive loss $\\mathcal{L}_\\text{supcon}$ utilizes multiple positive and negative samples, very similar to\nsoft nearest-neighbor loss\n:\n$$\n\\mathcal{L}_\\text{supcon} = - \\sum_{i=1}^{2n} \\frac{1}{2 \\vert N_i \\vert - 1} \\sum_{j \\in N(y_i), j \\neq i} \\log \\frac{\\exp(\\mathbf{z}_i \\cdot \\mathbf{z}_j / \\tau)}{\\sum_{k \\in I, k \\neq i}\\exp({\\mathbf{z}_i \\cdot \\mathbf{z}_k / \\tau})}\n$$\nwhere $\\mathbf{z}_k=P(E(\\tilde{\\mathbf{x}_k}))$, in which $E(.)$ is an encoder network (augmented image mapped to vector) $P(.)$ is a projection network (one vector mapped to another). $N_i= \\{j \\in I: \\tilde{y}_j = \\tilde{y}_i \\}$ contains a set of indices of samples with label $y_i$. Including more positive samples into the set $N_i$ leads to improved results.\nAccording to their experiments, supervised contrastive loss:\ndoes outperform the base cross entropy, but only by a small amount.\noutperforms the cross entropy on robustness benchmark (ImageNet-C, which applies common naturally occuring perturbations such as noise, blur and contrast changes to the ImageNet dataset).\nis less sensitive to hyperparameter changes.\nLanguage: Sentence Embedding\n#\nIn this section, we focus on how to learn sentence embedding.\nText Augmentation\n#\nMost contrastive methods in vision applications depend on creating an augmented version of each image. However, it is more challenging to construct text augmentation which does not alter the semantics of a sentence.  In this section we look into three approaches for augmenting text sequences, including lexical edits, back-translation and applying cutoff or dropout.\nLexical Edits\n#\nEDA\n(\nEasy Data Augmentation\n;\nWei & Zou 2019\n) defines a set of simple but powerful operations for text augmentation. Given a sentence, EDA randomly chooses and applies one of four simple operations:\nSynonym replacement (SR): Replace $n$ random non-stop words with their synonyms.\nRandom insertion (RI): Place a random synonym of a randomly selected non-stop word in the sentence at a random position.\nRandom swap (RS): Randomly swap two words and repeat $n$ times.\nRandom deletion (RD): Randomly delete each word in the sentence with probability $p$.\nwhere $p=\\alpha$ and $n=\\alpha \\times \\text{sentence_length}$, with the intuition that longer sentences can absorb more noise while maintaining the original label. The hyperparameter $\\alpha$ roughly indicates the percent of words in one sentence that may be changed by one augmentation.\nEDA is shown to improve the classification accuracy on several classification benchmark datasets compared to baseline without EDA. The performance lift is more significant on a smaller training set. All the four operations in EDA help improve the classification accuracy, but get to optimal at different $\\alpha$’s.\nEDA leads to performance improvement on several classification benchmarks. (Image source:\nWei & Zou 2019\n)\nIn\nContextual Augmentation\n(\nSosuke Kobayashi, 2018\n), new substitutes for word $w_i$ at position $i$ can be smoothly sampled from a given probability distribution, $p(.\\mid S\\setminus\\{w_i\\})$, which is predicted by a bidirectional LM like BERT.\nBack-translation\n#\nCERT\n(\nContrastive self-supervised Encoder Representations from Transformers\n;\nFang et al. (2020)\n;\ncode\n) generates augmented sentences via\nback-translation\n. Various translation models for different languages can be employed for creating different versions of augmentations. Once we have a noise version of text samples, many contrastive learning frameworks introduced above, such as\nMoCo\n, can be used to learn sentence embedding.\nDropout and Cutoff\n#\nShen et al. (2020)\nproposed to apply\nCutoff\nto text augmentation, inspired by\ncross-view training\n. They proposed three cutoff augmentation strategies:\nToken cutoff\nremoves the information of a few selected tokens. To make sure there is no data leakage, corresponding tokens in the input, positional and other relevant embedding matrices should all be zeroed out.,\nFeature cutoff\nremoves a few feature columns.\nSpan cutoff\nremoves a continuous chunk of texts.\nSchematic illustration of token, feature and span cutoff augmentation strategies. (Image source:\nShen et al. 2020\n)\nMultiple augmented versions of one sample can be created. When training,\nShen et al. (2020)\napplied an additional KL-divergence term to measure the consensus between predictions from different augmented samples.\nSimCSE\n(\nGao et al. 2021\n;\ncode\n) learns from unsupervised data by predicting a sentence from itself with only\ndropout\nnoise. In other words, they treat dropout as data augmentation for text sequences. A sample is simply fed into the encoder twice with different dropout masks and these two versions are the positive pair where the other in-batch samples are considered as negative pairs. It feels quite similar to the cutoff augmentation, but dropout is more flexible with less well-defined semantic meaning of what content can be masked off.\nSimCSE creates augmented samples by applying different dropout masks. The supervised version leverages NLI datasets to predict positive (entailment) or negative (contradiction) given a pair of sentences. (Image source:\nGao et al. 2021\n)\nThey ran experiments on 7 STS (Semantic Text Similarity) datasets and computed cosine similarity between sentence embeddings.  They also tried out an optional MLM auxiliary objective loss to help avoid catastrophic forgetting of token-level knowledge. This aux loss was found to help improve performance on transfer tasks, but a consistent drop on the main STS tasks.\nExperiment numbers on a collection of STS benchmarks with SimCES. (Image source:\nGao et al. 2021\n)\nSupervision from NLI\n#\nThe pre-trained BERT sentence embedding without any fine-tuning has been found to have poor performance for semantic similarity tasks. Instead of using the raw embeddings directly, we need to refine the embedding with further fine-tuning.\nNatural Language Inference (NLI)\ntasks are the main data sources to provide supervised signals for learning sentence embedding; such as\nSNLI\n,\nMNLI\n, and\nQQP\n.\nSentence-BERT\n#\nSBERT (Sentence-BERT)\n(\nReimers & Gurevych, 2019\n) relies on siamese and triplet network architectures to learn sentence embeddings such that the sentence similarity can be estimated by cosine similarity between pairs of embeddings. Note that learning SBERT depends on supervised data, as it is fine-tuned on several NLI datasets.\nThey experimented with a few different prediction heads on top of BERT model:\nSoftmax classification objective: The classification head of the siamese network is built on the concatenation of two embeddings $f(\\mathbf{x}), f(\\mathbf{x}’)$ and $\\vert f(\\mathbf{x}) - f(\\mathbf{x}’) \\vert$. The predicted output is $\\hat{y}=\\text{softmax}(\\mathbf{W}_t [f(\\mathbf{x}); f(\\mathbf{x}’); \\vert f(\\mathbf{x}) - f(\\mathbf{x}’) \\vert])$. They showed that the most important component is the element-wise difference $\\vert f(\\mathbf{x}) - f(\\mathbf{x}’) \\vert$.\nRegression objective: This is the regression loss on $\\cos(f(\\mathbf{x}), f(\\mathbf{x}’))$, in which the pooling strategy has a big impact. In the experiments, they observed that\nmax\nperforms much worse than\nmean\nand\nCLS\n-token.\nTriplet objective: $\\max(0, |f(\\mathbf{x}) - f(\\mathbf{x}^+)|- |f(\\mathbf{x}) - f(\\mathbf{x}^-)| + \\epsilon)$, where $\\mathbf{x}, \\mathbf{x}^+, \\mathbf{x}^-$ are embeddings of the anchor, positive and negative sentences.\nIn the experiments, which objective function works the best depends on the datasets, so there is no universal winner.\nIllustration of Sentence-BERT training framework with softmax classification head and regression head. (Image source:\nReimers & Gurevych, 2019\n)\nThe\nSentEval\nlibrary (\nConneau and Kiela, 2018\n) is commonly used for evaluating the quality of learned sentence embedding. SBERT outperformed other baselines at that time (Aug 2019) on 5 out of 7 tasks.\nThe performance of Sentence-BERT on the SentEval benchmark. (Image source:\nReimers & Gurevych, 2019\n)\nBERT-flow\n#\nThe embedding representation space is deemed\nisotropic\nif embeddings are uniformly distributed on each dimension; otherwise, it is\nanisotropic\n.\nLi et al, (2020)\nshowed that a pre-trained BERT learns a non-smooth\nanisotropic\nsemantic space of sentence embeddings and thus leads to poor performance for text similarity tasks without fine-tuning. Empirically, they observed two issues with BERT sentence embedding:\nWord frequency biases the embedding space. High-frequency words are close to the origin, but low-frequency ones are far away from the origin.\nLow-frequency words scatter sparsely. The embeddings of low-frequency words tend to be farther to their $k$-NN neighbors, while the embeddings of high-frequency words concentrate more densely.\nBERT-flow\n(\nLi et al, 2020\n;\ncode\n) was proposed to transform the embedding to a smooth and isotropic Gaussian distribution via\nnormalizing flows\n.\nIllustration of the flow-based calibration over the original sentence embedding space in BERT-flow. (Image source:\nLi et al, 2020\n)\nLet $\\mathcal{U}$ be the observed BERT sentence embedding space and $\\mathcal{Z}$ be the desired latent space which is a standard Gaussian. Thus, $p_\\mathcal{Z}$ is a Gaussian density function and $f_\\phi: \\mathcal{Z}\\to\\mathcal{U}$ is an invertible transformation:\n$$\n\\mathbf{z}\\sim p_\\mathcal{Z}(\\mathbf{z}) \\quad \n\\mathbf{u}=f_\\phi(\\mathbf{z}) \\quad\n\\mathbf{z}=f^{-1}_\\phi(\\mathbf{u}) \n$$\nA flow-based generative model learns the invertible mapping function by maximizing the likelihood of $\\mathcal{U}$’s marginal:\n$$\n\\max_\\phi\\mathbb{E}_{\\mathbf{u}=\\text{BERT}(s), s\\sim\\mathcal{D}} \\Big[ \\log p_\\mathcal{Z}(f^{-1}_\\phi(\\mathbf{u})) + \\log\\big\\vert\\det\\frac{\\partial f^{-1}_\\phi(\\mathbf{u})}{\\partial\\mathbf{u}}\\big\\vert \\Big]\n$$\nwhere $s$ is a sentence sampled from the text corpus $\\mathcal{D}$. Only the flow parameters $\\phi$ are optimized while parameters in the pretrained BERT stay unchanged.\nBERT-flow was shown to improve the performance on most STS tasks either with or without supervision from NLI datasets. Because learning normalizing flows for calibration does not require labels, it can utilize the entire dataset including validation and test sets.\nWhitening Operation\n#\nSu et al. (2021)\napplied\nwhitening\noperation to improve the\nisotropy\nof the learned representation and also to reduce the dimensionality of sentence embedding.\nThey transform the mean value of the sentence vectors to 0 and the covariance matrix to the identity matrix. Given a set of samples $\\{\\mathbf{x}_i\\}_{i=1}^N$, let $\\tilde{\\mathbf{x}}_i$ and $\\tilde{\\Sigma}$ be the transformed samples and corresponding covariance matrix:\n$$\n\\begin{aligned}\n\\mu &= \\frac{1}{N}\\sum_{i=1}^N \\mathbf{x}_i \\quad \\Sigma = \\frac{1}{N}\\sum_{i=1}^N (\\mathbf{x}_i - \\mu)^\\top (\\mathbf{x}_i - \\mu) \\\\\n\\tilde{\\mathbf{x}}_i &= (\\mathbf{x}_i - \\mu)W \\quad \\tilde{\\Sigma} = W^\\top\\Sigma W = I \\text{ thus } \\Sigma = (W^{-1})^\\top W^{-1}\n\\end{aligned}\n$$\nIf we get\nSVD\ndecomposition of $\\Sigma = U\\Lambda U^\\top$, we will have $W^{-1}=\\sqrt{\\Lambda} U^\\top$ and  $W=U\\sqrt{\\Lambda^{-1}}$. Note that within SVD, $U$ is an orthogonal matrix with column vectors as eigenvectors and $\\Lambda$ is a diagonal matrix with all positive elements as sorted eigenvalues.\nA dimensionality reduction strategy can be applied by only taking the first $k$ columns of $W$, named\nWhitening\n-$k$.\nPseudo code of the whitening-$k$ operation. (Image source:\nSu et al. 2021\n)\nWhitening operations were shown to outperform BERT-flow and achieve SOTA with 256 sentence dimensionality on many STS benchmarks, either with or without NLI supervision.\nUnsupervised Sentence Embedding Learning\n#\nContext Prediction\n#\nQuick-Thought (QT) vectors\n(\nLogeswaran & Lee, 2018\n) formulate sentence representation learning as a\nclassification\nproblem: Given a sentence and its context, a classifier distinguishes context sentences from other contrastive sentences based on their vector representations (\n“cloze test”\n). Such a formulation removes the softmax output layer which causes training slowdown.\nIllustration of how Quick-Thought sentence embedding vectors are learned. (Image source:\nLogeswaran & Lee, 2018\n)\nLet $f(.)$ and $g(.)$ be two functions that encode a sentence $s$ into a fixed-length vector. Let $C(s)$ be the set of sentences in the context of $s$ and $S(s)$ be the set of candidate sentences including only one sentence $s_c \\in C(s)$ and many other non-context negative sentences. Quick Thoughts model learns to optimize the probability of predicting the only true context sentence $s_c \\in S(s)$. It is essentially NCE loss when considering the sentence $(s, s_c)$ as the positive pairs while other pairs $(s, s’)$ where $s’ \\in S(s), s’\\neq s_c$ as negatives.\n$$\n\\mathcal{L}_\\text{QT} \n= - \\sum_{s \\in \\mathcal{D}} \\sum_{s_c \\in C(s)} \\log p(s_c \\vert s, S(s)) \n= - \\sum_{s \\in \\mathcal{D}} \\sum_{s_c \\in C(s)}\\frac{\\exp(f(s)^\\top g(s_c))}{\\sum_{s'\\in S(s)} \\exp(f(s)^\\top g(s'))}\n$$\nMutual Information Maximization\n#\nIS-BERT (Info-Sentence BERT)\n(\nZhang et al. 2020\n;\ncode\n) adopts a self-supervised learning objective based on\nmutual information maximization\nto learn good sentence embeddings in the\nunsupervised\nmanners.\nIllustration of Info-Sentence BERT. (Image source:\nZhang et al. 2020\n)\nIS-BERT works as follows:\nUse BERT to encode an input sentence $s$ to a token embedding of length $l$, $\\mathbf{h}_{1:l}$.\nThen apply 1-D conv net with different kernel sizes (e.g. 1, 3, 5) to process the token embedding sequence to capture the n-gram local contextual dependencies: $\\mathbf{c}_i = \\text{ReLU}(\\mathbf{w} \\cdot \\mathbf{h}_{i:i+k-1} + \\mathbf{b})$. The output sequences are padded to stay the same sizes of the inputs.\nThe final local representation of the $i$-th token $\\mathcal{F}_\\theta^{(i)} (\\mathbf{x})$ is the concatenation of representations of different kernel sizes.\nThe global sentence representation $\\mathcal{E}_\\theta(\\mathbf{x})$ is computed by applying a mean-over-time pooling layer on the token representations $\\mathcal{F}_\\theta(\\mathbf{x}) = \\{\\mathcal{F}_\\theta^{(i)} (\\mathbf{x}) \\in \\mathbb{R}^d\\}_{i=1}^l$.\nSince the mutual information estimation is generally intractable for continuous and high-dimensional random variables, IS-BERT relies on the Jensen-Shannon estimator (\nNowozin et al., 2016\n,\nHjelm et al., 2019\n) to maximize the mutual information between $\\mathcal{E}_\\theta(\\mathbf{x})$ and $\\mathcal{F}_\\theta^{(i)} (\\mathbf{x})$.\n$$\nI^\\text{JSD}_\\omega(\\mathcal{F}_\\theta^{(i)} (\\mathbf{x}); \\mathcal{E}_\\theta(\\mathbf{x})) = \\mathbb{E}_{\\mathbf{x}\\sim P} [-\\text{sp}(-T_\\omega(\\mathcal{F}_\\theta^{(i)} (\\mathbf{x}); \\mathcal{E}_\\theta(\\mathbf{x})))] \\\\ - \\mathbb{E}_{\\mathbf{x}\\sim P, \\mathbf{x}' \\sim\\tilde{P}} [\\text{sp}(T_\\omega(\\mathcal{F}_\\theta^{(i)} (\\mathbf{x}'); \\mathcal{E}_\\theta(\\mathbf{x})))]\n$$\nwhere $T_\\omega: \\mathcal{F}\\times\\mathcal{E} \\to \\mathbb{R}$ is a learnable network with parameters $\\omega$, generating discriminator scores. The negative sample $\\mathbf{x}’$ is sampled from the distribution $\\tilde{P}=P$. And $\\text{sp}(x)=\\log(1+e^x)$ is the softplus activation function.\nThe unsupervised numbers on SentEval with IS-BERT outperforms most of the unsupervised baselines (Sep 2020), but unsurprisingly weaker than supervised runs. When using labelled NLI datasets, IS-BERT produces results comparable with SBERT (See Fig. 25 & 30).\nThe performance of IS-BERT on the SentEval benchmark. (Image source:\nZhang et al. 2020\n)\nCitation\n#\nCited as:\nWeng, Lilian. (May 2021). Contrastive representation learning. Lil’Log. https://lilianweng.github.io/posts/2021-05-31-contrastive/.\nOr\n@article{weng2021contrastive,\n  title   = \"Contrastive Representation Learning\",\n  author  = \"Weng, Lilian\",\n  journal = \"lilianweng.github.io\",\n  year    = \"2021\",\n  month   = \"May\",\n  url     = \"https://lilianweng.github.io/posts/2021-05-31-contrastive/\"\n}\nReferences\n#\n[1] Sumit Chopra, Raia Hadsell and Yann LeCun.\n“Learning a similarity metric discriminatively, with application to face verification.”\nCVPR 2005.\n[2] Florian Schroff, Dmitry Kalenichenko and James Philbin.\n“FaceNet: A Unified Embedding for Face Recognition and Clustering.”\nCVPR 2015.\n[3] Hyun Oh Song et al.\n“Deep Metric Learning via Lifted Structured Feature Embedding.”\nCVPR 2016. [\ncode\n]\n[4] Ruslan Salakhutdinov and Geoff Hinton.\n“Learning a Nonlinear Embedding by Preserving Class Neighbourhood Structure”\nAISTATS 2007.\n[5] Michael Gutmann and Aapo Hyvärinen.\n“Noise-contrastive estimation: A new estimation principle for unnormalized statistical models.”\nAISTATS 2010.\n[6] Kihyuk Sohn et al.\n“Improved Deep Metric Learning with Multi-class N-pair Loss Objective”\nNIPS 2016.\n[7] Nicholas Frosst, Nicolas Papernot and Geoffrey Hinton.\n“Analyzing and Improving Representations with the Soft Nearest Neighbor Loss.”\nICML 2019\n[8] Tongzhou Wang and Phillip Isola.\n“Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere.”\nICML 2020. [\ncode\n]\n[9] Zhirong Wu et al.\n“Unsupervised feature learning via non-parametric instance-level discrimination.”\nCVPR 2018.\n[10] Ekin D. Cubuk et al.\n“AutoAugment: Learning augmentation policies from data.”\narXiv preprint arXiv:1805.09501 (2018).\n[11] Daniel Ho et al.\n“Population Based Augmentation: Efficient Learning of Augmentation Policy Schedules.”\nICML 2019.\n[12] Ekin D. Cubuk & Barret Zoph et al.\n“RandAugment: Practical automated data augmentation with a reduced search space.”\narXiv preprint arXiv:1909.13719 (2019).\n[13] Hongyi Zhang et al.\n“mixup: Beyond Empirical Risk Minimization.”\nICLR 2017.\n[14] Sangdoo Yun et al.\n“CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Features.”\nICCV 2019.\n[15] Yannis Kalantidis et al.\n“Mixing of Contrastive Hard Negatives”\nNeuriPS 2020.\n[16] Ashish Jaiswal et al.\n“A Survey on Contrastive Self-Supervised Learning.”\narXiv preprint arXiv:2011.00362 (2021)\n[17] Jure Zbontar et al.\n“Barlow Twins: Self-Supervised Learning via Redundancy Reduction.”\narXiv preprint arXiv:2103.03230 (2021) [\ncode\n]\n[18] Alec Radford, et al.\n“Learning Transferable Visual Models From Natural Language Supervision”\narXiv preprint arXiv:2103.00020 (2021)\n[19] Mathilde Caron et al.\n“Unsupervised Learning of Visual Features by Contrasting Cluster Assignments (SwAV).”\nNeuriPS 2020.\n[20] Mathilde Caron et al.\n“Deep Clustering for Unsupervised Learning of Visual Features.”\nECCV 2018.\n[21] Prannay Khosla et al.\n“Supervised Contrastive Learning.”\nNeurIPS 2020.\n[22] Aaron van den Oord, Yazhe Li & Oriol Vinyals.\n“Representation Learning with Contrastive Predictive Coding”\narXiv preprint arXiv:1807.03748 (2018).\n[23] Jason Wei and Kai Zou.\n“EDA: Easy data augmentation techniques for boosting performance on text classification tasks.”\nEMNLP-IJCNLP 2019.\n[24] Sosuke Kobayashi.\n“Contextual Augmentation: Data Augmentation by Words with Paradigmatic Relations.”\nNAACL 2018\n[25] Hongchao Fang et al.\n“CERT: Contrastive self-supervised learning for language understanding.”\narXiv preprint arXiv:2005.12766 (2020).\n[26] Dinghan Shen et al.\n“A Simple but Tough-to-Beat Data Augmentation Approach for Natural Language Understanding and Generation.”\narXiv preprint arXiv:2009.13818 (2020) [\ncode\n]\n[27] Tianyu Gao et al.\n“SimCSE: Simple Contrastive Learning of Sentence Embeddings.”\narXiv preprint arXiv:2104.08821 (2020). [\ncode\n]\n[28] Nils Reimers and Iryna Gurevych.\n“Sentence-BERT: Sentence embeddings using Siamese BERT-networks.”\nEMNLP 2019.\n[29] Jianlin Su et al.\n“Whitening sentence representations for better semantics and faster retrieval.”\narXiv preprint arXiv:2103.15316 (2021). [\ncode\n]\n[30] Yan Zhang et al.\n“An unsupervised sentence embedding method by mutual information maximization.”\nEMNLP 2020. [\ncode\n]\n[31] Bohan Li et al.\n“On the sentence embeddings from pre-trained language models.”\nEMNLP 2020.\n[32] Lajanugen Logeswaran and Honglak Lee.\n“An efficient framework for learning sentence representations.”\nICLR 2018.\n[33] Joshua Robinson, et al.\n“Contrastive Learning with Hard Negative Samples.”\nICLR 2021.\n[34] Ching-Yao Chuang et al.\n“Debiased Contrastive Learning.”\nNeuriPS 2020.",
  "metadata": {
    "title": "Contrastive Representation Learning | Lil'Log",
    "description": "The goal of contrastive representation learning is to learn such an embedding space in which similar sample pairs stay close to each other while dissimilar ones are far apart. Contrastive learning can be applied to both supervised and unsupervised settings. When working with unsupervised data, contrastive learning is one of the most powerful approaches in self-supervised learning.",
    "keywords": "representation-learning, long-read, language-model, unsupervised-learning, data-augmentation",
    "author": "Lilian Weng",
    "og:title": "Contrastive Representation Learning",
    "og:description": "The goal of contrastive representation learning is to learn such an embedding space in which similar sample pairs stay close to each other while dissimilar ones are far apart. Contrastive learning can be applied to both supervised and unsupervised settings. When working with unsupervised data, contrastive learning is one of the most powerful approaches in self-supervised learning.",
    "og:type": "article",
    "og:url": "https://lilianweng.github.io/posts/2021-05-31-contrastive/",
    "twitter:card": "summary",
    "twitter:title": "Contrastive Representation Learning",
    "twitter:description": "The goal of contrastive representation learning is to learn such an embedding space in which similar sample pairs stay close to each other while dissimilar ones are far apart. Contrastive learning can be applied to both supervised and unsupervised settings. When working with unsupervised data, contrastive learning is one of the most powerful approaches in self-supervised learning.",
    "article:section": "posts",
    "article:published_time": "2021-05-31T00:00:00+00:00",
    "article:modified_time": "2021-05-31T00:00:00+00:00"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [
    {
      "file": "attachments/c8f3f2a59bb8/code/body-000.txt",
      "chars": 259,
      "language": "text"
    }
  ],
  "images": [
    {
      "status": "ok",
      "url": "https://lilianweng.github.io/posts/2021-05-31-contrastive/triplet-loss.png",
      "bytes": 82669,
      "reused": false,
      "local_path": "attachments/c8f3f2a59bb8/images/body-c8feb8e3831d6f407222.png",
      "src": "https://lilianweng.github.io/posts/2021-05-31-contrastive/triplet-loss.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://lilianweng.github.io/posts/2021-05-31-contrastive/lifted-structured-loss.png",
      "bytes": 238558,
      "reused": false,
      "local_path": "attachments/c8f3f2a59bb8/images/body-864616500d13b290a57d.png",
      "src": "https://lilianweng.github.io/posts/2021-05-31-contrastive/lifted-structured-loss.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://lilianweng.github.io/posts/2021-05-31-contrastive/contrastive-sampling-bias.png",
      "bytes": 618000,
      "reused": false,
      "local_path": "attachments/c8f3f2a59bb8/images/body-4d1eafbd2698e4b13414.png",
      "src": "https://lilianweng.github.io/posts/2021-05-31-contrastive/contrastive-sampling-bias.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://lilianweng.github.io/posts/2021-05-31-contrastive/contrastive-debias-t-SNE.png",
      "bytes": 447800,
      "reused": false,
      "local_path": "attachments/c8f3f2a59bb8/images/body-2202dbf059680ff25f64.png",
      "src": "https://lilianweng.github.io/posts/2021-05-31-contrastive/contrastive-debias-t-SNE.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://lilianweng.github.io/posts/2021-05-31-contrastive/contrastive-hard-negatives-code.png",
      "bytes": 331316,
      "reused": false,
      "local_path": "attachments/c8f3f2a59bb8/images/body-56fdc531021fd8fc266d.png",
      "src": "https://lilianweng.github.io/posts/2021-05-31-contrastive/contrastive-hard-negatives-code.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://lilianweng.github.io/posts/2021-05-31-contrastive/SimCLR.png",
      "bytes": 92328,
      "reused": false,
      "local_path": "attachments/c8f3f2a59bb8/images/body-a3de29261702db65276c.png",
      "src": "https://lilianweng.github.io/posts/2021-05-31-contrastive/SimCLR.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://lilianweng.github.io/posts/2021-05-31-contrastive/SimCLR-algo.png",
      "bytes": 292083,
      "reused": false,
      "local_path": "attachments/c8f3f2a59bb8/images/body-9fbb7be242e78cf4064e.png",
      "src": "https://lilianweng.github.io/posts/2021-05-31-contrastive/SimCLR-algo.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://lilianweng.github.io/posts/2021-05-31-contrastive/barlow-twins.png",
      "bytes": 165448,
      "reused": false,
      "local_path": "attachments/c8f3f2a59bb8/images/body-7c6e570b2b4acaea53ee.png",
      "src": "https://lilianweng.github.io/posts/2021-05-31-contrastive/barlow-twins.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://lilianweng.github.io/posts/2021-05-31-contrastive/barlow-twins-algo.png",
      "bytes": 410905,
      "reused": false,
      "local_path": "attachments/c8f3f2a59bb8/images/body-0dc7fb2e267a076918a0.png",
      "src": "https://lilianweng.github.io/posts/2021-05-31-contrastive/barlow-twins-algo.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://lilianweng.github.io/posts/2021-05-31-contrastive/BYOL.png",
      "bytes": 160971,
      "reused": false,
      "local_path": "attachments/c8f3f2a59bb8/images/body-64c06b583b29bc8d849d.png",
      "src": "https://lilianweng.github.io/posts/2021-05-31-contrastive/BYOL.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://lilianweng.github.io/posts/2021-05-31-contrastive/instance-level-discrimination.png",
      "bytes": 575598,
      "reused": false,
      "local_path": "attachments/c8f3f2a59bb8/images/body-24abca83dac665bdfd2c.png",
      "src": "https://lilianweng.github.io/posts/2021-05-31-contrastive/instance-level-discrimination.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://lilianweng.github.io/posts/2021-05-31-contrastive/MoCo.png",
      "bytes": 191327,
      "reused": false,
      "local_path": "attachments/c8f3f2a59bb8/images/body-68a57a589ced1a8dcbeb.png",
      "src": "https://lilianweng.github.io/posts/2021-05-31-contrastive/MoCo.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://lilianweng.github.io/posts/2021-05-31-contrastive/CURL.png",
      "bytes": 458778,
      "reused": false,
      "local_path": "attachments/c8f3f2a59bb8/images/body-dec760a7d6d875b1e5f6.png",
      "src": "https://lilianweng.github.io/posts/2021-05-31-contrastive/CURL.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://lilianweng.github.io/posts/2021-05-31-contrastive/deepcluster.png",
      "bytes": 413067,
      "reused": false,
      "local_path": "attachments/c8f3f2a59bb8/images/body-3b05f2c9b09900dd678c.png",
      "src": "https://lilianweng.github.io/posts/2021-05-31-contrastive/deepcluster.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://lilianweng.github.io/posts/2021-05-31-contrastive/SwAV.png",
      "bytes": 307781,
      "reused": false,
      "local_path": "attachments/c8f3f2a59bb8/images/body-9fe78703c0d6dd59901b.png",
      "src": "https://lilianweng.github.io/posts/2021-05-31-contrastive/SwAV.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://lilianweng.github.io/posts/2021-05-31-contrastive/CLIP.png",
      "bytes": 395407,
      "reused": false,
      "local_path": "attachments/c8f3f2a59bb8/images/body-bcffab9e56e403b12eb6.png",
      "src": "https://lilianweng.github.io/posts/2021-05-31-contrastive/CLIP.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://lilianweng.github.io/posts/2021-05-31-contrastive/CLIP-algo.png",
      "bytes": 378369,
      "reused": false,
      "local_path": "attachments/c8f3f2a59bb8/images/body-14fc3bb2a028bd633430.png",
      "src": "https://lilianweng.github.io/posts/2021-05-31-contrastive/CLIP-algo.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://lilianweng.github.io/posts/2021-05-31-contrastive/CLIP-efficiency.png",
      "bytes": 269518,
      "reused": false,
      "local_path": "attachments/c8f3f2a59bb8/images/body-1cdb06387d4aabe19e7f.png",
      "src": "https://lilianweng.github.io/posts/2021-05-31-contrastive/CLIP-efficiency.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://lilianweng.github.io/posts/2021-05-31-contrastive/sup-con.png",
      "bytes": 1172082,
      "reused": false,
      "local_path": "attachments/c8f3f2a59bb8/images/body-dc3294612801cee60ee0.png",
      "src": "https://lilianweng.github.io/posts/2021-05-31-contrastive/sup-con.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://lilianweng.github.io/posts/2021-05-31-contrastive/EDA-exp1.png",
      "bytes": 299081,
      "reused": false,
      "local_path": "attachments/c8f3f2a59bb8/images/body-4ecf4ae1a9c6c75e0b98.png",
      "src": "https://lilianweng.github.io/posts/2021-05-31-contrastive/EDA-exp1.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://lilianweng.github.io/posts/2021-05-31-contrastive/text-cutoff.png",
      "bytes": 102477,
      "reused": false,
      "local_path": "attachments/c8f3f2a59bb8/images/body-2f938fd27f3112a7668b.png",
      "src": "https://lilianweng.github.io/posts/2021-05-31-contrastive/text-cutoff.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://lilianweng.github.io/posts/2021-05-31-contrastive/SimCSE.png",
      "bytes": 302804,
      "reused": false,
      "local_path": "attachments/c8f3f2a59bb8/images/body-476b14449fa38214486c.png",
      "src": "https://lilianweng.github.io/posts/2021-05-31-contrastive/SimCSE.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://lilianweng.github.io/posts/2021-05-31-contrastive/SimCSE-STS-exp.png",
      "bytes": 459618,
      "reused": false,
      "local_path": "attachments/c8f3f2a59bb8/images/body-d417f179aa932198443f.png",
      "src": "https://lilianweng.github.io/posts/2021-05-31-contrastive/SimCSE-STS-exp.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://lilianweng.github.io/posts/2021-05-31-contrastive/SBERT.png",
      "bytes": 248555,
      "reused": false,
      "local_path": "attachments/c8f3f2a59bb8/images/body-61290a159906f77df170.png",
      "src": "https://lilianweng.github.io/posts/2021-05-31-contrastive/SBERT.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://lilianweng.github.io/posts/2021-05-31-contrastive/SBERT-SentEval.png",
      "bytes": 263986,
      "reused": false,
      "local_path": "attachments/c8f3f2a59bb8/images/body-25636cdf26fe48bc32a6.png",
      "src": "https://lilianweng.github.io/posts/2021-05-31-contrastive/SBERT-SentEval.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://lilianweng.github.io/posts/2021-05-31-contrastive/BERT-flow.png",
      "bytes": 127994,
      "reused": false,
      "local_path": "attachments/c8f3f2a59bb8/images/body-1f6149bb39a134da1e75.png",
      "src": "https://lilianweng.github.io/posts/2021-05-31-contrastive/BERT-flow.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://lilianweng.github.io/posts/2021-05-31-contrastive/whitening-SBERT.png",
      "bytes": 167429,
      "reused": false,
      "local_path": "attachments/c8f3f2a59bb8/images/body-75a0434d42ea9e13e5f2.png",
      "src": "https://lilianweng.github.io/posts/2021-05-31-contrastive/whitening-SBERT.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://lilianweng.github.io/posts/2021-05-31-contrastive/quick-thought.png",
      "bytes": 235880,
      "reused": false,
      "local_path": "attachments/c8f3f2a59bb8/images/body-0873d959165b59705852.png",
      "src": "https://lilianweng.github.io/posts/2021-05-31-contrastive/quick-thought.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://lilianweng.github.io/posts/2021-05-31-contrastive/IS-BERT.png",
      "bytes": 403903,
      "reused": false,
      "local_path": "attachments/c8f3f2a59bb8/images/body-7fc9d0b8fcf86efd800b.png",
      "src": "https://lilianweng.github.io/posts/2021-05-31-contrastive/IS-BERT.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://lilianweng.github.io/posts/2021-05-31-contrastive/IS-BERT-SentEval.png",
      "bytes": 237372,
      "reused": false,
      "local_path": "attachments/c8f3f2a59bb8/images/body-4a85aecfe0dc34b27289.png",
      "src": "https://lilianweng.github.io/posts/2021-05-31-contrastive/IS-BERT-SentEval.png",
      "alt": "",
      "downloaded": true
    }
  ],
  "audio": [],
  "videos": [],
  "links": [
    {
      "href": "https://lilianweng.github.io/posts/2019-11-10-self-supervised/",
      "label": "self-supervised learning",
      "type": "a"
    },
    {
      "href": "http://yann.lecun.com/exdb/publis/pdf/chopra-05.pdf",
      "label": "Chopra et al. 2005",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1503.03832",
      "label": "Schroff et al. 2015",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1511.06452",
      "label": "Song et al. 2015",
      "type": "a"
    },
    {
      "href": "https://papers.nips.cc/paper/2016/hash/6b180037abbebea991d8b1232f8a8ca9-Abstract.html",
      "label": "Sohn 2016",
      "type": "a"
    },
    {
      "href": "http://proceedings.mlr.press/v9/gutmann10a.html",
      "label": "Gutmann & Hyvarinen",
      "type": "a"
    },
    {
      "href": "https://lilianweng.github.io/posts/2017-10-15-word-embedding/",
      "label": "here",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1807.03748",
      "label": "van den Oord, et al. 2018",
      "type": "a"
    },
    {
      "href": "http://proceedings.mlr.press/v2/salakhutdinov07a.html",
      "label": "Salakhutdinov & Hinton 2007",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1902.01889",
      "label": "Frosst et al. 2019",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2005.10242",
      "label": "Wang & Isola 2020",
      "type": "a"
    },
    {
      "href": "https://lilianweng.github.io/posts/2020-10-29-odqa/",
      "label": "DPR",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2004.04906",
      "label": "Karpukhin et al., 2020",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2007.00224",
      "label": "Chuang et al. (2020)",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2010.04592",
      "label": "Robinson et al. (2021)",
      "type": "a"
    },
    {
      "href": "https://lilianweng.github.io/posts/2019-05-05-domain-randomization/",
      "label": "AutoAugment",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1805.09501",
      "label": "Cubuk, et al. 2018",
      "type": "a"
    },
    {
      "href": "https://lilianweng.github.io/posts/2020-08-06-nas/",
      "label": "NAS",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1909.13719",
      "label": "Cubuk et al., 2019",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1905.05393",
      "label": "Ho et al., 2019",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1711.09846",
      "label": "Jaderberg et al, 2017",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1904.12848",
      "label": "Xie et al., 2019",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1710.09412",
      "label": "Zhang et al., 2018",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1905.04899",
      "label": "Yun et al., 2019",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1708.04552",
      "label": "DeVries & Taylor 2017",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2010.01028",
      "label": "Kalantidis et al. 2020",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2002.05709",
      "label": "Chen et al, 2020",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2103.03230",
      "label": "Zbontar et al. 2021",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2006.07733",
      "label": "Grill, et al 2020",
      "type": "a"
    },
    {
      "href": "https://untitled-ai.github.io/understanding-self-supervised-contrastive-learning.html",
      "label": "post",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1805.01978v1",
      "label": "Wu et al, 2018",
      "type": "a"
    },
    {
      "href": "https://web.stanford.edu/~boyd/papers/prox_algs.html",
      "label": "proximal optimization method",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1911.05722",
      "label": "He et al, 2019",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2003.04297",
      "label": "Chen et al, 2020",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2004.04136",
      "label": "Srinivas, et al. 2020",
      "type": "a"
    },
    {
      "href": "https://lilianweng.github.io/posts/2018-02-19-rl-overview/",
      "label": "Reinforcement Learning",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1807.05520",
      "label": "Caron et al. 2018",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2006.09882",
      "label": "Caron et al. 2020",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1306.0895",
      "label": "Cuturi 2013",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2103.00020",
      "label": "Radford et al. 2021",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2004.11362",
      "label": "Khosla et al. 2021",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1901.11196",
      "label": "Wei & Zou 2019",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1805.06201",
      "label": "Sosuke Kobayashi, 2018",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2005.12766",
      "label": "Fang et al. (2020)",
      "type": "a"
    },
    {
      "href": "https://github.com/UCSD-AI4H/CERT",
      "label": "code",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2009.13818",
      "label": "Shen et al. (2020)",
      "type": "a"
    },
    {
      "href": "https://lilianweng.github.io/posts/2019-01-31-lm/",
      "label": "cross-view training",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2104.08821",
      "label": "Gao et al. 2021",
      "type": "a"
    },
    {
      "href": "https://github.com/princeton-nlp/SimCSE",
      "label": "code",
      "type": "a"
    },
    {
      "href": "https://nlp.stanford.edu/projects/snli/",
      "label": "SNLI",
      "type": "a"
    },
    {
      "href": "https://cims.nyu.edu/~sbowman/multinli/",
      "label": "MNLI",
      "type": "a"
    },
    {
      "href": "https://www.kaggle.com/c/quora-question-pairs",
      "label": "QQP",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1908.10084",
      "label": "Reimers & Gurevych, 2019",
      "type": "a"
    },
    {
      "href": "https://github.com/facebookresearch/SentEval",
      "label": "SentEval",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1803.05449",
      "label": "Conneau and Kiela, 2018",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2011.05864",
      "label": "Li et al, (2020)",
      "type": "a"
    },
    {
      "href": "https://github.com/bohanli/BERT-flow",
      "label": "code",
      "type": "a"
    },
    {
      "href": "https://lilianweng.github.io/posts/2018-10-13-flow-models/",
      "label": "normalizing flows",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2103.15316",
      "label": "Su et al. (2021)",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/Singular_value_decomposition",
      "label": "SVD",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1803.02893",
      "label": "Logeswaran & Lee, 2018",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2009.12061",
      "label": "Zhang et al. 2020",
      "type": "a"
    },
    {
      "href": "https://github.com/yanzhangnlp/IS-BERT",
      "label": "code",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1606.00709",
      "label": "Nowozin et al., 2016",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1808.06670",
      "label": "Hjelm et al., 2019",
      "type": "a"
    },
    {
      "href": "https://github.com/rksltnl/Deep-Metric-Learning-CVPR16",
      "label": "code",
      "type": "a"
    },
    {
      "href": "http://proceedings.mlr.press/v97/frosst19a.html",
      "label": "“Analyzing and Improving Representations with the Soft Nearest Neighbor Loss.”",
      "type": "a"
    },
    {
      "href": "https://ssnl.github.io/hypersphere/",
      "label": "code",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1805.01978",
      "label": "“Unsupervised feature learning via non-parametric instance-level discrimination.”",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2011.00362",
      "label": "“A Survey on Contrastive Self-Supervised Learning.”",
      "type": "a"
    },
    {
      "href": "https://github.com/facebookresearch/barlowtwins",
      "label": "code",
      "type": "a"
    },
    {
      "href": "https://github.com/dinghanshen/cutoff",
      "label": "code",
      "type": "a"
    },
    {
      "href": "https://github.com/bojone/BERT-whitening",
      "label": "code",
      "type": "a"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-14T10:51:20Z",
  "source_url": "https://lilianweng.github.io/posts/2021-05-31-contrastive/",
  "papers": [
    {
      "status": "ok",
      "url": "http://yann.lecun.com/exdb/publis/pdf/chopra-05.pdf",
      "bytes": 313804,
      "reused": false,
      "local_path": "papers/repair/84881b743b9447eb9afe.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1503.03832",
      "bytes": 4705274,
      "reused": false,
      "local_path": "papers/repair/a71730a10b0aef666206.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1511.06452",
      "bytes": 9011569,
      "reused": false,
      "local_path": "papers/repair/738c3f04b7e4d7dad3a9.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1807.03748",
      "bytes": 3182089,
      "reused": false,
      "local_path": "papers/repair/7ace1819f0b552bee2ab.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1902.01889",
      "bytes": 3823806,
      "reused": false,
      "local_path": "papers/repair/66ab4748bfca0238682e.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2005.10242",
      "bytes": 4185717,
      "reused": false,
      "local_path": "papers/repair/8a0c585591f7f1918d90.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2004.04906",
      "bytes": 383508,
      "reused": true,
      "local_path": "papers/repair/8b76986dbe24871ab2c9.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2007.00224",
      "bytes": 5343399,
      "reused": false,
      "local_path": "papers/repair/7dcaf70ed91e5defcf23.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2010.04592",
      "bytes": 2199907,
      "reused": false,
      "local_path": "papers/repair/3f84b66b72f5c381c05a.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1805.09501",
      "bytes": 1138440,
      "reused": true,
      "local_path": "papers/repair/9a0542ec64439d672cc9.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1909.13719",
      "bytes": 1263406,
      "reused": true,
      "local_path": "papers/repair/c267ff367c4770ddb6c0.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1905.05393",
      "bytes": 873126,
      "reused": true,
      "local_path": "papers/repair/4634b6280b855d570ac8.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1711.09846",
      "bytes": 2370925,
      "reused": true,
      "local_path": "papers/repair/3bbdbf9c8931b9ea9ea0.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1904.12848",
      "bytes": 1625724,
      "reused": true,
      "local_path": "papers/repair/17af125202b45606d9ad.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1710.09412",
      "bytes": 740562,
      "reused": true,
      "local_path": "papers/repair/7af2d52fbf38a69ec251.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1905.04899",
      "bytes": 1682836,
      "reused": true,
      "local_path": "papers/repair/fec62c4fe8efdeebd877.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1708.04552",
      "bytes": 741919,
      "reused": true,
      "local_path": "papers/repair/c3d681ed19bf88f43a98.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2010.01028",
      "bytes": 970998,
      "reused": true,
      "local_path": "papers/repair/d4d3c2412459021d347a.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2002.05709",
      "bytes": 6564895,
      "reused": false,
      "local_path": "papers/repair/702dc85ba981458cb06f.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2103.03230",
      "bytes": 2434163,
      "reused": false,
      "local_path": "papers/repair/7f141d9ed4155ccf53f5.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2006.07733",
      "bytes": 4837213,
      "reused": false,
      "local_path": "papers/repair/7c03b7b76959caf6d3ab.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1805.01978v1",
      "bytes": 4452342,
      "reused": false,
      "local_path": "papers/repair/e90b74cda724c45827a6.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1911.05722",
      "bytes": 593470,
      "reused": false,
      "local_path": "papers/repair/5c3c001fd908f6dae2c6.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2003.04297",
      "bytes": 262792,
      "reused": false,
      "local_path": "papers/repair/43fc2eaa495db084a96e.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2004.04136",
      "bytes": 4108339,
      "reused": false,
      "local_path": "papers/repair/c9a2ff8f5b0e5380a701.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1807.05520",
      "bytes": 8629266,
      "reused": false,
      "local_path": "papers/repair/0a5b97b834b6a105dd9e.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2006.09882",
      "bytes": 702914,
      "reused": false,
      "local_path": "papers/repair/0838c337bef6ccd33d62.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1306.0895",
      "bytes": 1004298,
      "reused": false,
      "local_path": "papers/repair/598da2f78a40c9c822ff.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2103.00020",
      "bytes": 6813639,
      "reused": true,
      "local_path": "papers/repair/7db0b9244bb1fd42ae98.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2004.11362",
      "bytes": 2197586,
      "reused": false,
      "local_path": "papers/repair/a64eec551d692894bad9.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1901.11196",
      "bytes": 365049,
      "reused": true,
      "local_path": "papers/repair/389e34d78a0045e6d861.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1805.06201",
      "bytes": 722323,
      "reused": true,
      "local_path": "papers/repair/aebd3f77254c72e012a3.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2005.12766",
      "bytes": 412104,
      "reused": false,
      "local_path": "papers/repair/bae8e6b74f13390eb0a6.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2009.13818",
      "bytes": 398629,
      "reused": true,
      "local_path": "papers/repair/40a2d9b786e78e3cf30e.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2104.08821",
      "bytes": 1672580,
      "reused": true,
      "local_path": "papers/repair/e7a40652bdd04cd4b25f.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1908.10084",
      "bytes": 548961,
      "reused": true,
      "local_path": "papers/repair/20151599fbbbabacaa9e.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1803.05449",
      "bytes": 1257905,
      "reused": false,
      "local_path": "papers/repair/748bb3642ea0aef5571f.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2011.05864",
      "bytes": 408978,
      "reused": false,
      "local_path": "papers/repair/3bfe284fc1ebd8a9d3a3.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2103.15316",
      "bytes": 659541,
      "reused": false,
      "local_path": "papers/repair/6e8262dafcfd96e9b679.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1803.02893",
      "bytes": 232651,
      "reused": false,
      "local_path": "papers/repair/12ee892560b48ed47fb8.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2009.12061",
      "bytes": 408861,
      "reused": false,
      "local_path": "papers/repair/5c6f2a504c1b0ff66969.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1606.00709",
      "bytes": 3615251,
      "reused": false,
      "local_path": "papers/repair/b555d3a1dac849208314.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1808.06670",
      "bytes": 4984410,
      "reused": false,
      "local_path": "papers/repair/ca0c4e18cba031b31bd0.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1805.01978",
      "bytes": 4452342,
      "reused": false,
      "local_path": "papers/repair/2408633e99d87285a643.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2011.00362",
      "bytes": 5998084,
      "reused": false,
      "local_path": "papers/repair/20fd1312b2e030a81069.pdf"
    }
  ],
  "actual_video_files": [],
  "body_chars": 62279,
  "expected_body_images": 30,
  "discovery_status": "ok",
  "delivery_status": "ok",
  "missing_reason": "",
  "raw_html": "attachments/c8f3f2a59bb8/pages/source.html",
  "local_html": "attachments/c8f3f2a59bb8/pages/article.html",
  "body_base_path": "attachments/c8f3f2a59bb8/pages/",
  "body_markdown_path": "attachments/c8f3f2a59bb8/pages/article.md"
}