论文标题
生成还是对比？改良的短语重建，以进行更好的句子表示学习
Generative or Contrastive? Phrase Reconstruction for Better Sentence Representation Learning
论文作者
论文摘要
尽管提供了令人惊叹的上下文化令牌级表示，但当前的预训练的语言模型实际上对在其自我监督的预训练期间获得句子级表示的关注较少。如果可以将自我监督的学习分为两个子类别，即生成和对比，那么大多数现有研究表明，句子表示学习可能会从对比度方法中受益，而不是生成方法。但是，对比度学习不能与共同的令牌生成的自我监督学习兼容，并且不能保证在下游语义检索任务上的良好表现。因此，为了减轻这种明显的不便，我们改为提出了一个基于短语重建的新颖生成的自我监督学习目标。实证研究表明，我们的生成学习可能会产生足够的句子表示，并在句子文本相似性（STS）任务中实现与对比度学习相当的绩效。此外，就无监督的设置而言，我们的生成方法在下游语义检索任务的基准上优于先前最先进的SIMCSE。
Though offering amazing contextualized token-level representations, current pre-trained language models actually take less attention on acquiring sentence-level representation during its self-supervised pre-training. If self-supervised learning can be distinguished into two subcategories, generative and contrastive, then most existing studies show that sentence representation learning may more benefit from the contrastive methods but not the generative methods. However, contrastive learning cannot be well compatible with the common token-level generative self-supervised learning, and does not guarantee good performance on downstream semantic retrieval tasks. Thus, to alleviate such obvious inconveniences, we instead propose a novel generative self-supervised learning objective based on phrase reconstruction. Empirical studies show that our generative learning may yield powerful enough sentence representation and achieve performance in Sentence Textual Similarity (STS) tasks on par with contrastive learning. Further, in terms of unsupervised setting, our generative method outperforms previous state-of-the-art SimCSE on the benchmark of downstream semantic retrieval tasks.
