论文标题
通过基于概念的课程蒙版对蒙版语言模型的有效预培训
Efficient Pre-training of Masked Language Model via Concept-based Curriculum Masking
论文作者
论文摘要
蒙面语言建模（MLM）已被广泛用于培训前有效的双向表示，但会产生大量的培训成本。在本文中，我们提出了一种基于概念的新型课程掩盖（CCM）方法，以有效地预先培训语言模型。 CCM与现有课程学习方法有两个关键区别，以有效反映MLM的性质。首先，我们引入了精心设计的语言难度标准，该标准评估了每个令牌的MLM难度。其次，我们构建了一个课程，该课程逐渐通过检索知识图逐渐掩盖了与先前掩盖的单词相关的单词。实验结果表明，CCM显着提高了训练前效率。具体而言，接受CCM训练的模型显示了与原始BERT在一半培训成本的一半理解评估基准上的原始BERT的比较性能。
Masked language modeling (MLM) has been widely used for pre-training effective bidirectional representations, but incurs substantial training costs. In this paper, we propose a novel concept-based curriculum masking (CCM) method to efficiently pre-train a language model. CCM has two key differences from existing curriculum learning approaches to effectively reflect the nature of MLM. First, we introduce a carefully-designed linguistic difficulty criterion that evaluates the MLM difficulty of each token. Second, we construct a curriculum that gradually masks words related to the previously masked words by retrieving a knowledge graph. Experimental results show that CCM significantly improves pre-training efficiency. Specifically, the model trained with CCM shows comparative performance with the original BERT on the General Language Understanding Evaluation benchmark at half of the training cost.
