MPNet
Masked and Permuted Pre-training for Language Understanding
About MPNet
MPNet is a pre-training technique for language models which combines the concepts of masked language modeling (MLM) and permuted language modeling (PLM). It takes into account the relationship between the words that are predicted by using PLM, thus avoiding the issue of BERT.