MPNet

MPNet is a pre-training technique for language models which combines the concepts of masked language modeling (MLM) and permuted language modeling (PLM). It takes into account the relationship between the words that are predicted by using PLM, thus avoiding the issue of BERT.