TY - GEN
T1 - An experimental analysis of Noise-Contrastive Estimation
T2 - 15th Conference of the European Chapter of the Association for Computational Linguistics, EACL 2017
AU - Labeau, Matthieu
AU - Allauzen, Alexandre
N1 - Publisher Copyright:
© 2017 Association for Computational Linguistics.
PY - 2017/1/1
Y1 - 2017/1/1
N2 - Noise Contrastive Estimation (NCE) is a learning procedure that is regularly used to train neural language models, since it avoids the computational bottleneck caused by the output softmax. In this paper, we attempt to explain some of the weaknesses of this objective function, and to draw directions for further developments. Experiments on a small task show the issues raised by the unigram noise distribution, and that a context dependent noise distribution, such as the bigram distribution, can solve these issues and provide stable and data-efficient learning.
AB - Noise Contrastive Estimation (NCE) is a learning procedure that is regularly used to train neural language models, since it avoids the computational bottleneck caused by the output softmax. In this paper, we attempt to explain some of the weaknesses of this objective function, and to draw directions for further developments. Experiments on a small task show the issues raised by the unigram noise distribution, and that a context dependent noise distribution, such as the bigram distribution, can solve these issues and provide stable and data-efficient learning.
U2 - 10.18653/v1/e17-2003
DO - 10.18653/v1/e17-2003
M3 - Conference contribution
AN - SCOPUS:85021688542
T3 - 15th Conference of the European Chapter of the Association for Computational Linguistics, EACL 2017 - Proceedings of Conference
SP - 15
EP - 20
BT - Short Papers
PB - Association for Computational Linguistics (ACL)
Y2 - 3 April 2017 through 7 April 2017
ER -