Publication Details
Extensions of Recurrent Neural Network Language Model
Kombrink Stefan, Dipl.-Inf -Ling (DCGM FIT BUT)
Burget Lukáš, doc. Ing., Ph.D. (DCGM FIT BUT)
Černocký Jan, prof. Dr. Ing. (DCGM FIT BUT)
Khudanpur Sanjeev (JHU)
language modeling, recurrent neural networks, speech recognition
This paper describes results that we obtained when using extensions of Recurrent Neural Network (RNN) Language Model.
We present several modifications of the original recurrent neural network language model (RNN LM).While this model has been shown to significantly outperform many competitive language modeling techniques in terms of accuracy, the remaining problem is the computational complexity. In this work, we show approaches that lead to more than 15 times speedup for both training and testing phases. Next, we show importance of using a backpropagation through time algorithm. An empirical comparison with feedforward networks is also provided. In the end, we discuss possibilities how to reduce the amount of parameters in the model. The resulting RNN model can thus be smaller, faster both during training and testing, and more accurate than the basic one.
@INPROCEEDINGS{FITPUB9658,
author = "Tom\'{a}\v{s} Mikolov and Stefan Kombrink and Luk\'{a}\v{s} Burget and Jan \v{C}ernock\'{y} and Sanjeev Khudanpur",
title = "Extensions of Recurrent Neural Network Language Model",
pages = "5528--5531",
booktitle = "Proceedings of the 2011 IEEE International Conference on Acoustics, Speech, and Signal Processing, ICASSP 2011",
year = 2011,
location = "Praha, CZ",
publisher = "IEEE Signal Processing Society",
ISBN = "978-1-4577-0537-3",
language = "english",
url = "https://www.fit.vut.cz/research/publication/9658"
}