Speeding Up Maximum Entropy Training with Class-Based Modeling

Title: Speeding Up Maximum Entropy Training with Class-Based Modeling

Research Question: How can we speed up the training of maximum entropy models by using class-based modeling?

Methodology: The researchers proposed a novel technique to speed up the training of maximum entropy models. Instead of predicting words directly, they first predicted the class that the next word belongs to, and then predicted the word itself, conditioned on its class. This technique is more general and can be applied to any problem with a large number of outputs, such as language modeling.

Results: The researchers found that this technique significantly reduced training time, by up to a factor of 35. They compared their technique to previous research and found that it applied to a wider range of machine learning techniques and problems. They also provided experimental results showing both the increased speed of training and a slight reduction in perplexity of the resulting models.

Implications: This technique can be applied to other machine learning techniques and problems that involve a large number of outputs. It can also be used to speed up the training of other maximum entropy models, making them more accessible for research and development.

Link to Article: https://arxiv.org/abs/0108006v1 Authors: arXiv ID: 0108006v1