How to develop small workable embeddings for debugging

Question

Processing language data with deep learning models often involves a lookup of a pre-trained embedding model. In the model development phase, it's very annoying that every time the entire embeddings are loaded, as embeddings could be very big (e.g., Glove) and would consume a lot of timing loading it.

Is there a way to build/find a smaller and workable embedding just for debugging?

Right now what I've done is shrink the dimensions of Glove, for example, reduce the dimension of embeddings from 300d to 30d by taking the first 30 dims, but that might potentially induce a risk that some of words will share the same embedding.

Answer 1

The embeddings are floating point vectors, it's very unlikely that they are exactly the same. And even if they were, it will only minorly impact the performance of your model, which is not what you are after anyway, because you are debugging. That said, if you want to use it for model selection you have a few alternatives. The easiest one is to apply PCA one time to your whole embedding space and take the first n dimensions. This way you reduce your dimensionality without throwing away too much information. A better but more difficult approach would be to train your own Glove vectors, either on a public corpus or a private one, and set a low dimensionality during the training procedure. It might be difficult to use your findings on these embeddings for model selection because more expressiveness could lead to much better performance with deeper models or could lead to more overfitting.

Answer 2

GloVe has a wide selection of pre-trained vectorizers. You can simply choose one of the smaller datasets, e.g. the Wikipedia trained model from https://github.com/stanfordnlp/GloVe/blob/master/README.md. For me, loading the 50 dimensional file with 400K vocab size happens in only a second or two.