2
$\begingroup$

I'm working on a document classification project and I'm using a neural net in tensorflow, where the features are 300-dimensional word embeddings, either from fastext or word2vec (yes I know that there are newer better embeddings -- this is for backwards compatibility).

Each of the documents is very long -- 3000+ tokens.

Featurization involves tokenizing the document, embedding it, and stacking the embeddings into an array of dimension batchsize, max_doc_length, features. It takes more than 5x as long to make a batchfile (on a CPU) as it does to do one iteration of backprop (also on a CPU). My problem would be worse on a GPU, if backprop went faster.

My batches are almost 1GB in size, so I'd use almost a whole TB if I were to write everything to disk before starting backprop. That's too much, especially given that I want to try different featurization strategies.

So, what are some good solutions to this problem? Are there software packages that are designed to solve this problem?

Right now I've got two processes going in parallel -- one that makes batches and saves them to the disk (in parallel) and the other which loads them, uses them for backprop, then deletes them. It feels sort of janky, and I can't be the first person who has faced this issue.

Suggestions appreciated!

$\endgroup$
0
$\begingroup$

There are many options to spend this up:

  • Get a better CPU.
  • Distribute the process across a cluster since each document is independent.
  • Reduce the size of the vocabulary. If only the top-n most popular words are used, it greats reduces the size of the data.
  • Reduce the size of the embedding space.
  • Switch to doc2vec so the document themselves are a learned embedding.
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.