10
$\begingroup$

I came across two interesting papers that describe promising approaches for document classification using word embedding.

1. The fasttext algorithm

Described in the paper Bag of Tricks for Efficient Text Classification here.

(With further explanation here).

2. DANs

Described in the paper Deep Unordered Composition Rivals Syntactic Methods for Text Classification here.

Question:

What is the difference between both approaches?

Are they essentially the same as they both seem to average word embedding and pass it through an MLP or am I missing something crucial?

Improve this question
$\endgroup$
3
  • 1
    $\begingroup$ Until somebody takes the time to read both, here's my opinion: they might be; people independently invent similar algorithms often and it takes time for the dust to settle. I note that the first article is newer and not peer reviewed. If your interest is practical, I'd compare benchmarks and test them myself: fasttext, dan. $\endgroup$
    – Emre
    Commented Apr 2, 2017 at 4:33
  • $\begingroup$ The first one (fasttext) is a standard method in the field, has 750 citations and the peer reviewed version can be found at aclweb.org/anthology/papers/E/E17/E17-2068. The second one sounds more fringe and is also quite old, anno 2015. And cs224n has a 2019 version, regarding the link in the original post. $\endgroup$
    – BookYourLuck
    Commented Apr 20, 2019 at 16:14
  • 1
    $\begingroup$ And the second one seems to be indeed merely a bag of words model with a fancy title. $\endgroup$
    – BookYourLuck
    Commented Apr 20, 2019 at 16:31

1 Answer 1

Reset to default
1
$\begingroup$

The first most important difference consists in the fact that when using fasttext you are training a language model, i.e. your own embedding vecotrs, while DAN is an architecture (not a language model) that require either a random initializion of embedding layers (which are then trained along with the other layers) or to use pre-trained embeddgings like GloVe (or even fasttext vectors!).

DAN is something that has become popular in some sense (even though I never saw this paper before now). Aveaging embedding vectors of single words before feeding them to a dense layer is a common practice if you need to perform some task at a paragraph or document level.

Just to add some peculiarities of fasttext embeddings, their are trained not for single words, but for n-grams. So during the preprocesing of the corpus from which the embedding are learnt, words are splitted into several chunks of character. For example:

'matter' would become [ma, mat, att, tte, ter, er]

and a unique embedding is then learnt for each chunk like 'ma' or 'mat'. The training follow the same logic of word2vec vectors, which means that from every chunk the model tries to predict context chunks. The advantage of learning embeddings for each chunk relies on the ability of these vectors to learn specific morphological features that classic token-level embeddings usually miss.

If it might help, for a good survey on word embeddings I suggest to take a look at this.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.