First of all, I want to say that I am asking this question because I am interested in using BERT embeddings as document features to do clustering. I am using Transformers from the Hugging Face library. I was thinking of averaging all of the Word Piece embeddings for each document so that each document has a unique vector. I would then use those vectors for clustering. Please feel free to comment if you think this is not a good idea, or if I am missing something or not understanding something.
The issue that I see with this is that you are only using the first N tokens which is specified by max_length
in Hugging Face library. What if the first N tokens are not the best representation for that document? Wouldn't it be better to randomly choose N tokens, or better yet randomly choose N tokens 10 times?
Furthermore, I realize that using the WordPiece tokenizer is a replacement for lemmatization so the standard NLP pre-processing is supposed to be simpler. However, since we are already only using the first N tokens, and if we are not getting rid of stop words then useless stop words will be in the first N tokens. As far as I have seen, in the examples for Hugging Face, no one really does more preprocessing before the tokenization.
[See example below of the tokenized (from Hugging Face), first 64 tokens of a document]
Therefore, I am asking a few questions here (feel free to answer only one or provide references to papers or resources that I can read):
- Why are the first N tokens chosen, instead of at random? 1a) is there anything out there that randomly chooses N tokens perhaps multiple times?
- Similar to question 1, is there any better way to choose tokens? Perhaps using TF-IDF on the tokens to at least rule out certain useless tokens?
- Do people generally use more preprocessing before using the Word Piece tokenizer?
- To what extent does the choice of
max_length
affect performance? - Why is there a limit of 512 max length in Hugging Face library? Why not just use the length of the longest document?
- Is it a good idea to average the WordPiece embeddings to get a matrix (if you want to do clustering)?
- Is it a good idea to use BERT embeddings to get features for documents that can be clustered in order to find similar groups of documents? Or is there some other way that is better?
original:
'Trump tries to smooth things over with GOP insiders. Hollywood, Florida (CNN) Donald Trump\'s new delegate guru told Republican Party insiders at a posh resort here on Thursday that the billionaire front-runner is recalibrating the part "that he\'s been playing" and is ready
tokenized:
['[CLS]',
'trump',
'tries',
'to',
'smooth',
'things',
'over',
'with',
'go',
'##p',
'insider',
'##s',
'.',
'hollywood',
',',
'florida',
'(',
'cnn',
')',
'donald',
'trump',
"'",
's',
'new',
'delegate',
'guru',
'told',
'republican',
'party',
'insider',
'##s',
'at',
'a',
'po',
'##sh',
'resort',
'here',
'on',
'thursday',
'that',
'the',
'billionaire',
'front',
'-',
'runner',
'is',
'rec',
'##ali',
'##bra',
'##ting',
'the',
'part',
'"',
'that',
'he',
"'",
's',
'been',
'playing',
'"',
'and',
'is',
'ready',
'[SEP]']