3
$\begingroup$

I see people are talking mostly about bag-of-words, td-idf and word embeddings. But these are at word levels. BoW and tf-idf fail to represent word orders, and word embeddings are not meant to represent any order at all. What's the best practice/most popular way of representing word order for texts of varying lengths? Simply concatenating word embeddings of individual words into long vector appearly not working for texts of varying lengths...

Or there exists no method of doing that except relying on network architectures like the positional encoding in transformer family?

By the way, ngram is not a solution to me, as it still fails to solve the problem in representing texts in varying lengths. (Or can it and how? It seems to me ngram is more for next word prediction rather than representing texts with varying lengths.)

TIA :)

$\endgroup$
1
  • $\begingroup$ Besides transformers - LSTMs, try searching this site. On this site you can also find suggestions of averaging word embeddings (which I don't like). $\endgroup$
    – Valentas
    Apr 3, 2022 at 11:31

2 Answers 2

0
$\begingroup$

I recommend working with parts of speach (POS), more specifically with the RDF-Triple of Subject, Predicate and Object.

It both acts as the major structure of the sentence and preserves the order (i.e. the Subject Predicates upon the Object).

See if you can go with that alone. If not, you can add to it from the techniques you mentioned (bagging, tf-idf, etc..).

See my answer here for a suggested combined tf-idf score upon an rdf-triple, to check whether the triple itself is "unique-enough".

$\endgroup$
0
$\begingroup$

You can manually add each word index as additional vector to the word embedding, and this applies to a set of word embedding techniques, like Glove. For example, while forming the words representation from a sentence, you get each word embedding from Glove dictionary, and let us assume that it returns 50 vectors. Now add additional vector that represents the word index, so the new word embedding consists of 51 vectors, I tested this technique and was helpful in enhancing my model's performance.

Check this Python code:

words_embeddings = []
for i in range(len(sentence)):
  single_word_emd = glove_dict[ sentence[i] ]
  single_word_emd.append(i)
  words_embeddings.append(single_word_emd)
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.