5
$\begingroup$

I am currently selecting features of products by using LDA to group 6000 keywords of product into topics. Here is the sample of my dataset after being organized into list of keywords for each product id. enter image description here I consider each id as a "document" and each keywords as the "word" in a "document" for the case of LDA. It didn't work out as I expected as each topic have many identical keywords with different weight. I removed 100 most common keywords but there are still some identical keywords in the topics. Here is the sample output: enter image description here

How can I deal with the identical keywords in my topics? and also how to deal with the 100 most common keywords I removed?

Here is the graph showing the frequency of the words in all "documents" enter image description here Each word only presents once in each document, but may present in different documents. I updated a new graph of the frequency of the words

Thank you so much. Any suggestion is appreciated.

Improve this question
$\endgroup$

3 Answers 3

Reset to default
4
$\begingroup$

Try before doing LDA look at the data - like doing TF, IDF and TFIDF analysis to identify such words which happen in all subject. If You have some taxonomy of Your product definition - consider using it. In my case it was really helpful.

I experimented with LDA topic modeling for recommendation systems purposes. I've few runs for offers in our marketplace service. It is in some aspect similar to Your problem with products and a list of keywords. But in our case, we do not have a product definition and an offer description is created by a seller. So You can imagine what tokens soup You will get :-)

The key point for me wast to analyze TF, IDF and TFIDF inside categories where I've tested LDA. E.g. for Toys category (http://allegro.pl/zabawki-11818?ref=simplified-category-tree) ~ .5M items, LDA based only on its names results in:

TOPIC:
lego    0.02282500803373657
puzzle  0.016679246682968853
gra     0.01092812347216676
klocki  0.010650330607355207
trefl   0.006797740719581059
zabawka 0.006365688767908592
lalka   0.0063546887775893105
maskotka        0.006111593588558912
dzieci  0.005888846995546501
24h     0.005736876716620104

TOPIC:
lego    0.02296167186908205
puzzle  0.016613064812376045
gra     0.010928234683331952
klocki  0.010607432835126579
trefl   0.006689061156804526
lalka   0.006317220992078405
zabawka 0.0062838718987015575
maskotka        0.006121026383268814
dzieci  0.0058569443925760144
24h     0.00578544965708585

TOPIC:
lego    0.02285507528449409
puzzle  0.016539723998111246
gra     0.010892673154789407
klocki  0.010624244094063881
trefl   0.006683424205358961
lalka   0.006350949499009139
zabawka 0.006324510823409019
maskotka        0.006148070280704085
dzieci  0.0058401580163702565
24h     0.005782372531889951
...

And the rest looks almost the same. Before running LDA I just did basic text preprocessing - our dictionary based item-name stop words removal and text tokenization (but without steaming). Even giving more terms for topic descriptions results doesn't look promising. Then, after exporting word frequencies with its offer categories by looking at data and plots I've decided to remove terms with TF-IDF above some threshold. Yes, above - I've used the calculation provided by Spark 1.3.1 implementation (HasingTF + IDF from mllib, no ml). After doing this I received:

TOPIC:
interaktywny    0.0026985965530064884
pony    0.0026721637823727343
dmuchany        0.0015777211309733249
baterie 0.0012447186564456534
pojazdy 0.0011171017074143481
thomas  9.476877418459519E-4
dinozaury       8.823212274401862E-4
monsters        7.426822230409613E-4
heroes  7.365256561824247E-4
ninjago 7.344344320326593E-4

TOPIC:
pony    0.0026867880330479327
interaktywny    0.0026803244251279693
dmuchany        0.0015797940843537916
baterie 0.001263424692131187
pojazdy 0.0010685733429671523
thomas  9.87888159025782E-4
heroes  8.26066846602493E-4
ptaszek 7.952454816383102E-4
dinozaury       7.849838129629457E-4
batman  7.408185413186286E-4

Where results started to differ from each other. Taking like last time only 10-terms for every subject description. Still, it's not perfect but better.

So, for me doing the experiments which few categories, the way to overcome problem was to remove it based on TF-IDF value. But, for every category the threshold was calculated separately. Mainly: mean(tf-idf) + 3*sd(tf-idf).

I know that Idf multiplication factor in Tf-Idf should resolve this problem by itself and the term should be punished for occurring in every document. But, using Spark implementation (idf = log((m + 1) / (d(t) + 1))) and our data it was simpler during a quick and dirty experiment to filter it that way.

When I find few sec. I will get back on this and place results with code and online to share this.

Improve this answer
$\endgroup$
3
  • $\begingroup$ Sorry for this. My mistake. I expected it to be a comment. But after all, I decided to describe my test with LDA. $\endgroup$
    – Bartłomiej Twardowski
    Jun 5, 2015 at 12:49
  • $\begingroup$ Thank you for your detailed reply. I am going to try TF-IDF then! Look forward to your update. $\endgroup$
    – sylvia
    Jun 5, 2015 at 23:46
  • $\begingroup$ I tried TF-IDF and it did work better. Thanks for sharing again. but do u know how to deal with the terms that you removed based on TF-IDF? Are those not the important information for your recommendation system? $\endgroup$
    – sylvia
    Jun 9, 2015 at 20:46
3
$\begingroup$

I think, even before doing LDA, you should remove words which appear in more than "x" percent of your documents. Try different "x" starting from 80% and then going down. The logic is that if the word is common for many documents, it does not distinguished those and should be neglected.

Improve this answer
$\endgroup$
2
  • $\begingroup$ You are welcome, Please provide more feedback. Did this way improve the result in your case? What did you get? $\endgroup$
    – lanenok
    Jun 2, 2015 at 4:15
  • $\begingroup$ There are still the same words shown in different topics. Each word only presents once in each document, but may present in different documents. I updated a new graph of the frequency of the words. The few words with the highest frequency each take only 14% of the documents. I also wonder how to deal with words I removed. Could they be considered as features or too common to exist? $\endgroup$
    – sylvia
    Jun 3, 2015 at 18:45
0
$\begingroup$

You may encounter this problem of identical keywords when the index of the documents in the corpus is wrong, i.e. the list of occurrences vector has bad document ids (for instance the same document id for all documents):

<id_doc_0>, <vector of occurrences for doc 0>
<id_doc_1>, <vector of occurrences for doc 1>
...
<id_doc_1>, <vector of occurrences for doc 1>
Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.