2
$\begingroup$

Let’s explain the whole goal firstly, then go through the question.

I am using topic modeling like LAtent Dirichlet Allocation and NMF to extract the topic from a collection of documents.

My dataset is PubMed, I used about three categories of this collection and went through the abstract part(in each category there is 10 abstract file so totally I have 30 abstract)

After applying LDA on my data, for the evaluation process, to see what is the accuracy of the topics generated for each document, I evaluated that with OneVsRestClassifier in sklearn.

As OneVsRestClassifier is a classification method, it needs the data to have label, so I am going to explain how I generated the label for the output of LDA model.

In LDA, I used the document-topic matrix and this matrix is like for each document there are some topics. for the topics, there is probability that shows with which probability this topic belongs to this document.

So the label for each row is the highest probability of the topics(indixes that shows the topic e.g. the probability in the first column shows the probability for the topic0).

Hope the explanations are clear, if not, please let me know I will elaborate more with examples.

this is the code that doing the above approach:

import os

from nltk.tokenize import RegexpTokenizer
from nltk.stem.porter import PorterStemmer
from sklearn.cross_validation import train_test_split
from sklearn.decomposition import LatentDirichletAllocation
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.multiclass import OneVsRestClassifier
from sklearn.svm import LinearSVC
import numpy as np
from sklearn import metrics

tokenizer = RegexpTokenizer(r'\w+')

# Create p_stemmer of class PorterStemmer
lines=[]
p_stemmer = PorterStemmer()
lisOfFiles=[x[2] for x in os.walk("data")]

fullPath = [x[0] for x in os.walk("data")]

for j in lisOfFiles[2]:
    with open(os.path.join(fullPath[2],j)) as f:
                    a=f.read()
                    lines.append(a)


for j in lisOfFiles[3]:
    with open(os.path.join(fullPath[3],j)) as f:
                    a=f.read()
                    lines.append(a)

for j in lisOfFiles[4]:
    with open(os.path.join(fullPath[4],j)) as f:
                    a=f.read()
                    lines.append(a)

# compile sample documents into a list
doc_set = lines

tf_vectorizer = CountVectorizer(max_features=1000,
                                stop_words='english')

tf = tf_vectorizer.fit_transform(doc_set)

# Trains the LDA models.
lda = LatentDirichletAllocation(n_topics=10, max_iter=5,
                                learning_method='online',
                                learning_offset=50.,
                                random_state=0)

lda_x=lda.fit_transform(tf)

#creating the labels

new_y = np.argmax(lda_x, axis=1)

Xtrain, Xvalidate, ytrain, yvalidate = train_test_split(lda_x, new_y, test_size=.3)

predictclass=OneVsRestClassifier(LinearSVC(random_state=0)).fit(Xtrain, ytrain).predict(Xvalidate)
yacc=metrics.accuracy_score(yvalidate,predictclass)
print (yacc)

Now my question:

when I run this code, it seems like most of the time gives me 100 accuracy :|

and when I run it for several times the accuracy is changing, and never drop below 66 but it is changing between 66 and 100, and most of the time is 100.

I understand these changing is because of the random sampling of LDA. firstly I believe something is wrong with my code that I am getting that much good accuracy.

Secondly, if the accuracy is changing like this, which one is reliable, is there any approach to fix it out in one output?

The same process also is happening for my NMF model.

My Doubt

my only doubt was the data I was working on. as I explained in the first paragraph, it includes 30 abstract file which I have transferred to a list, so I have a list seperated by comma, but if the abstract file has already had the comma, how it can realize the correct document?!!!

but I tested it by iterating through the list and getting the size of the list and it was 30**

Update 2

when I changed the data from 300 abstract to 3000 abstract, Accuracy is still changing BUT just from 93 to 97, I mean the difference has decressed

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.