Let’s explain the whole goal firstly, then go through the question.
I am using topic modeling like LAtent Dirichlet Allocation and NMF to extract the topic from a collection of documents.
My dataset is PubMed, I used about three categories of this collection and went through the abstract part(in each category there is 10 abstract file so totally I have 30 abstract)
After applying LDA on my data, for the evaluation process, to see what is the accuracy of the topics generated for each document, I evaluated that with OneVsRestClassifier in sklearn.
As OneVsRestClassifier is a classification method, it needs the data to have label, so I am going to explain how I generated the label for the output of LDA model.
In LDA, I used the document-topic matrix and this matrix is like for each document there are some topics. for the topics, there is probability that shows with which probability this topic belongs to this document.
So the label for each row is the highest probability of the topics(indixes that shows the topic e.g. the probability in the first column shows the probability for the topic0).
Hope the explanations are clear, if not, please let me know I will elaborate more with examples.
this is the code that doing the above approach:
import os from nltk.tokenize import RegexpTokenizer from nltk.stem.porter import PorterStemmer from sklearn.cross_validation import train_test_split from sklearn.decomposition import LatentDirichletAllocation from sklearn.feature_extraction.text import CountVectorizer from sklearn.multiclass import OneVsRestClassifier from sklearn.svm import LinearSVC import numpy as np from sklearn import metrics tokenizer = RegexpTokenizer(r'\w+') # Create p_stemmer of class PorterStemmer lines= p_stemmer = PorterStemmer() lisOfFiles=[x for x in os.walk("data")] fullPath = [x for x in os.walk("data")] for j in lisOfFiles: with open(os.path.join(fullPath,j)) as f: a=f.read() lines.append(a) for j in lisOfFiles: with open(os.path.join(fullPath,j)) as f: a=f.read() lines.append(a) for j in lisOfFiles: with open(os.path.join(fullPath,j)) as f: a=f.read() lines.append(a) # compile sample documents into a list doc_set = lines tf_vectorizer = CountVectorizer(max_features=1000, stop_words='english') tf = tf_vectorizer.fit_transform(doc_set) # Trains the LDA models. lda = LatentDirichletAllocation(n_topics=10, max_iter=5, learning_method='online', learning_offset=50., random_state=0) lda_x=lda.fit_transform(tf) #creating the labels new_y = np.argmax(lda_x, axis=1) Xtrain, Xvalidate, ytrain, yvalidate = train_test_split(lda_x, new_y, test_size=.3) predictclass=OneVsRestClassifier(LinearSVC(random_state=0)).fit(Xtrain, ytrain).predict(Xvalidate) yacc=metrics.accuracy_score(yvalidate,predictclass) print (yacc)
Now my question:
when I run this code, it seems like most of the time gives me 100 accuracy :|
and when I run it for several times the accuracy is changing, and never drop below 66 but it is changing between 66 and 100, and most of the time is 100.
I understand these changing is because of the random sampling of LDA. firstly I believe something is wrong with my code that I am getting that much good accuracy.
Secondly, if the accuracy is changing like this, which one is reliable, is there any approach to fix it out in one output?
The same process also is happening for my NMF model.
my only doubt was the data I was working on. as I explained in the first paragraph, it includes 30 abstract file which I have transferred to a list, so I have a list seperated by comma, but if the abstract file has already had the comma, how it can realize the correct document?!!!
but I tested it by iterating through the list and getting the size of the list and it was 30**
when I changed the data from 300 abstract to 3000 abstract, Accuracy is still changing BUT just from 93 to 97, I mean the difference has decressed