# How can I label (predict) an unseen set of data based on an existing model?

I'm working on a learning multi-label classification project, for which I've taken 16K lines of text and kind of manually classified them achieving around 94% of accuracy/recall (out of three models).

Good results I'd say.

I then though I would have been ready to use my model to predict the label for a set of new similar text but not previously seen/predicted. However, it appears that - at least with the sklearns models - I can't simply run the predict against the new dataset as the prediction label array is of a different size.

I am missing something for sure, but at this stage I wonder what considering that I always thought that the classification would have helped in such a task. If I need to know the "answer", I struggle to understand the benefit of the approach.

Below the approach taken in short:

from gensim import corpora

corpus = df_train.Terms.to_list()

# build a dictionary
texts = [
word_tokenizer(document, False)
for document in corpus
]

dictionary = corpora.Dictionary(texts)

from gensim.models.tfidfmodel import TfidfModel

# create the tfidf vector
new_corpus = [dictionary.doc2bow(text) for text in texts]
tfidf_model = TfidfModel(new_corpus, smartirs='Lpc')
corpus_tfidf = tfidf_model[new_corpus]

# convert into a format usable by the sklearn
from gensim.matutils import corpus2csc

X = corpus2csc(corpus_tfidf).transpose()

# Let fit and predict

from sklearn.naive_bayes import ComplementNB
clf = ComplementNB()
clf.fit(X.toarray(), y)

y_pred = clf.predict(X.toarray())

# At this stage I have my model with the 16K text label.

# Running again almost the above code till X = corpus2csc(corpus_tfidf).transpose().
# Supplying a new dataframe should give me a new vector that I can predict via the clf.predict(X.toarray())

corpus = df.Query.to_list()

# build a dictionary
.....
.....

X = corpus2csc(corpus_tfidf).transpose()
y_pred = clf.predict(X.toarray()) # here I get the error


So everything works fine in using the df_train (shape (16496, 2)), by the time I repeat the above with my new dataset df (shape (831, 1), I got the error as above mentioned. Of course, the second dimension in the first dataset, is the one containing the label, which are used with the fit method, so the problem is not there.

The error is due to the fact that a much smaller corpus has generated just 778 columns, whereas the first set of data with 16k row has generated 3226 columns. This is because I vectorised my corpus as I was after using the TF-IDF to give terms some importance. Perhaps this is the error?

I understand that there are models like PCS that can reduce the dimensionality, but I'm not sure about the opposite.

Anybody can kindly explain?

UPDATE

Nicholas helped to figure out where the error is, though a new one is now appearing always in connection of some missing columns.

See below the code and errors as it stands.

from gensim import corpora

corpus = df_train.Terms.to_list()

# build a dictionary
texts = [
word_tokenizer(document, False)
for document in corpus
]

dictionary = corpora.Dictionary(texts)

from gensim.models.tfidfmodel import TfidfModel

# create the tfidf vector
new_corpus = [dictionary.doc2bow(text) for text in texts]
tfidf_model = TfidfModel(new_corpus, smartirs='Lpc')
corpus_tfidf = tfidf_model[new_corpus]

# convert into a format usable by the sklearn
from gensim.matutils import corpus2csc

X = corpus2csc(corpus_tfidf).transpose()

# Let fit and predict

from sklearn.naive_bayes import ComplementNB
clf = ComplementNB()
clf.fit(X.toarray(), y)

y_pred = clf.predict(X.toarray())

# At this stage I have my model with the 16K text label.

corpus = df.Query.to_list()

unseen_tokens = [word_tokenizer(document, False) for document in corpus]
unseen_bow = [dictionary.doc2bow(t) for t in unseen_tokens]
unseen_vectors = tfidf_model[unseen_bow]
X = corpus2csc(unseen_vectors).transpose() # here I get the errors in the first screenshot

y_pred = clf.predict(X.toarray()) # here I get the errors in the second screenshot


UPDATE 2

I've tried also a second approach, using the TfidfVectorizer from sklearn. I did it just in case I was missing something obvious on the previous implementation (you know ... the KISS method).

In that circumstance the output is as expected, I got a prediction. So not sure, but I suspect there is a problem somewhere with the corpus2csc library.

UPDATE 3 Have uploaded the datasets here and here if you want to try. Also a gist is available here.

Cheers

• Please provide the parts of your code where the error actually occurs, and the full error traceback. Possibly you're not using the training dictionary to preprocess the test data? Aug 31 '20 at 19:39
• @BenReiniger thanks for returning. See update above. The stack above is produced when on the second time I recreate a new corpus that I would expect to fit into my X. This new X contains in fact the new terms I want to classify and for which I don't have labels. Aug 31 '20 at 20:02
• Sorry but I do not see why the data that you are trying to predict has fewer features since you are supposed to apply the same vectorization process for both train data and new data, even though they are different texts, the should have the same length. Hope It helps since maybe the error is not in the sklearn API but in the preprocessing step Aug 31 '20 at 21:57
• "Of course, the second dimension in the first dataset, is the one containing the label, which are used with the fit method, so the problem is not there" by this you meant you are passing the y label to the trainig data? Aug 31 '20 at 22:05
• @JulioJesusLuna by the time I trained the model with the 16K rows, yes, I had to give the y label too. This is required by the fit method. During the second time, when I vectorised the new sets of labels (using the code above), the resulting columns in X were less than in the first circumstance. At that point, when calling the predict method, I got the error above. Does it make sense? Sep 1 '20 at 7:57

You need to use the same preprocessing elements (dictionary etc) that you used to create your tfidf matrix during training when you come to apply your model to unseen data.

Do not create a new dictionary, tfidf_model, etc. for the unseen data, or else

• the dimensionality of the data you are passing to your model may not be the same.
• you will lose the information you learned by doing the tfidf on your training data

Straight after the line

corpus = df.Query.to_list()


You want something like

unseen_tokens = [word_tokenizer(document, False) for document in corpus]
unseen_bow = [dictionary.doc2bow(t) for t in unseen_tokens]
unseen_vectors = tfidf_model[unseen_bow]


i.e. not creating a new tfidf model or a new dictionary - using the ones you created and used in training.

• thanks, but I'm a this point not sure on how to proceed. That's what I thought I was doing by processing the second dataframe in the way I did? Since the first dataframe has been converted into a vector, I was expecting to be in need of the same for the second? Any chance you can add a small gist for me to understand? Sep 1 '20 at 10:17
• It’s hard to say exactly what you are doing wrong, as you have only posted a bit of your code, but what i mean is that the dictionary you create in your first few lines of code (you call it “dictionary”) should be the dictionary you use to create the embedding for your unseen examples. However, the comments in your code seem to suggest that you’re going to create a second dictionary (“# Running again almost the above code till X = corpus2csc(corpus_tfidf).transpose() but supplying a new dataframe should give me a new vector...”) Sep 1 '20 at 12:44
• Yes @Nicholas. That's the case. So do you know how can I only transform my list of new words into a suitable vector that I can ran past the fit method? Sep 1 '20 at 14:03
• See my updated answer Sep 1 '20 at 15:22
• Thanks @Nicholas. I'm almost getting there. The error I'm getting now is matmul: Input operand 1 has a mismatch in its core dimension 0, with gufunc signature (n?,k),(k,m?)->(n?,m?) (size 3626 is different from 3619). By the time that I got X converted into a corpus suitable for the model, two new errors were thrown Mean of empty slice. return (1 + np.log2(tf)) / (1 + np.log2(tf.mean(axis=0))) and RuntimeWarning: invalid value encountered in double_scalars ret = ret.dtype.type(ret / rcount). Something that must cause the returning columns not to be again the same. Sep 1 '20 at 16:31

Kudos to @Nicholas to have put myself on the right way.

The specific answer on why this was not working with the Corpora model is due on what I guessed over time. The corpus2csc was kind of compressing/forgetting some details.

The solution is to specify the length of the dictionary when transposing the values.
Therefore, from X = corpus2csc(unseen_vectors).transpose() the code has to become X = corpus2csc(unseen_vectors, num_terms=len(dictionary)).transpose().

Hope this may help somebody one day.

Therefore