I'm classifying data into 2 classes, by Logistic Regression from python scikit-learn.

I'm trying different types of feature vectors:

  • Binary
  • Frequency
  • TF-IDF

In Binary and TF-IDF feature vectors, I get great results. However, in Frequency feature vectors, I get poor results unless I multiply the values by some factor (for example 10, or 100). The multiplication improves the results significantly (the higher the factor, the better).

Is this a normal behavior or is it more likely that I'm doing something wrong? I suspected that the values are too small and my array rounded some of them down to zero because its type was merely float (unspecified bits number). But I tried changing it to np.float64, and the results didn't change.

What other factors can be causing this?

Edit: Learning Curves:

Binary | TF-IDF | Frequency X 10 (from left to right):


Normal frequency (3 attempts):

Edit 2: (data and code)

Frequency Data: X axis: a dictionary of 1K words , Y axis: 1K samples - 500 each class

Labels (1K binary labels):
note that it's a transpose vector. so X here is the samples

Code for frequency:

vec[i] = np.divide(sample_vals, len(sample.split()))

Code for frequency X 10:

vec[i] = np.multiply(np.divide(sample_vals, len(sample.split())), 10)

Classification code:

#in `X` I have the data, and in `y` I have the labels
scores = [clf.fit(X[train], y[train]).score(X[test], y[test])
          for train, test in kfold.split(X)]
  • $\begingroup$ Can you show all your learning curves? I suspect it is just a question of optimization (step size, iterations). Welcome to the site! $\endgroup$
    – Emre
    May 19 '17 at 20:25
  • $\begingroup$ @Emre Thank you! I added learning curves. Please take a look $\endgroup$
    – Alaa M.
    May 20 '17 at 16:59
  • $\begingroup$ Your small frequency plots have not converged. What happens after a few thousand iterations? Are you persisting the loss only once every 200 iterations; why is it piecewise linear? $\endgroup$
    – Emre
    May 20 '17 at 18:09
  • $\begingroup$ Do you mean if I add more training examples? If so then I get this plot $\endgroup$
    – Alaa M.
    May 20 '17 at 18:13
  • $\begingroup$ I'm not sure what you mean by persisting the loss. But I think you mean why do I have results only every 200 examples on the plot. I'm using this function to draw. Maybe I need to play with the parameters... $\endgroup$
    – Alaa M.
    May 20 '17 at 18:18

The problem is solved if I let TfidfVectorizer do the job:

tfidf_vec = TfidfVectorizer(vocabulary=dictionaryList, use_idf=False)
final_vec = tfidf_vec.fit_transform(df[DF_X_DATA])

use_idf=False is the solution. It lets it calculate frequency automatically. I don't have to manually divide by the sample's length this way.

The results now are ~90% and the learning curve looks normal.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.