0
$\begingroup$

So I have a following scenario:

Pipeline, that transform text/dict/numerical data and classifies the result with linear regresion. It looks something like this:

args = {
    'y': labels,
    'cv': 5,
    'scoring': 'accuracy',
    'n_jobs': -1
}

feats = ('features', FeatureUnion([
    fulltext_pipeline,
    numerical_pipeline,
    nela_desc_pipeline,
    v_tags_pipeline,
    os_pipe
]))

clf = ('clf', LogisticRegression(multi_class='auto',
    solver='liblinear',
    tol=0.02,
    C=500,
    penalty='l1',
    random_state=0))

os_pred = Pipeline([
    feats,
    clf
])

And for evaluation I am using cross_val_score. So far so good, everything works as suppoused, but if I try to extract "features" and run them separately, I get worse results. The example looks as follows:

features = feats[1].transform(data)
np.average(cross_val_score(clf[1], features, **args))
np.average(cross_val_score(os_pred, data, **args))

I expected to get an exact same results here (as in theory they should do the same - maybe I am missing something?)

I've tried with .transform(data), .fit_transform(data), .fit_transform(data, y=labels). The odd thing for me was that all three methods generate the same output (sparse matrix) which is far worse compared to results from pipeline.

Data is in form of sparse matrix

<421x862 sparse matrix of type '<class 'numpy.float64'>'
    with 361371 stored elements in Compressed Sparse Row format>

While labels are normal strings (3 classes).

Results are as follows:

0.672436 from np.average(cross_val_score(clf[1], features, **args))

0.772509 from np.average(cross_val_score(os_pred, data, **args))

I double checked for any random seed or something of this sort, but the results are consistend and pipeline/classifier return always the same result. And for me it looks like that this is quite a big difference (given the fact that the range is [0.00; 1.00]).

Question at the end is: Why classifier seems to be biased in a pipeline scenario compared to just prefeeling it with features directly?

I do aknowledge that pipeline does some kind of additional steps of transforming/fit-transforming/etc., but with my current knowledge of sklearn library I cannot reproduce them "outside" the pipeline.

Improve this question
$\endgroup$
4
  • $\begingroup$ Can you tell us exactly the difference? $\endgroup$
    – Victor Oliveira
    Mar 14 '19 at 1:07
  • $\begingroup$ I've updated question @VictorOliveira. $\endgroup$
    – Yoan Dinkov
    Mar 14 '19 at 9:57
  • $\begingroup$ Hum, that is really a huge difference. One thing I have noticed is that you have a small data set. Have you tried to run more than once? To see if you have variance between your average estimates? Or is it always these scores? $\endgroup$
    – Victor Oliveira
    Mar 14 '19 at 12:46
  • $\begingroup$ Scores are always the same (I would say that cross_val_score splits the same data in the same place, so estimators fit on exactly same data each time). Thing that I do notice is that in pipeline transformation step will be performed for each fold, as it splits the raw "data" and tranform to features, while I am missing this step in "pretransformed" features. $\endgroup$
    – Yoan Dinkov
    Mar 14 '19 at 17:12
1
$\begingroup$

Normalization

Long story short normalization was the problem in my case. Because I haven't shared the transformer pipelines code in FeatureUnion, I've missed to tell that each one of them ends with normalization. So that's what the biggest difference.

  • In case with pretrained -> normalization is done over the whole dataset and nothing more

  • In case of pipeline -> normalization is done 10 times (twice per each fold -> once for train data, once for test data)

And that's how I do get different values => such difference in results.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.