So I have a following scenario:
Pipeline, that transform text/dict/numerical data and classifies the result with linear regresion. It looks something like this:
args = {
'y': labels,
'cv': 5,
'scoring': 'accuracy',
'n_jobs': -1
}
feats = ('features', FeatureUnion([
fulltext_pipeline,
numerical_pipeline,
nela_desc_pipeline,
v_tags_pipeline,
os_pipe
]))
clf = ('clf', LogisticRegression(multi_class='auto',
solver='liblinear',
tol=0.02,
C=500,
penalty='l1',
random_state=0))
os_pred = Pipeline([
feats,
clf
])
And for evaluation I am using cross_val_score
. So far so good, everything works as suppoused, but if I try to extract "features" and run them separately, I get worse results. The example looks as follows:
features = feats[1].transform(data)
np.average(cross_val_score(clf[1], features, **args))
np.average(cross_val_score(os_pred, data, **args))
I expected to get an exact same results here (as in theory they should do the same - maybe I am missing something?)
I've tried with .transform(data)
, .fit_transform(data)
, .fit_transform(data, y=labels)
. The odd thing for me was that all three methods generate the same output (sparse matrix) which is far worse compared to results from pipeline.
Data is in form of sparse matrix
<421x862 sparse matrix of type '<class 'numpy.float64'>'
with 361371 stored elements in Compressed Sparse Row format>
While labels are normal strings (3 classes).
Results are as follows:
0.672436
from np.average(cross_val_score(clf[1], features, **args))
0.772509
from np.average(cross_val_score(os_pred, data, **args))
I double checked for any random seed or something of this sort, but the results are consistend and pipeline/classifier return always the same result. And for me it looks like that this is quite a big difference (given the fact that the range is [0.00; 1.00]).
Question at the end is: Why classifier seems to be biased in a pipeline scenario compared to just prefeeling it with features directly?
I do aknowledge that pipeline does some kind of additional steps of transforming/fit-transforming/etc., but with my current knowledge of sklearn
library I cannot reproduce them "outside" the pipeline.