sklearn pipeline ValueError: Found input variables with inconsistent numbers of samples

Question

I am receiving the following error. I have check shapes of X and y, and did no find error

from sklearn.model_selection import train_test_split
from sklearn.utils import check_consistent_length

labels = ['non-role','role']
X = df[["POS", "NER", "DEF", "SYN"]]
y = df["Label"]
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=0, test_size=0.2, shuffle=True)
print(X_train.shape)
print(y_train.shape)
print(X_test.shape)
print(y_test.shape)

print(check_consistent_length(X_train, y_train))

And here is the output:

(25238, 4)

(25238,)

(6310, 4)

(6310,)

None

I was trying to fit in the model:

NB_pipeline = Pipeline([('tfidf-vect', TfidfVectorizer()),('clf', RandomForestClassifier())])
NB_pipeline.fit(X_train, y_train)

But received following error:

ValueError: Found input variables with inconsistent numbers of samples: [4, 25238]

Answer 1

After a lot of searching I just found the answer to this, as I was running into this myself.

You're inputting a 2D dataframe to your vectorizer, but it's only expecting one sequence of data. Unlike a lot of estimators and transforers (e.g. SVM, XGBoost) a vectorizer here really only wants an individual series of strings to vectorize. It's trying here to vectorize a collection of features, and getting confused that there are only 4 things to vectorize (which is each series in the dataframe), rather than the sample-size amount, which would be the case if it was fed each column individually. That's how the information is getting flipped horizontally. TLDR you want to feed it only a single column you want it to vectorize, rather than multiple at once.