1
$\begingroup$

I am receiving the following error. I have check shapes of X and y, and did no find error

from sklearn.model_selection import train_test_split
from sklearn.utils import check_consistent_length

labels = ['non-role','role']
X = df[["POS", "NER", "DEF", "SYN"]]
y = df["Label"]
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=0, test_size=0.2, shuffle=True)
print(X_train.shape)
print(y_train.shape)
print(X_test.shape)
print(y_test.shape)

print(check_consistent_length(X_train, y_train))

And here is the output:

(25238, 4)

(25238,)

(6310, 4)

(6310,)

None

I was trying to fit in the model:

NB_pipeline = Pipeline([('tfidf-vect', TfidfVectorizer()),('clf', RandomForestClassifier())])
NB_pipeline.fit(X_train, y_train)

But received following error:

ValueError: Found input variables with inconsistent numbers of samples: [4, 25238]
$\endgroup$
1

1 Answer 1

0
$\begingroup$

After a lot of searching I just found the answer to this, as I was running into this myself.

You're inputting a 2D dataframe to your vectorizer, but it's only expecting one sequence of data. Unlike a lot of estimators and transforers (e.g. SVM, XGBoost) a vectorizer here really only wants an individual series of strings to vectorize. It's trying here to vectorize a collection of features, and getting confused that there are only 4 things to vectorize (which is each series in the dataframe), rather than the sample-size amount, which would be the case if it was fed each column individually. That's how the information is getting flipped horizontally. TLDR you want to feed it only a single column you want it to vectorize, rather than multiple at once.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.