3
$\begingroup$

I have data with multiple labels, for example

enter image description here

My X set is fromt second to third column, and I want to classify either first column or the last column, so I made my Y the last column.

The goal is so that if I would classify Vios it would return me Car or 0 in other words it can find its way to the first row.

Classification use case:

classify("poodle") #just pretend this is a working function

returns: Pets

How I did it in attempt to train my model:

from sklearn.feature_extraction.text import TfidfVectorizer
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state = 72)
count_vect = CountVectorizer()
X_train_counts = count_vect.fit_transform(X_train)
tfidf_transformer = TfidfTransformer()
X_train_tfidf = tfidf_transformer.fit_transform(X_train_counts)
clf3 = RandomForestClassifier().fit(X_train_tfidf, y_train)

I'm using a guide from somewhere on the net that works a bit the same with it, but at the end im getting returned:

ValueError: Found input variables with inconsistent numbers of samples: [5, 4156]

I knew immediately i was doing it wrong. How do I train a model so that it achieves my goal? Any relevant guides or techniques I should be following intead of this? I dont even know the correct way to use vectors on this case.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
0
$\begingroup$

Couple of things: Cant replicate the problem exactly but if you follow these steps you are not exposing yourself

TfidfVectorizer is CountVectorizer + TfidfTransformer, you are exposing yourself to unnecessary complexity and potential errors

Use pipelines, cant stress this enough, its a compact way to pack all the sklearn transformers together AND THEN use fit, predict methods...

I would advise you to follow something like that, or find similiar problem here

Improve this answer
$\endgroup$
6
  • 1
    $\begingroup$ thabks, but do you think my method of vectorizing it in order to classify it would work in this kind of problem in the first place? $\endgroup$
    – user87205
    Commented Dec 20, 2019 at 14:16
  • $\begingroup$ I would try Countvectorizer and Tf-idf sperately and they should give good results $\endgroup$
    – Noah Weber
    Commented Dec 20, 2019 at 14:19
  • $\begingroup$ Im a bit confused when only using vectorizer, heres my data: prnt.sc/qdm2iz Vectorized it: prnt.sc/qdm2rs, when correlating im getting this error: prnt.sc/qdm2y7 as such I cant fit it to any of my models: prnt.sc/qdm38s $\endgroup$
    – user87205
    Commented Dec 20, 2019 at 14:59
  • $\begingroup$ I have already categorized it by id: prnt.sc/qdm3v6, so whats left is to train a model with the 'combined' column, that is vectorized, then i thought id be able to predict to which category it belongs to. $\endgroup$
    – user87205
    Commented Dec 20, 2019 at 15:00
  • $\begingroup$ This is why I thought having a countvectorizer would do the trick, I think this is a factor of how I arrange my data? Im using bagging method to classify, if a vector belong from 1 to 19 categories $\endgroup$
    – user87205
    Commented Dec 20, 2019 at 15:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.