2
$\begingroup$

I was implementing a decision tree on a dataset. Before that, I wanted to transform a particular column with CountVectorizer. For this, I am using pipeline to make it simpler.

But there is an error of incompatible row dimensions.

code

# Imported the libraries....
from sklearn.feature_extraction.text import CountVectorizer as cv
from sklearn.preprocessing import OneHotEncoder as ohe
from sklearn.compose import ColumnTransformer as ct
from sklearn.pipeline import make_pipeline as mp
from sklearn.tree import DecisionTreeClassifier as dtc

transformer=ct(transformers=[('review_counts',cv(),['verified_reviews']),
                             ('variation_dummies', ohe(),['variation'])
                            ],remainder='passthrough')

pipe= mp(transformer,dtc(random_state=42))

x= data[['rating','variation','verified_reviews']].copy()
y= data.feedback

x_train,x_test,y_train,y_test= tts(x,y,test_size=0.3,random_state=42,stratify=y)
print(x_train.shape,y_train.shape)             # ((2205, 3), (2205,))

pipe.fit(x_train,y_train)                       # Error on this line

Error

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-79-a981c354b190> in <module>()
----> 1 pipe.fit(x_train,y_train)

7 frames
/usr/local/lib/python3.6/dist-packages/scipy/sparse/construct.py in bmat(blocks, format, dtype)
    584                                                     exp=brow_lengths[i],
    585                                                     got=A.shape[0]))
--> 586                     raise ValueError(msg)
    587 
    588                 if bcol_lengths[j] == 0:

ValueError: blocks[0,:] has incompatible row dimensions. Got blocks[0,1].shape[0] == 2205, expected 1.

pic

Questions

  1. How is this error of incompatible row dimension forming?
  2. How it can be solved?
$\endgroup$
12
  • $\begingroup$ Does it give this error only with the pipeline? $\endgroup$ May 16 '20 at 10:33
  • $\begingroup$ @BlackCurrant, i did not try indivudually. i tried directly with the pipeline. $\endgroup$
    – teddcp
    May 16 '20 at 10:55
  • 1
    $\begingroup$ Well, I tried and found so far- there is an issue in trace showing with return self._hstack(list(Xs)), because while its going through hstck its reshaping its value incorrect. we can change the data into numpy array and pass it but then in the pipeline some of the transformation isn't accepting numpy values. you may want to run it individually first and check which one is accepting which and transform the data before passing fit fr the complete pipeline.stackoverflow.com/questions/25795511/… $\endgroup$ May 16 '20 at 12:13
  • $\begingroup$ @BlackCurrant..thanks for the input. Just clarifying..now i have to do the CountVectorizer and OneHotEncoder indivudually to the respective columns and then apply the decision tree..right ? Any way to use the pipeline l? $\endgroup$
    – teddcp
    May 16 '20 at 12:24
  • 1
    $\begingroup$ well.. if that is the case then we see why its doing so.understand what its expecting and why. then may be call it before passing the data to pipeline. $\endgroup$ May 16 '20 at 14:31
3
$\begingroup$

As per the documentation, whenever the transformer expects a 1D array as input, the columns were specified as a string ("xxx"). For the transformers which expects 2D data, we need to specify the column as a list of strings (["xxx"]).

so the code below will work.

## Important: i have passed the columns a string to CV and list of columns to OHE

transformer=ct(transformers=[('review_counts',cv(),'verified_reviews'), 
                             ('variation_dummies', ohe(),['variation'])
                            ],remainder='passthrough')

Credit goes to Another man who helped me on this.

$\endgroup$
4
  • 1
    $\begingroup$ That's good. Its actually here- stackoverflow.com/questions/56298242/… $\endgroup$ May 17 '20 at 6:40
  • $\begingroup$ @BlackCurrant...yes :) $\endgroup$
    – teddcp
    May 17 '20 at 6:41
  • 1
    $\begingroup$ I opened it to post the solution to get me some votes and found that you got it already. its good to know.in future if you would like to debug it in detail here is something-stackoverflow.com/questions/58911481/… $\endgroup$ May 17 '20 at 6:44
  • $\begingroup$ @BlackCurrant..Thanks for guiding me...i really appreciate it. ps - i have recently started reading ML :) $\endgroup$
    – teddcp
    May 17 '20 at 6:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.