Have a piece of code where i am cleaning the text from the 'Description' column and storing it as "cleaned"

Then i create a ML model using the above as one of my features.

       X = data[['originalname','cleaned']]
       Y = data['Total score']

       X_train, X_test, y_train, y_test = train_test_split(X,Y, 

       pipeline = Pipeline([('vect', TfidfVectorizer(ngram_range=(1, 2), 
       stop_words="english", sublinear_tf=True)),
       ('chi',  SelectKBest(chi2, k='all')),
       ('clf', LinearSVC(C=1.0, penalty='l1', max_iter=300, dual=False))])

       X_train.shape--->(44, 2)

Trying to train the model gives me the above error

       model = pipeline.fit(X_train, y_train)

How do i use both 'original description' and 'cleaned' as my feature to predict 'Total score' without the above error ?

  • $\begingroup$ A first guess: TfidfVectorizer doesn't know how to deal with a 2D array? $\endgroup$ – Ben Reiniger Mar 5 '20 at 13:01
  • $\begingroup$ Is there a way i can use both the columns in my model as independent variables and create a score(target) $\endgroup$ – Gaurav Hegde Mar 5 '20 at 16:52
  • $\begingroup$ (If that is the problem -- I haven't gotten around to checking, then) you can use ColumnTransformer to apply two different vectorizers to the two columns. I'm not sure if there's an easy way to treat the two columns with the same vectorizer? $\endgroup$ – Ben Reiniger Mar 5 '20 at 21:37