I have a corpus of data

Class   Tweet
-1  toxic phenol ingredient in vaccines  the detrimental effects of injected phenol  have not been fully evaluated   
-1  doctors give flu shots to pregnant women despite evidence of harm to fetus  
-1   hearus autism and the mmr vaccine  the most diabolical medical scandal of the century       cdcwhistleblower  vaxxed
-1  rt   how to naturally detox from mandatory vaccine injections     hearus  cdcwhistleblower
-1  rt   the removal of vaccine exemptions forces parents who know that vaccines injure and sicken their children into coerced ha   

I'll be doing some supervised learning on the text. I put the text through a pipeline as follows:

selector = SelectKBest(chi2, k = K)

clf = SVC(kernel = 'linear')

p = Pipeline([('vect', CountVectorizer()),
                    ('tfidf',TfidfTransformer() ),

SelectKBest will return K of the best features. I suppose then that those will be used to train the model.

Do I need to predict new tweets on the same features? Or can I just leave the pipeline as is and use CV to find the best K?


Your Pipeline is now a model that expects similar input as before. You can use a CV scheme to choose your K inside your Pipeline, after it has evaluated the options and chosen the correct K it will use that to predict. For actual predictions, enter a list of tweets and it will classify them, no need to manually specify which features to use. If you want to know which features your Pipeline has chosen you can look in the properties of your 'p' object.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.