I have a corpus of data
Class Tweet -1 toxic phenol ingredient in vaccines the detrimental effects of injected phenol have not been fully evaluated -1 doctors give flu shots to pregnant women despite evidence of harm to fetus -1 hearus autism and the mmr vaccine the most diabolical medical scandal of the century cdcwhistleblower vaxxed -1 rt how to naturally detox from mandatory vaccine injections hearus cdcwhistleblower -1 rt the removal of vaccine exemptions forces parents who know that vaccines injure and sicken their children into coerced ha
I'll be doing some supervised learning on the text. I put the text through a pipeline as follows:
selector = SelectKBest(chi2, k = K) clf = SVC(kernel = 'linear') p = Pipeline([('vect', CountVectorizer()), ('tfidf',TfidfTransformer() ), ('feat',selector), ('clf',clf)])
SelectKBest will return
K of the best features. I suppose then that those will be used to train the model.
Do I need to predict new tweets on the same features? Or can I just leave the pipeline as is and use CV to find the best K?