0
$\begingroup$

I am trying to train a few Machine Learning (ML) algorithms such as SVM, NB and Random Forest to do binary classification on disaster tweets. During this project, I want to train ML algorithms for a combined disaster dataset. However, I want to apply it to individual disasters separately (e.g, train on multiple disasters, apply to earthquake; train on multiple disasters, apply to flood). I have the combined dataset in one pandas DataFrame and individual disasters in separate DataFrames. In this case, how should I split my train and test datasets? At the moment, I split my combined dataset and apply ML algorithms.

from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(all_disaster['cleaned'], all_disaster['target'], test_size=0.33, random_state=42)

But, I feel that is not the proper way, as I want to apply it into individual disasters.

$\endgroup$
0
$\begingroup$
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(all_disaster['cleaned'],specific_disaster['target'], test_size=0.33, random_state=42)

Like this, you train your model on all the disasters data, and you test it on a specific disaster.

$\endgroup$
  • $\begingroup$ Many thanks. But then how sklearn can split as train and test. Train and test data sets are of different sizes. $\endgroup$ – Nilani Algiriyage Oct 28 '20 at 20:30
  • $\begingroup$ you need to have the same number of features, otherwise, it won't be logical to do so. $\endgroup$ – MXK Oct 29 '20 at 9:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.