1
$\begingroup$

I am trying to train a few Machine Learning (ML) algorithms such as SVM, NB and Random Forest to do binary classification on disaster tweets. During this project, I want to train ML algorithms for a combined disaster dataset. However, I want to apply it to individual disasters separately (e.g, train on multiple disasters, apply to earthquake; train on multiple disasters, apply to flood). I have the combined dataset in one pandas DataFrame and individual disasters in separate DataFrames. In this case, how should I split my train and test datasets? At the moment, I split my combined dataset and apply ML algorithms.

from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(all_disaster['cleaned'], all_disaster['target'], test_size=0.33, random_state=42)

But, I feel that is not the proper way, as I want to apply it into individual disasters.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$ 
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(all_disaster['cleaned'],specific_disaster['target'], test_size=0.33, random_state=42)

Like this, you train your model on all the disasters data, and you test it on a specific disaster.

Improve this answer
$\endgroup$
2
  • 1
    $\begingroup$ Many thanks. But then how sklearn can split as train and test. Train and test data sets are of different sizes. $\endgroup$
    – Nilani Algiriyage
    Oct 28, 2020 at 20:30
  • $\begingroup$ you need to have the same number of features, otherwise, it won't be logical to do so. $\endgroup$
    – MXK
    Oct 29, 2020 at 9:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.