1
$\begingroup$

I am new to this Data Science field. I have a question to apply Random forest to new data.

I have this table.

Y prop_A prop_B
A   0.8    0.2
A   0.7    0.3
B   0.5    0.5
B   0.4    0.6
B   0.1    0.9

I assumed that if the proportion of the group is high, chances are high that it is in the group. I built a model using random forest and test it with validation set (8/2 splits).

I thought the above model can be used for new data. This is an example of the data. The data structure and variable meaning is same, but the number of variable is different.

Y prop_C prop_D prop_E prop_F
-   0.8    0.1   0.05   0.05
-   0.6    0.3   0.05   0.05
-   0.5    0.4   0.05   0.05
-   0.4    0.2   0.4     0
-   0.1    0.5   0.4    0.4

The new data is unlabeled so I would like to make a label using the Random forest I used with previous data. Is it right approach to label the new data?

In the model, it doesn't works (due to different independent variables).

How should I do to label the new data based on a model using labelled data, which is different?

$\endgroup$
0
$\begingroup$

Yes, you can do that. However, how accurate your new labels are depends on the ability of your model to generalize to new data and the similarity of the new data to your training data. Therefore, it is something you need to test for in the first place by using a separate test dataset to assess model performance. In contrast to that, a validation dataset is used for model selection and hyperparameter tuning (e.g. max depth of your trees).

An in-depth coverage of the topic you can find in chapter 7 of "The Elements of statistical Learning" by Hastie et al.

With regards to your second question: If your training data does not contain the variables prop_C and prop_D then your model cannot make use of them. So either include these in the training data or ignore them in the new data.

$\endgroup$
2
  • $\begingroup$ Thanks to your comment. Ah, there might be wrong example. Actually, I have new unlabeled data, which is not matched with previous labelled data. It means, there are prop_C, prop_D, and prop_F in the new data. I think the tendency (more proportion, more likely to be the group) can make the label from unlabeled data. Is it not right approach? $\endgroup$
    – jhyeon
    Jan 7 '20 at 7:24
  • $\begingroup$ @Juhyeon That is possible but not guaranteed. I suggest to use a test dataset which comes from the same distribution as your new unlabeled data to get an estimate of how accurate your labels are with regards to this new data. If that is not available manual labeling could be one approach to obtain it. $\endgroup$
    – Sammy
    Jan 7 '20 at 7:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.