1
$\begingroup$

I am relatively new on data science sector. I started tampering with random forest models and some strange occasions aroused. To be precise, I have data from sensors that record pollutants and a column that acts as the labels populated with data that shows if a person has developed allergies or not depicted with 0 or 1. The amount of data is 1300 people. So i have run some random forest classifier models and started to remove or add some columns. What i dont understand is that: 1. In a model with all pollutants as features and the column SYMPT as labels when i remove a random column (ex styrene) the order of the feature importance changes significantly, 2. In a model with all pollutants as features and the column SYMPT as labels when i remove a large number of columns or even let just 1 column of the pollutants the accuracy remains the same (about 71%) and lastly 3. when i added the column with AA (increasing serial number) data which was originaly excluded from the model along with SBR column, the accuracy increased. What can you understand from these results?

Portion of Dataset(RF without AA and SBR)

$\endgroup$
2
  • $\begingroup$ I have the feeling, I do not fully understand point 2 and point 3: Ff you remove columns the accuracy stays the same but if you add columns, it increases. What if you remove the column that you added before? It seems that there are some details unsaid. And what do you call AA data? $\endgroup$
    – Broele
    Mar 10, 2023 at 10:04
  • $\begingroup$ I mean AA column which is an increasing serial number. I only add this column. In the original model it is excluded along with SBR column which also has identifier data. $\endgroup$
    – Sirick
    Mar 10, 2023 at 11:36

1 Answer 1

2
$\begingroup$

Welcome to the data science sector. Your three points seem to relate to different aspects. I will try to address all three:

1. Feature Importance

To explain the effect, I would go the other way and start with a model trained on few features that are mainly uncorrelated (just for the sake of explaination).
One our features will get the highest feature importance (let's call it A). To simplify things, this roughly means that the feature is used in a lot of important nodes of a lot of trees of our random forest.
Now assume we add another feature (B) that is very strongly correlated with feature A. If we now train another random forest, it does make a huge difference whether feature A or feature B is used in a node. So approximately half of the time A is used and half of the time B is used. This means, A only appears in half of the nodes compared to the first model. Which would give A half of the feature importance it got before (again, we simplify things. Depending on the definition of feature importance, the effect might differ, but the general concept stays the same)

Following the above explanation, one can see, that removing some features will affect the other feature's importance differently, depending on how strongly they were correlated with the removed features. This might already explain your observation.
Conclusion: Feature importance measures only the importance for one concrete model. Not the value of the information behind the feature.

2. Accuracy

It looks like you have some sort of imbalanced data. I suggest to look for the frequency of occurances of values in SYMPT. I assume there are just two values 0 and 1 and 71% of all values are labeled 1. In that case, a model that always says 1 would get an accuracy of 71%. So in that case, if you models would not lear much, all would get 71% accuracy.

In such a case of implanaced data, one better uses othere metrics to evaluate the model, such as roc-auc, precision and recall (look a both), true- and false-positive-rate.

3. Adding Column AA

It looks like there is a trend in your data. If the ID-Column is a valuable information, then the data might be sorted in some way. If you are not sure how the data is sorted, it would be the best to ignore the ID and to shuffle the data. Otherwise, you need to analyze the data to understand the ordering.

Note: In scikit-learn you need to explicitly set set shuffle=True for cross validation (see for example https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.KFold.html).

I hope this helps you a bit to understand what is going on with your models

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.