3
$\begingroup$

This is my R script for a decision tree:

    library(caret)
    library(rpart.plot)
    library(plyr)
    library(dplyr)
    library(rpart)
    
    data("iris")
    
    names(iris) = tolower(names(iris))
    
    table(iris$species)
    suppressMessages(library(caret))
    
    index = createDataPartition(y=iris$species, p=0.7, list=FALSE)
    
    train = iris[index,]
    test = iris[-index,]
    
    trainctrl <- trainControl(method = "cv", number = 5, verboseIter = FALSE)

dt.model <- train(species~., data=train, method = "rpart", 
                  tuneLength = 10,
                  preProcess = c("center", "scale"),
                  trControl = trainctrl,
                  metric="Kappa")

dt.predict <-predict(dt.model, test)
confusionMatrix(dt.predict, test$species)

rpart.plot(dt.model$finalModel)
varImp(dt.model)

my feature importance are:

> varImp(dt.model)
rpart variable importance

             Overall
petal.width   100.00
petal.length   96.95
sepal.length   45.08
sepal.width     0.00

Is there a way to consider less petal.width? For example, I want that my tree use more petal.length and sepal.lenght than petal.width. Is it possible?


My problem is that in my dataset that I am using the decision tree on, one variable x is used more than the others. This x variable, however, affects the value of another y variable which is the one that actually characterizes my classes. Using an example with Iris: it's like saying that petal.length is related proportionally to petal.width, ie:

petal.width petal.length class
1            4           Virginica
1            6           Setosa
2            6            Virginica
2            7            Setosa

my model discriminates more the classes in base to the "petal.width" of my dataset rather than to the "petal.lenght". For this I would want to give a various weight to the characteristics

$\endgroup$
4
  • 1
    $\begingroup$ Erwan's answer is spot on! I want to know another thing: Why do you want to do that? To make one feature less important? $\endgroup$ Dec 17, 2021 at 4:28
  • 1
    $\begingroup$ I don't really understand your problem. Based on the iris example that you show, petal.width would not be used at all by the decision tree because it doesn't help to determine the class: whatever the value, the class is split 50%. In this example petal.length would be used because it determines the class at least for values 4 and 7. I understand that this simplified example might not show the problem that you want to express, but then you need a better example ;) It's also possible that the data isn't exactly as you think it is, it's common to discover counter-intuitive things in the data. $\endgroup$
    – Erwan
    Dec 17, 2021 at 18:31
  • $\begingroup$ In my original dataset I have several feature, but the model don't consider the most ones and it mainly considers one. This one it's not really important, it's like a counter. I can't exclude it because it is related on an other feature. So my question is: how can teach to my model to consider less this variable? $\endgroup$
    – Inuraghe
    Dec 20, 2021 at 10:43
  • 2
    $\begingroup$ @Inuraghe if the model considers this feature as the most important, then this feature is the most important in your data for predicting the target class. If it was just a counter as you say, then the model would ignore it. So you should try to understand why this happens: either there is some pattern in your data about this feature that you don't realize, or there is an error in the design or the code and the model doesn't do what you think it does. Currently you're trying to force the model to make "the wrong choice" statistically. You should instead investigate what happens in the data, $\endgroup$
    – Erwan
    Dec 20, 2021 at 18:33

2 Answers 2

1
$\begingroup$

In the comments you wrote:

I can't exclude it because it is related on an other feature

If you mean related as in a function of two other features, you could add columns to explicitly show those relations, then delete the original column. E.g. add a column called petal.width.plus.length and then remove petal.width.

Taking a step back, I think your problem could be rephrased as decision trees end up in a local minima, and so fail to find the best solution? If so, consider another algorithm? E.g. random forest, GBMs (e.g. XGBoost) or even deep learning.

If other algorithms also don't do what you expect, you should also consider that the data is not saying what you think it is. Maybe you need to collect more data, or oversample some rows, or look for bugs (e.g. some data has turned into NAs).

Finally (and I suppose this is the answer to the question you are asking) you can manually partition the data on the field you believe is most important, and then build a decision tree for each partition. As you need to do that each time you want to do predict you definitely want to write your own wrapper class to store the partitions.

$\endgroup$
2
  • $\begingroup$ In your opinion, might it be a good idea to use a regression to find a relation? So, I have this "counter" and I suppose it is my x (independent variable) and the other feature related is my y (dependent variable). Could a linear regression give me results (good or bad)? $\endgroup$
    – Inuraghe
    Dec 21, 2021 at 8:45
  • 1
    $\begingroup$ @Inuraghe Really hard to say without knowing more about your real data and what you are trying to do with it. But as linear models are quick and easy, it might be simplest to just try it? $\endgroup$ Dec 21, 2021 at 11:21
4
$\begingroup$

As far as I know there would be only two ways to use a particular feature "less" than its normal use by the decision tree:

  • Remove the feature completely.
  • Modify the data so that the feature becomes less important compared to the others (don't do it, this is complex and pointless)

It's important to understand that the decision tree learning algorithm decides which feature to use at every node by picking the feature which is the most discriminative in the data, i.e. the one which is the most informative to predict the target label. So using a particular feature "less" would mean that the choice of which feature to use is not based on the best way to predict the target according to the data... So it would not really follow the logic of training a decision tree.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.