1
$\begingroup$

Is there a way in fitting a decisionTreeClassifier in SKLearn to sparse tuples? The data that I have is based on about 100 features, but only a few of them are ever used to make the decision. Effectively, each row of data is a sparse tuple.

My input data is 30k entries like:

(yes if f1=v1 and f3=v2) 
(no  if f1=v3 and f5=v4)
...

and so on.

There are tens of thousands of entries and the conclusions are not necessarily consistent. My goal is to find an automated method to reduce this down to tens of nodes that approximate the same behaviour. I need access to the resulting structure - which is why I have considered using SKLearn rather than, say DataRobot.

My understanding of the use of SKLearn decisionTreeClassifier is that I have to convert this to a array of tuples of 100 values, and have the features just be given an index number. But, that means I have to provide a value for every one of the 100 features on every row. And that does not well represent the data that I have.

This is like having data to be classified looking like

(yes,v1,__,v2,__,__)
(no ,v3,__,__,__,v4)

Hence, the interest is in, for example, being able to list the value of the data as None to indicate that there is no constraint on that field.

The data as given has only equalities. But, the expected output will have inequalities, since that appears to be what SKLearn provides. I would really prefer a decision tree with equalities and inequalities.

Improve this question
$\endgroup$
10
  • $\begingroup$ I'm not sure what you mean by a don't care value for a feature. The example you show tests two equalities. Decision trees don't work like that. Each node tests X > t, where X is the feature's value, and t is some threshold. If you don't want a decision tree to use a feature, simply remove it fromthe dataset. $\endgroup$
    – pseudoabdul
    Jan 12, 2022 at 1:56
  • $\begingroup$ @pseudoabdul I will expand on the terms and concepts in the main body of the question. $\endgroup$
    – Bruce
    Jan 12, 2022 at 2:46
  • 1
    $\begingroup$ I do not quite understand the problem. But if you are looking for a tree tool that supports categoricals and missings - H2o, rpart, lightgbm, catboost. There are also other encodings you can do for the categoricals. Target, riser, and more. Patsy implements many encoding schemes. Once encoded, you can use any tree package. I do not use 1-hot in my work since there are better options. I also searched for sparse trees and high dimensional trees and found some papers that might help you. $\endgroup$
    – Craig
    Jan 13, 2022 at 10:43
  • 1
    $\begingroup$ rpart for R. You can call R from Python, checkout rpy2, but rpart is an R package. H2o, lightgbm, catboost all can be called from R or Python. xgboost, if you encoding the categorical yourself can also be called by R and Python. $\endgroup$
    – Craig
    Jan 14, 2022 at 11:25
  • 1
    $\begingroup$ Glad you are getting closer to solving the problem. rPart uses surrogate splits and here for missing, if configured. Surrogates are other split features to use for missing. Say Feature A is the best splitter. Calculate surrogates (the 2nd, 3rd, ... best splitter) to use if feature A is missing. If surrogate is missing, move to the next. If your data is very sparse may need a lot of surrogates or surrogates may not work. $\endgroup$
    – Craig
    Jan 20, 2022 at 11:26

1 Answer 1

Reset to default
0
$\begingroup$

My own investigation on this forum and on SKLearn bug reports and forums is that the core of the problem - that SKLearn decision tree classifier does not handle categorical or sparse data - has been complained about for some years with no important changes to the code in this respect.

Three approaches suggest themselves. Firstly, expand out the tuples to non-sparse with some default value such as 0 for all the don't-care cases with the idea that the constancy of this value will result in SKLearn being forced to use other values to make the decisions on. Secondly, to shift the circus to R - where, apparently, the equivalent routines actually do handle categorical and sparse data. Or, thirdly, write the decision tree training code myself from scratch.

It is my intention to come back later and tell you how that worked out for me.

Improve this answer
$\endgroup$
2
  • 1
    $\begingroup$ Hi @Bruce, how did you end up solving the problem? $\endgroup$
    – Dudelstein
    May 31 at 8:44
  • 1
    $\begingroup$ Hi @Dudelstein, the short version is that I ended up pretty much writing everything myself. I did use the entropy partitioning part of the R module code. But, the rest was pretty much all mine. I have not had time to get into the Quine reduction stuff, as shortly after asking this question most of the code was switched to Sagemaker and Python anyway. I filled blanks with a very large number so that if the module used it, it would be very obvious. $\endgroup$
    – Bruce
    Jun 1 at 5:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.