2
$\begingroup$

I am currently discretizing my training set in R with discretize from the bnlearn package.

library(bnlearn) 
discretize(train, method = "quantile", breaks = 2)

The lower bound of one interval and the upper bound of the other interval are the minimum and the maximum of the respective column.

Executing the same command on the test set

discretize(test, method = "quantile", breaks = 2)

will result in a different discretization, as the minimum and the maximum will likely be different on the test set. Is it possible in R (with another library or command) to transfer a discretization from a training set to a test set?

Apparantly this is easy to do in Weka and Orange, however, I would prefer to do this in R (not using RWeka).

$\endgroup$
2
$\begingroup$

You can obtain a master-set of cutpoints by using arules::discretize(x, ..., onlycuts = T). Then, do as @hssay does in his/her answer.

library(arules)

train <- data.frame(dat = runif(100))
test <- data.frame(dat = runif(100))

mastercuts <- arules::discretize(train, method = "interval", categories = 4, onlycuts = T)

train$bin <- as.numeric(cut(train, breaks = mastercuts))
$\endgroup$
3
$\begingroup$

You need to save the breaks/endpoints when you perform the bucketing/discretization on the training set as a (named) vector. The same breaks/endpoints can be then re-used on test set.

The I am giving below sample code using cut function in base R to keep the answer more generally applicable (rather than giving answer specific to bnlearn package). Note the way output of quantile function is saved in vector breaks_to_use in line 3 below and re-used while applying cut on test data. You can (hopefully) do the same using the breaks argument in discretize function.

training_data <- runif(100)
test_data <- runif(100)
breaks_to_use <- quantile(training_data, seq(0, 1, 0.25))
discretized_training_data <- cut(training_data, breaks = breaks_to_use)
discretized_test_data <- cut(test_data, breaks = breaks_to_use)
$\endgroup$
3
  • $\begingroup$ Does that not assume the minimum and the maximum are the same across the training and the test set? Since the upper and lower boundaries in the discretization are not infinity/-infinity, these boundaries inevitably change. $\endgroup$
    – castle
    Sep 14 '16 at 9:06
  • $\begingroup$ Yes. That would be a problem. But, I would expect the training set and test set to be having (almost) same distribution for data. If it is not the case, the model will not work well on test data irrespective of the discretization strategy employed. You can edit the breaks such that the minimum and maximum in test set always go to extreme bucket on either side. $\endgroup$
    – hssay
    Sep 14 '16 at 11:08
  • $\begingroup$ Except for building this functionality oneself, are you aware of a way to have this done automatically? Editing the breaks manually becomes impossible when working with large data sets. $\endgroup$
    – castle
    Sep 14 '16 at 13:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.