6
$\begingroup$

I have a dependent variable measuring the net revenue. One of the major predictor affecting this is "product" i.e. the product sold to the customer. My randomly sampled dataset contains 1.4 million entries.

Products are assigned a specific categorical value. I feel that using dummy variables to represent the products would be apt however, there are 4481 levels of products. I do not know how to code so many levels in R.

model.matrix(~ product, data=salesdata) returns an error. (Needs 38.4GB of memory)

Can someone guide me a little on how to code these categorical variables?

Dependent: Net revenue (quantitative) Independent: Product code (quantitative but treated as qualitative since values are nominal)

Improve this question
$\endgroup$

2 Answers 2

Reset to default
3
$\begingroup$

You can use either sparse matrices or feature hashing.

Sparse Matrix

I suppose that using a sparse matrix is the only choice. I suspect that this line of code will work. This uses the Matrix package.

sparseProducts <- sparse.model.matrix(~ product, data=salesdata)

Take my example:

sparseDiagonalMatrix <- sparse.model.matrix(~., data.frame(V1 = as.factor(seq(1, 10))))

each column represents a different factor, this will yield:

1  1 . . . . . . . . .
2  1 1 . . . . . . . .
3  1 . 1 . . . . . . .
4  1 . . 1 . . . . . .
5  1 . . . 1 . . . . .
6  1 . . . . 1 . . . .
7  1 . . . . . 1 . . .
8  1 . . . . . . 1 . .
9  1 . . . . . . . 1 .
10 1 . . . . . . . . 1

> class(sparseDiagonalMatrix)
[1] "dgCMatrix"
attr(,"package")
[1] "Matrix"

alternatively you can remove the intercept and have all zeros represent class 1

sparseDiagonalMatrix <- sparse.model.matrix(~., data.frame(V1 = as.factor(seq(1, 10))))[, -1, drop=FALSE]

10 x 9 sparse Matrix of class "dgCMatrix"
V12 V13 V14 V15 V16 V17 V18 V19 V110
1    .   .   .   .   .   .   .   .    .
2    1   .   .   .   .   .   .   .    .
3    .   1   .   .   .   .   .   .    .
4    .   .   1   .   .   .   .   .    .
5    .   .   .   1   .   .   .   .    .
6    .   .   .   .   1   .   .   .    .
7    .   .   .   .   .   1   .   .    .
8    .   .   .   .   .   .   1   .    .
9    .   .   .   .   .   .   .   1    .
10   .   .   .   .   .   .   .   .    1

> class(sparseDiagonalMatrix)
[1] "dgCMatrix"
attr(,"package")
[1] "Matrix"

You will need a package that supports sparse matrices for measuring the net revenue though. Fortunately, most modern mainstream packages support sparse matrices.

Feature Hashing

Here is a great explanation of feature hashing in R (among other techniques) which is also an alternative, specially useful when you have hundreds of thousands or millions of multiple levels.

https://amunategui.github.io/feature-hashing/

Improve this answer
$\endgroup$
3
  • $\begingroup$ I will try your method and let you know. The link that you have provided is really helpful! Alternatively, can I use multinomial logit regression? It seems to support categorical as well as quantitative variables $\endgroup$
    – Anonymint
    Commented May 3, 2016 at 5:29
  • $\begingroup$ You have alternatives there like SparseM, a linear kernel svm from e1071 (equivalent to logistic regression) or MatrixModel (See ?MatrixModels:::lm.fit.sparse) $\endgroup$
    – wacax
    Commented May 3, 2016 at 17:29
  • $\begingroup$ by the way, xgboost also supports sparse matrices. $\endgroup$
    – wacax
    Commented Sep 16, 2016 at 17:00
2
$\begingroup$

For the most part, models built in R (for example, linear regression using lm ) can handle the categorical data coded as factor and do not need any dummy coding. You just need to do this before passing data to lm:

salesdata$product <- factor(salesdata$product)

So, depending on the model you are about to build, you might not need to create dummy variables.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.