0
$\begingroup$

I am studying caret package in R. I know in this package, all the data are assumed to be numeric. But I always have some data that has a lot of strings or factors. For example, a product category could include hundreds of thundred of strings indicating all the different products. A worker ID list may includes a lot of numbers. They look like numeric, but you cannot do any calculation because it does not make any sense. They are actually supposed to be characters or strings. So, we cannot use model.matrix or dummyVars to transfer them to numeric because 1). It should not be transferred like this; or 2). The matrix will be so huge after transferring that we cannot deal with. So my question is: How can we deal with an attribute with a lot of characters or a lot of so-called numbers when using caret package in R?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

Yeah, some (not all) libraries have that numeric input limitation.

What you're looking for is called "one-hot encoding" which can basically convert an N level factor like your product categories to N binary columns. Here is a post describing a few ways of doing that:

https://discuss.analyticsvidhya.com/t/how-to-do-one-hot-encoding-in-r/6982/5

Improve this answer
$\endgroup$
3
  • $\begingroup$ Thanks for your answer. But this is not what I meant. caret package can do this when there is a few factors in a variables. But I am asking what if there are thundreds of factors (or characters) in one variables? For example, a variable of product includes 1000 products like iphone, printer, toner.... So the current method cannot deal with this situation. $\endgroup$
    – Feng Chen
    Apr 3, 2017 at 3:08
  • $\begingroup$ Explain why you cannot deal with it. Is it giving an error when creating the variables or the model doesn't like so many? There are algorithms (e.g. Vowpal Wabbit) that can handle 10^12 features. $\endgroup$
    – CalZ
    Apr 3, 2017 at 11:54
  • $\begingroup$ Thanks. Let's say I have a product variable including 1000 product names. When transfer this variable to a set of variables that can be dealt with in caret package using the method mentioned in the above answer, Every product name becomes a new variable. So, I have 1000 more variables. So, the matrix becomes so huge that it cannot be processed by my desktop. I take 1000 as an example. In fact, there are about more than 10 thousand product names in my data. Here I am just trying caret package in R, I have not used Vowpal Wabbit yet. Thanks $\endgroup$
    – Feng Chen
    Apr 3, 2017 at 23:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.