I'm trying to build a model to solve a regression task.
Simplified, the data look like:
7.16, A 1, B 4, C 15, D 3 8.21, C 3, D 2, G 7, M 4, Y 9 1.85, D 3, N 1, L 1
The first column is a label, consequent columns are a class and respective number.
For example, the first row demonstrates that there is one of A, four of B, fifteen of C, and three of D - such a configuration gives us 7.16.
My Primary Question:
How do I represent my data so that I can fit a Random Forest Regression model?
Figure out the whole list of classes (there are 26 letters, but I have way more classes) and put zeros for all of the unknown classes into each record.
Vector.sparse()might help here as it can reduce the amount of required memory.
Figure out the entire list of classes, correspond a number starting from 0 and convert each record into a vector (i.e. the first one would look): 0 1 1 4 2 15 3 3. Then each column is treated as a separate feature, but in this case I have to specify that some of my features are categorical and that's not possible with Spark since different records are of different length (In Spark I have to provide columns of features, which are categorical with number of classes. In the example that would be a map(0 -> 26, 2 -> 26, 4 -> 26) - like every second column is a class with 26 categories).
It's possible to build features like A:1, B:4, C:15, D:3 and treat all of them as categorical. Or move decision burden (categorical vs continuous) to Spark itself.