Timeline for Merging sparse and dense data in machine learning to improve the performance
Current License: CC BY-SA 3.0
8 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Apr 13, 2016 at 6:05 | comment | added | Sagar Waghmode | sorry about that. Will share the snippet of the code in few hours. | |
| Apr 12, 2016 at 19:01 | comment | added | Diego | Sagar, it is in your interest to give us as much information as possible. Do you mind giving a snippet of code so it is clear how do you feed the data to the classifiers and which exact SciKit classifiers are those? I have a similar set up that you have - dense + sparse datasets and I use the SGDClassifier for both + kind of majority voting. So I am interested to help as your case is kind of similar but so far it is a rather tedious task to get information out of you :-) My current hypothesis is that the sparse dataset is somehow neglected by your classifier when you bring the two together. | |
| Apr 12, 2016 at 17:27 | comment | added | Sagar Waghmode | I have tried out quite a few algorithms and settled on Gradient Boosted Model, also I do use Random Forests quite a lot for my problem. | |
| Apr 12, 2016 at 12:21 | comment | added | Diego | What predictor algorithm do you use? | |
| Apr 12, 2016 at 9:31 | comment | added | Sagar Waghmode | yes, I have done exactly that. Though the predictions are not entirely different, the number of samples where predictions differ are quite high (around 15-20%) of the data. For these samples model with sparse features performs better than that of model with dense features. My point is if sparse features perform better, why don't they come as important features in any of the models which I have tried so far. | |
| Apr 12, 2016 at 9:20 | comment | added | Diego | So do you see a lot of overlap then between the predictions from the two datasets? May be there indeed is no new information? I.e. the data tells the same story. | |
| Apr 12, 2016 at 8:15 | comment | added | Sagar Waghmode | I have already tried out the ensemble techniques as well as voting classifiers. Still no luck. | |
| Apr 12, 2016 at 4:30 | history | answered | Diego | CC BY-SA 3.0 |