I'm trying some different techniques to optimise a Boosted Gradient Regressor by using an evolutionary programming technique to try and find the most efficient set of features. So far I've been having some good results (Been able to remove 65% of the features with an increase in accuracy), but I'm not impressed with how expensive each epoch is to run (in terms of time). Currently, this is a very expensive problem to optimise,
2^79 as there are a maximum of 79 features to choose from, which is
6.0446291e+23 possible feature permutations.
So far, each individual in my population has a binary encoded gene, where
1 = use this feature, and
0 = do not use this feature. Each individual in a population is evaluated by running the boosted regressor with the selected features, and then the RMSLE is calculated. I start to see good results around 3 hours into the optimisation function (I'm not doing any parallel computing).
I've been doing a little bit of research into how I can attempt to "predict" how good my boosted regressor will be without actually needing to evaluate its performance. So far I've found the following techniques:
- Problem Approximation. I can see how this would be useful in some specific cases, but as I'm not really able to reduce the complexity of evaluating how accurate my boosted regressor is it seems irrelevant.
- Functional Approximation. I think this is quite an interesting technique. They whole premise is to approximate the fitness of an individual without actually needing to evaluate the function
- Fitness Inheritance. This also seems rather interesting, and perhaps the most efficient of the methods listed above? Individuals are clustered into specific groups, and then the "represented" individual of each cluster has its fitness evaluated, and then the remaining individual's fitness are approximated by upon a distance measurement.
I could see some issues with these techniques though. If the combination of features used don't have a linear relationship with the output, then surely the function approximation would have a hard time approximating the potential fitness of an individual? I'm a bit stumped on what to further look at.