How to approach the numer.ai competition with anonymous scaled numerical predictors?

Question

Numer.ai has been around for a while now and there seem to be only few posts or other discussions about it on the web.

The system has changed from time to time and the set-up today is the following:

1. train (N=96K) and test (N=33K) data with 21 features with continuous values in [0,1] and a binary target.
2. The data is clean (no missing values) and updated every 2 weeks. You can upload your predictions (on the test set) and see the log-loss. Part of the test data is even live data and you get paid for good predictions.

What I would like to discuss:

As the features are totally anonymous I think that there is not much feature engineering we can do. So my approach is very mechanical:

1. inspired by this I use a classification algorithm to filter out those training data which fit to my test data best.
2. Figure out some nice preprocessing
3. train nice classification algorithms
4. build ensembles of them (stacking, ..).

The concrete question:

Concerning step 1: Do you have experience with such an approach? Let's say I order the probability of train samples to belong to test (usually below 0.5) and then I take the largest K probabilities. How would you choose K? I tried with 15K .. but mainly to have a small training data set in order to speed up training in step 3.

Concerning step 2: The data is already on a 0,1 scale. If I apply any (PCA like) linear transformation then I would break this scale. What would you try in preprocessing if you have such numerical data and no idea that this actually is.

PS: I am aware that because numer.ai pays people discussing this could help me make some money. But as this is public this would help anybody out there...

PPS: Today's leaderboard has an interesting pattern: The top two with logloss of 0.64xx, then number 3 with 0.66xx and then most of the predictors reach 0.6888x.

Thus there seems to be a very small top field and lot of moderately successful guys (including me).

Answer 1

I've looked at the approach and I'd select K by trying a range, i.e. 5k, 10k, 15k etc and then exploring the range in which the best result falls, say the best is 15k then I might do 13, 14, 15, 16, 17 and so on.

So far I've not found any pre-processing to be effective.

Answering the comment:

I've tried using LogisticRegression, SVM, Neural Networks, RandomForests, Multinomial NB, Extra Trees. All except Neural Networks using the implementations in sklearn. PyBrain for the NN.