My dataset contains half a million unlabeled entries with over 100 binary features. A third of these features are present in less than 1000 samples.
I want to classify a few samples by hand (into one of two classes) and use them as trainings set for either an SVM or a neural network.
How do I select samples for classifying them by hand? Are there some preferred methods? I fear that by selecting them randomly not all possible features are present in my training set.
Do I have to select the training set and the dataset used for later evaluation in the same way? For example could I just use 10 examples of each feature usage for training and random samples for evaluation?
Is there a rule of thumb about how large my training set has to be?