8
$\begingroup$

I'm looking for recommendations as to the best way forward for my current machine learning problem

The outline of the problem and what I've done is as follows:

  • I have 900+ trials of EEG data, where each trial is 1 second long. The ground truth is known for each and classifies state 0 and state 1 (40-60% split)
  • Each trial goes through preprocessing where I filter and extract power of certain frequency bands, and these make up a set of features (feature matrix: 913x32)
  • Then I use sklearn to train the model. cross_validation is used where I use a test size of 0.2. Classifier is set to SVC with rbf kernel, C = 1, gamma = 1 (I've tried a number of different values)

You can find a shortened version of the code here: http://pastebin.com/Xu13ciL4

My issues:

  • When I use the classifier to predict labels for my test set, every prediction is 0
  • train accuracy is 1, while test set accuracy is around 0.56
  • my learning curve plot looks like this:

enter image description here

Now, this seems like a classic case of overfitting here. However, overfitting here is unlikely to be caused by a disproportionate number of features to samples (32 features, 900 samples). I've tried a number of things to alleviate this problem:

  • I've tried using dimensionality reduction (PCA) in case it is because I have too many features for the number of samples, but accuracy scores and learning curve plot looks the same as above. Unless I set the number of components to below 10, at which point train accuracy begins to drop, but is this not somewhat expected given you're beginning to lose information?
  • I have tried normalizing and standardizing the data. Standardizing (SD = 1) does nothing to change train or accuracy scores. Normalizing (0-1) drops my training accuracy to 0.6.
  • I've tried a variety of C and gamma settings for SVC, but they don't change either score
  • Tried using other estimators like GaussianNB, even ensemble methods like adaboost. No change
  • Tried explcitly setting a regularization method using linearSVC but didn't improve the situation
  • I tried running the same features through a neural net using theano and my train accuracy is around 0.6, test is around 0.5

I'm happy to keep thinking about the problem but at this point I'm looking for a nudge in the right direction. Where might my problem be and what could I do to solve it?

It's entirely possible that my set of features just don't distinguish between the 2 categories, but I'd like to try some other options before jumping to this conclusion. Furthermore, if my features don't distinguish then that would explain the low test set scores, but how do you get a perfect training set score in that case? Is that possible?

$\endgroup$
  • 1
    $\begingroup$ What did the data look like in 2 or 3 dimensions after you applied PCA, were there noticeable clusters ? What examples are being misclassified, is there a pattern ? $\endgroup$ – image_doctor Aug 12 '15 at 7:04
  • $\begingroup$ What do the power spectra of the traces look like? if you plot the mean spectra for each class, do they look different, if so how and can you optimise a classifier to capture that difference ? $\endgroup$ – image_doctor Aug 12 '15 at 7:12
  • $\begingroup$ 1) Can you show us the PCA cluster plot?, 2) Have you tried decision trees? If the original features are somewhat human-scrutinizable, you might be able to make sense of where it is going wrong. Otherwise (barring some silly bug on your part) it would seem your features are simply not discriminative enough. $\endgroup$ – lollercoaster Aug 12 '15 at 16:20
  • $\begingroup$ Very possible that the EEG data isn't separable, but have you inspected the training v test sets to ensure they have aren't biased (e.g. one has only positive examples or is normalized differently)? $\endgroup$ – jamesmf Nov 11 '15 at 13:05
  • 1
    $\begingroup$ Can you post the data somewhere? Either "allData" or "features_all" (without normalization and PCA). $\endgroup$ – stmax Mar 2 '16 at 14:10
1
$\begingroup$

To see if SVM can capture any signal at all, try to balance your data: create training and test sets that consist of exactly 50% positive and 50% negative samples (i.e., by subsampling randomly from whichever one is bigger). Also standardize the data (subtract the mean and divide by standard deviation).

(For balancing, you might try changing the class_weight parameter in sklearn, but we found the manual method (subsampling) to work better.)

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.