2
$\begingroup$

Working on my master's thesis, this is a problem I'm unable to find good resources about.

I'm working with data of 18 participants, who are either active or passive. Each participant is then subjected to a 3 x 3 experiment and results in a total of around 676 trials per participant (around 12.168 trials in total). There are 100 data points in each trial but cannot be used separately from the trial (since its an EEG epoch).

My data consists of 579 features, so I need some sort of feature selection as literature shows that most of them are irrelevant, but I want to use a bottom-up machine learning approach (to verify this).

Is there a rule of thump/literature to use for the amount of data needed for feature selection?

$\endgroup$
1
$\begingroup$

TLDR

Use all your data, throw a bunch of ensemble ML (probably just random forest) at it, and pick a good model. Usually, that works exceptionally well.

How much data should you use?

Ideally, you should always be separating your data into Train, Test, and Validation. Due to small dataset sizes, this isn't always possible, but it is still important to prevent overfitting. You can use all your training and testing data for feature selection and that shouldn't introduce any significant biases to your ML model. Your validation set should only be used to approximate the error once you are entirely done training your model.

Simple ML Solutions:

Use some kind of random forest or gradient boosting model on all your training/testing data. These models are designed for high performance in high dimensional data and by checking feature importances/permutation importances/mean decrease in accuracy you will know which features are/aren't important. It is common to use these kinds of models in the biological space where there are millions of features and only a thousand or so responses. Despite the disparity, I've still reached high accuracy in these situtations.

Simple Statistical Solutions

Measure the correlation of each feature and keep only those features that have an absolute correlation above/below a certain amount. You can use a few statistical tests to filter out features that are not significant. Here are a few examples:

  1. Pearson Correlation
  2. F-test
  3. Variance
  4. Lasso Regression (technically an ML algorithm)

Pitfalls of the Above

Each of the above uses some kind of assumption to figure out which features you should select for final model training. Sometimes you don't actually need to pick a subset of features ie when you use RF. Additionally, the statistical tests often miss abnormal types of correlations or miss multidimensional relationships. Feature selection is really a case-by-case decision that no-one can give you a definitive answer on without seeing the actual data you use.

$\endgroup$
2
  • $\begingroup$ In small datasets from a few participants, how does inter-participant variability affect the end goal? I've seen where folks use N-1 to train, then test the model on the Nth's participant's data. How valid is that? $\endgroup$ – sAguinaga Jan 21 '19 at 12:30
  • 1
    $\begingroup$ @sAguinaga Do you mean that if you have 10 datapoints, you use 9 to train and 1 to test? Assuming high dimensionality and that you are trying to classify (not regress), I'd suggest using very shallow learners to prevent overfitting and find a place where you minimize your type I/II error. Of course, the output will still be prone to error since you have a low sample size, but the above is the best solution I can think of. $\endgroup$ – Joe B Jan 22 '19 at 21:09
1
$\begingroup$

Have you considered an approach that wouldn't require you to do feature reduction? For example, if you were to use a neural network, the need for feature reduction is greatly reduced. Now, obviously I haven't seen your data so I don't want to suggest that neural networks should be your only answer. But with that many features I would be more concerned with how I'm actually going to reduce them rather than the number of data points that I need to collect.

$\endgroup$
2
  • 1
    $\begingroup$ I have, and I probably will also use some sort of ANN to compare performances (if I have time), but since I'm quite limited (time and performance of my notebook) some sort of feature selection is somehow required. For algorithms to use when doing feature selection I've looked at a few (RFE, mRMR and ReliefF). I prefer these since they (among others) can return some sort of feature ranking. But since FS on all the data seems counter-intuitive, I'm curious to find out if there is a rule of thumb that states "For FS you should use **X**% of the total data iff its distributed evenly across labels" $\endgroup$ – DevRavaege Jan 15 '19 at 5:46
  • $\begingroup$ @DevRavaege to answer your last question, no, I don't think you're going to find any hard & fast rules about this type of metric. There's way too much variety across all different types of datasets that this type of heuristic simply does not exist (nor should it). $\endgroup$ – I_Play_With_Data Jan 15 '19 at 12:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.