I am toying with AWS Machine Learning and I have a dataset with about 200 records with about 220,000 variables each.

Apparently, AWS Machine Learning has a hard limit of 1000 variables.

How can I still analyse my dataset through AWS ML? Should I apply a MDR procedure through my 200x200000 dataset or should I split my variables into chunks of 1000 and combine them somehow after processing each of those chunks through AWS ML?

  • $\begingroup$ Do you have an -omics data set or something; what kind of features do you have? You definitely should do some dimensionality reduction. $\endgroup$ – Emre Aug 23 '17 at 0:26
  • $\begingroup$ Yes @Emre, it's a ChIP-seq style omics dataset. Any recommendations on how to reduce the dimensionality to 1000 variables? Should I split the question into two parts? $\endgroup$ – 719016 Aug 23 '17 at 8:00
  • $\begingroup$ Multiplying the dimensions, it seems to me that your table should fit in memory, so you should be able to use MDR or some other feature selection method without resorting to big data software. If it came to the worst you could evaluate the mutual information for each column. $\endgroup$ – Emre Aug 23 '17 at 8:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.