0
$\begingroup$

I have a data set of 80,000 samples (40k 3 axis accelerometer and 40k Gyro data). I am trying to implement KNN and Random Forest for activity recognition on ESP8266 Node MCU. The limited memory of the MCU is the bottleneck of the process.

Is there any method that can reduce the dataset to, say 5000 records without losing any vital information and without affecting overall accuracy? Dimensionality reduction, as I could understand with my non-mathematical background, is reducing the data by dropping less important columns. However, in my case, I cannot drop any column (only 6 columns are there which are x,y,z values of accelerometer and gyroscope).

Sample data:

enter image description here

$\endgroup$
3
  • $\begingroup$ undersampling maybe..? $\endgroup$
    – Nikos M.
    Apr 11 at 18:17
  • $\begingroup$ The sampling rate is pre-specified @40 data/sec. So can not alter the input data. Any alterations may be done to the collected data only. $\endgroup$
    – Bukaida
    Apr 12 at 4:03
  • $\begingroup$ Someone has rephrased few sentences ( although with typo ) in the OP and downvoted the question. It is a technical question which the community people had no problem in understanding so far, found interesting and trying to help. I do not get the logic of downvoting a technical question due to a few linguistic errors ( not complained by Grammarly either) $\endgroup$
    – Bukaida
    Apr 12 at 4:40
1
$\begingroup$

You are the domain expert on this, but these are the two practical approaches, I have utilized depending on the use case and effect on performance:

1. Downsample: Let say, randomly choose 5,000 out of 80,000 records. To maintain the same proportion of classes as in the population, go for stratified random sampling.

2. Reduce the precision of measurements (i.e. less decimal points) and aggregate data to capture the frequency of similar records as a new feature. If you have repeating records you can do this without compromising on the precision too.

$\endgroup$
4
  • $\begingroup$ I need to capture the data atleast @40hz ( 40 data/sec) as per the requirement. Random sampling from the population may need a basis, I think ( Not an expert in statistics). What can be such basis, I mean sampling/filtering criteria? $\endgroup$
    – Bukaida
    Apr 11 at 19:14
  • $\begingroup$ @Bukaida Sampling from an unknown multivariate distribution is a whole stats topic in itself. Try finding it on stats StackExchange. The whole idea is to obtain an approximate representation in an unbiased way within a certain margin of error. $\endgroup$ Apr 11 at 20:00
  • $\begingroup$ @Bukaida updated for maintaining class proportions. $\endgroup$ Apr 11 at 20:10
  • $\begingroup$ I need to study stratified random sampling. Seems like it may solve my probem. $\endgroup$
    – Bukaida
    Apr 12 at 3:57
0
$\begingroup$

You can use PCA (Principal Component Analysis) that is a technique of dimensionality reduction. Basically you will change the basis of the data, you will not drop any columns, minimizing information loss. So, for sure you will lose some information, but it's up to you decide how much information lose, depending on the number of principal components you will compute.

$\endgroup$
10
  • $\begingroup$ I am going through the PCA method and hope it can show some logical way to decrease the volume. I just want to reduce the number of records, not attributes. $\endgroup$
    – Bukaida
    Apr 11 at 19:19
  • $\begingroup$ With PCA probably you will avoid the problem of bottleneck of memory. Otherwise you have to cut the dataset and take only some rows if you want to reduce the number of rows. $\endgroup$
    – CasellaJr
    Apr 11 at 19:34
  • $\begingroup$ I have started reading on PCA. However, my initial impression is that it is used to reduce the number of variables (Need to study further), which is basically the attributes(columns). In my case, a reduction in number of records (rows) is required. Can PCA achieve that with any kind of modification ( example, interchanging rows and columns). Pardon me if I propose any silly thing, due to my lack of statistical background. $\endgroup$
    – Bukaida
    Apr 11 at 19:44
  • $\begingroup$ No, it is not possible this solution you proposed. However, if you have a bottleneck of memory, pca will reduce the number of information to be processed, so your dataset will be less heavy. You can try. $\endgroup$
    – CasellaJr
    Apr 11 at 19:50
  • $\begingroup$ I have 6 variables only, Ax, Ay Az and Gx, Gy, Gz. Dropping any of it is not an option. Is it possible to take the mean and select the required number of data around it based on some standard deviation parameter? How will it affect overall accuracy? $\endgroup$
    – Bukaida
    Apr 11 at 19:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.