1
$\begingroup$

I have a .csv file with data in the following form:

moment_1;moment_2;moment_3;force_x;force_y;force_z;...
-0,02131267;-1,6032766088;5,9906811787;5,40010285;0,0203;86,44227467;...
2599;-1,70091039344;-1,3044809;-0,0406673590;-2,60896180797;43,2334;...

The file is very large and I need to put it in an interactive visualization, that's why I need to reduce the data points without changing the overall structure too much.

Many data points are very close to each other as seen in the following image:

too close

My approach was to define a threshold and filter all points which have a distance to the previous point lower than the threshold. But I think that's not an optimal solution because, when I remove one index, I need to remove it from the other data array too, otherwise the structure is changed.

Are there better approaches?

$\endgroup$
3
  • $\begingroup$ What are you using to process the data? Are you trying to reduce noise in the signal as well? $\endgroup$ Aug 21 '17 at 15:24
  • $\begingroup$ Currently, I just display the data in a visualization. Yes, I'm trying to figure out how I can smooth the data. $\endgroup$
    – Chryb
    Aug 22 '17 at 9:49
  • $\begingroup$ Have a search on dsp.stackexchange.com It has more questions on smoothing / subsampling. Is this from a gyro / accelerometer? $\endgroup$ Aug 22 '17 at 11:36
2
$\begingroup$

Instead of filtering single points I would suggest that you smooth your data using established techniques, e.g. Savitzky–Golay filter. Another option would be to employ Kernel Density Estimation, where you can then visualize the curves using a reduced, regular set of supporting points.

$\endgroup$
1
  • 1
    $\begingroup$ Thank you. I'm using Savitzky–Golay filter to smooth the data and Ramer–Douglas–Peucker algorithm to reduce the data points. $\endgroup$
    – Chryb
    Aug 22 '17 at 12:53
0
$\begingroup$

You could try a linear filter like a Gaussian. It will smooth the data and has the property that no new local maxima or minima will be created (see Gaussian scale space theory).

However, I think a median filter (non-linear) may give you better results, as it will remove single spikes but preserve step edges. The kalman filter is also suggested in one of the answers here: https://dsp.stackexchange.com/questions/4680/what-is-smoothing-in-very-basic-terms/4682#4682

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.