2
$\begingroup$

Problem : There are several events (eventA, eventB,....) represented by waveforms. For each event there are several csv files (eventA1.csv, eventA2.csv,...eventAn.csv) having the points(x,y) from which the corresponding waveform can be generated.

Using these waveforms as training data, waveform segments on (test data) larger data should be identified i.e. the interval during which an event occurs is needed to be labelled on the test data (See image for reference).

  1. What would be the best strategy, tools and algorithms to address this problem?
  2. If classification model is to be generated then,

    1. The points are just to describe the nature of the waveform i.e. the test waveform will not be on same points but with similar nature e.g. the concavity, convexity, slope etc.
    2. Different events in training graphs are represented using different number of points i.e. event A can be determined by 20 points, while event B has 200 points.

      So,

      1. How do I create feature vectors to include all these things for classification?
      2. How would the test data be analysed, as it has large number of points on the large waveform i.e. how to segment it to generate features and give it to classification model?

enter image description here

$\endgroup$
  • $\begingroup$ This seems to be a signal processing problem. Could you either resample your signal to be relative to your signal features, or vary the size of your signal features? Then you could convolve the signal features with the test signal and retrieve an impulse where there are similarities. $\endgroup$ – Hobbes Aug 10 '16 at 22:14
2
$\begingroup$

Statistical Data Analysis

I would start with the following:

  1. Statistically analyse your training events and see the mean and the variance of length and width and other properties to have an overall image of what they are.
  2. For each event take the mean and the standard deviation.
  3. Walk through the test data and compute the error between test signal and learned events and plot it.
  4. At the end for each event type, you will have an error curve whit many local minimums.
  5. Choose a threshold based on standard deviation of training events and detect minimums behind that threshold as events.

Pattern Recognition

More advanced option is to learn a model to classify events. At the beginning I should say the last question is theoretically wrong because you do not generate features from test data but only compute them. Features have been extracted (generated) from training data. I assume you mean how to feed the model with test data for classification according to its size. Well, I'd say you can segment it to chunk which may lead to losing some events because of cutting points but if test data is large enough that would not be a problem.

The point is that time-series are a special kind of data and there are methods specifically for them so the classic feature extraction and learning may lead you to an ocean of confusing methods. I suggest to search the term Time-series Mining instead to see more direct solutions.

Time-Series Mining

The very first approach I proposed was kind of this! There are many more algorithms for detecting similarity based on two time-series mostly based on Euclidean distance. For example look at LCSS or DTW. These methods are what you are looking for and both are advanced version of my first suggested algorithm. Please note that for all of them normalizing your training and test data plays an important role.

See this answer as well however that is about unsupervised segmentation of time-series.

$\endgroup$
  • $\begingroup$ Thanks for providing a deeper insight into possible solutions! Can you provide some links to statistical segmentation examples using time series, as I couldn't find any relevant source? Moreover, on discussions, I also came across that "Hidden Markov Models" can also be used here, whats your say? Any links? $\endgroup$ – Makarand Aug 22 '16 at 8:34
  • $\begingroup$ Actually I provided the link to a paper in the answer I hyeperlinked for you. It's here lancs.ac.uk/~khaleghi/Publications_files/khaleghi16a.pdf . But 2 points: 1) it's an unsupervised method and you need to modify it to your question as I stated in my answer. 2) The paper is deeply mathematical and if you are not that into theoretical background of time-series analysis then that might be pretty difficult to follow. I would suggest you to have a look at web.science.mq.edu.au/~cassidy/comp449/html/ch11s02.html for some insight into DTW plus explanations of HMM usage as u asked $\endgroup$ – Kasra Manshaei Aug 22 '16 at 11:47
  • $\begingroup$ But my favourite tutorial is one published by Springer in which DTW is explained theoretically rigorous but totally understandable. The great point about that tutorial is that it contains subsequence matching which is exactly your problem. I could not copy the link here but if you google "dynamic time warping" one of top 5 results would be a PDF file from Springer. I'd say go for that directly. $\endgroup$ – Kasra Manshaei Aug 22 '16 at 11:50
  • $\begingroup$ About HMMs: As you will see in the second link, HMMs are pretty fine for your application but again; it depends on your background. If you are from Machine Learning community go for it otherwise it might be a bit overdeep mathematically. But after all, HMMs are great and do the job like perfect. The best HMM tutorial is by Andrew Moore from Carnegie Mellon. His tutorials are widely read in ML community because they are just great! Here you can find them: autonlab.org/tutorials/hmm14.pdf $\endgroup$ – Kasra Manshaei Aug 22 '16 at 11:54
  • $\begingroup$ @KasraManshaei great summary for anybody starting with solving similar problems. Do you see any problems in using this approach to analyse data from Googles project Soli radar? $\endgroup$ – Heisenbug Aug 21 at 17:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.