Dear fellow data scientists

I work in finance and have decent experience with neuronal networks. Right now I am facing a challenge where I need to classify a set of signals.

The data set looks roughly as follows (signal can be thought of as a vector which might also contain floats instead of being binary):

Signal - Target
001110   0
001110   1
001100   1
011100   0
...      ...

The problem is that the data is really noisy. That means that most of the signals do not really have a lot of predictive power to forecast the target. That makes using a neuronal network (or other classical classification algorithm) rather impractical since we will be fitting a lot of noise. At the same time, simply looking at correlations between signals and targets does not help, since the signals themselves are noisy. So e.g. [0,0,0,1,1,1] and [0,0,0,1,1,0] are not the same, but similar and both signals might be a very good predictor for, let's say, the target 1.

My goal is to have an algorithm that tries to find the 10 good signal "types" (in a fuzzy way) in the haystack of 1000s of signals. I am looking for suggestions as to what that algorithm could be.

One idea would be to first cluster the signals and then use the clusters as input, but maybe there are better suited algorithms exactly for such tasks.

Hope my question is clear. :)

  • $\begingroup$ this might be relevant or not: datascience.stackexchange.com/a/63543/64377 $\endgroup$
    – Erwan
    Commented Nov 22, 2019 at 12:36
  • $\begingroup$ This very much looks like a clustering problem. Do you have reasons to think there could be better approaches? $\endgroup$ Commented Nov 22, 2019 at 12:40
  • $\begingroup$ I don't have any specific reason except for the feeling that I just may not know about a better suited algorithm. For example, I was once tackling a market phase regime detection / classification problem and I found that Hidden Markov Models where just so much better suited for the task than neuronal networks, etc. Ideally the model would be clustering and classification all-in-one in a sense if you know what I mean. Like when a human looks at a chart and identifies a certain pattern in the noise (which might slightly differ from time to time) that is a good predictor for e.g. a trend. $\endgroup$
    – Neo
    Commented Nov 22, 2019 at 18:06


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.