8
$\begingroup$

Suppose I have a set of time-domain signals with absolutely no labels. I want to cluster them in 2 or 3 classes. Autoencoders are unsupervised networks that learn to compress the inputs. So given an input $x^{(i)}$, weights $W_1$ and $W_2$, biases $b_1$ and $b_2$, and output $\hat{x}^{(i)}$, we can find the following relationships:

$$z^{(i)} =W_1x^{(i)}+b_1$$ $$\hat{x}^{(i)} =W_2z^{(i)}+b_2$$

So $z^{(i)}$ would be a compressed form of $x^{(i)}$, and $\hat{x}^{(i)}$ the reconstruction of the latter. So far so good.

What I don't understand is how this could be used for clustering (if there is any way to do it at all). For example, in the first figure of this paper, there is a block diagram I'm not sure I understand. It uses the $z^{(i)}$ as the inputs to the feed-forward network, but there is no mention to how that network is trained. I don't know if there is something I'm ignoring or if the paper is incomplete. Also, this tutorial at the end shows the weights learned by the autoencoder, and they seem to be kernels a CNN would learn to classify images. So... I guess the autoencoder's weights can be used somehow in a feed-forward network for classification, but I'm not sure how.

My doubts are:

  1. If $x^{(i)}$ is a time-domain signal of length $N$ (i.e. $x^{(i)}\in\mathbb{R}^{1\times N}$), can $z^{(i)}$ only be a vector as well? In other words, would it make sense for $z^{(i)}$ to be a matrix with one of its dimensions greater than $1$? I believe it would not, but I just want to check.
  2. Which of these quantities would be the input to a classifier? For example, if I want to use a classic MLP that has as many output units as classes I want to classify the signals in, what should I put at the input of this fully-connected network ($z^{(i)}$,$\hat{x}^{(i)}$, any other thing)?
  3. How can I use the learned weights and biases in this MLP? Remember that we assumed that absolutely no labels are available, so it is impossible to train the network. I think the learned $W_i$ and $b_i$ should be useful somehow in the fully-connected network, but I don't see how to use them.

Observation: note that I used an MLP as an example because it is the most basic architecture, but the question applies to any other neural network that could be used to classify time-domain signals.

$\endgroup$
10
$\begingroup$

Clustering is difficult to do in high dimensions because the distance between most pairs of points is similar. Using an autoencoder lets you re-represent high dimensional points in a lower-dimensional space. It doesn't do clustering per se - but it is a useful preprocessing step for a secondary clustering step. You would map each input vector $x_i$ to a vector $z_i$ (not a matrix...) with a smaller dimensionality, say 2 or 3. You'd then use some other clustering algorithm on all the $z_i$ values.

Maybe someone else can chime in on using auto-encoders for time series, because I have never done that. I would suspect that you would want one of the layers to be a 1D convolutional layer, but I am not sure.

Some people use autoencoders as a data pre-processing step for classification too. In this case, you would first use an autoencoder to calculate the $x$-to-$z$ mapping, and then throw away the $z$-to-$\hat{x}$ part and use the $x$-to-$z$ mapping as the first layer in the MLP.

$\endgroup$
  • $\begingroup$ And in the last case, how would the weights of the other layers in the MLP be learned if the data is totally unlabeled? Or would that approach (i.e. autoencoder-MLP combination) only make sense if labels are available? $\endgroup$ – Tendero Dec 15 '17 at 20:21
  • $\begingroup$ Yes, an MLP (aka feed-forward neural network) is only really used if the data is labeled. Otherwise you have no info to use to update the weights. An autoencoder is sort of a 'trick' way to use neural networks because you are trying to predict the original input and don't need labels. $\endgroup$ – tom Dec 15 '17 at 20:27
  • $\begingroup$ So the only way to use a NN to do clustering would be the method you mentioned, right? Namely, use an autoencoder and then run a standard clustering algorithm such as K-means. $\endgroup$ – Tendero Dec 15 '17 at 20:28
  • $\begingroup$ That's the only way I know. If someone else has an idea I'd be happy to hear it. You might try other algorithms besides K-means though, since there are some pretty strict assumptions associated with that particular algorithm (but still it's a good thing to try first b/c it's fast and easy). $\endgroup$ – tom Dec 15 '17 at 21:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.