I'm not sure how to implement this architecture. I'm following this thesis (pages 17-19) or this paper but I'm not sure how to train it. I want to use this to extract features from raw audio.

I know I have to compute the positive and negative correlations, but I don't know how to do this exactly since I can not find any detailed documentation of this.

What I have done so far is:

Positive correlation

To compute it I do a matrix multiplication: $$\langle xh\rangle^+ = \frac{x_{data}^Th_{act}}{batchsize}$$

Where $x^T_{data}$ is be my batch data transposed and $h_{act}$ are the hidden units activations. $h_{act} = relu(x_{data}W + bias_{hidden})$. Not sure if I have to add noise in this point.

Negative Correlation

To do this I use contrastive divergence (no tag??) with $k=1$

Same thing: $$\langle xh\rangle^- = \frac{x_{free}^Th_{free_{act}}}{batchsize}$$

but I'm really not sure how I have to compute these terms. So for me $x^t_{free}$ would be the samples of $\mathcal{N}(ΣWh+b;Σ)$, where $h$ here is activation from previous step ($h_{act}$) without noise. Lastly $h_{free_{act}}$ is computed as the same as the previous $h_{act}$ but using $x^t_{free}$

Is this correct? Because I don't know when I have to use samples or activations or maybe meanfields


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.