# Training Gaussian Restricted Boltzmann Machines with Noisy Rectified (nrelu or ssu) linear hidden units

I'm not sure how to implement this architecture. I'm following this thesis (pages 17-19) or this paper but I'm not sure how to train it. I want to use this to extract features from raw audio.

I know I have to compute the positive and negative correlations, but I don't know how to do this exactly since I can not find any detailed documentation of this.

What I have done so far is:

# Positive correlation

To compute it I do a matrix multiplication: $$\langle xh\rangle^+ = \frac{x_{data}^Th_{act}}{batchsize}$$

Where $$x^T_{data}$$ is be my batch data transposed and $$h_{act}$$ are the hidden units activations. $$h_{act} = relu(x_{data}W + bias_{hidden})$$. Not sure if I have to add noise in this point.

# Negative Correlation

To do this I use contrastive divergence (no tag??) with $$k=1$$

Same thing: $$\langle xh\rangle^- = \frac{x_{free}^Th_{free_{act}}}{batchsize}$$

but I'm really not sure how I have to compute these terms. So for me $$x^t_{free}$$ would be the samples of $$\mathcal{N}(ΣWh+b;Σ)$$, where $$h$$ here is activation from previous step ($$h_{act}$$) without noise. Lastly $$h_{free_{act}}$$ is computed as the same as the previous $$h_{act}$$ but using $$x^t_{free}$$

Is this correct? Because I don't know when I have to use samples or activations or maybe meanfields