0
$\begingroup$

EDIT: Found one mistake, confused rows and columns in the step of summing over column operations.

I'm trying to prove the Ridge regression duality of Dropout, as described in section 9.1 of this paper. Check the image below for the section. I've taken the liberty to rewrite the distribution of $R$ as

$$R \sim Bernoulli(\{0^N, 1^N\}\vert p)^D$$

to accentuate that $R$ is a $N\times D$ matrix where each column is either entirely 0 or 1, with probabilities $1-p$ and $p$. The notation of the paper suggests that each element of $R$ is distributed independently, but this not how I understand dropout. Please correct me if I'm wrong.

$$\textrm{minimize}_{\textbf{w}}\mathbb{E}_{R \sim Bernoulli(\{0^N, 1^N\}\vert p)^D} \Vert \mathbb{y} - R*X\cdot \mathbb{w}\Vert^2$$

We can rewrite the dot product between $X$ and $\mathbb{w}$ as the sum over all the columns:

$$=\textrm{minimize}_{\textbf{w}}\sum_{c=1}^D\mathbb{E}_{r \sim Bernoulli(\{0, 1\}\vert p)} \Vert \mathbb{y} - r*X_c* \mathbb{w}_c\Vert^2$$

Expanding the expectation value to the weighted sum of the options:

$$=\textrm{minimize}_{\textbf{w}}\sum_{c=1}^D (1-p) \Vert \mathbb{y}\Vert^2 + p\Vert \mathbb{y} - X_c * \mathbb{w}_c\Vert^2$$

This is where I get stuck. I can see how $\Vert X_c\cdot \mathbb{w}\Vert^2$ would create the $\Vert \Gamma \cdot \mathbb{w}\Vert^2$. But the $(1-p)$ seems awkward here.

Question: How can one prove the equivalence with

$$\textrm{minimize}_{\textbf{w}}\Vert \mathbb{y} - p*X\cdot \mathbb{w}\Vert^2 + p(1-p)\Vert \Gamma \mathbb{w}\Vert^2$$

?

Section from the paper:

Dropout Ridge regression duality paper

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.