I am reading this blog from Sebastian Ruder blog-link, researcher scientist at Deepmind, and having problems understanding this pseudocode for Democratic Co-learning. Can somebody help me understand what is going on; specifically, I'm just trying to understand line 1-6.

Description of Democratic Co-learning from blog:

Democratic Co-learning Rather than treating different feature sets as views, democratic co-learning (Zhou and Goldman, 2004) [18] employs models with different inductive biases. These can be different network architectures in the case of neural networks or completely different learning algorithms. Democratic co-learning first trains each model separately on the complete labelled data L. The models then make predictions on the unlabelled data U. If a majority of models confidently agree on the label of an example, the example is added to the labelled dataset. Confidence is measured in the original formulation by measuring if the sum of the mean confidence intervals w of the models, which agreed on the label is larger than the sum of the models that disagreed. This process is repeated until no more examples are added. The final prediction is made with a majority vote weighted with the confidence intervals of the models. The full algorithm can be seen below. M is the set of all models that predict the same label j for an example x.

pseudocode for co-learning

I guess line 1-3 is the training of several models on labeled data. Is i each sample in the labeled dataset? Is mi a trained model using all the samples from the labeled dataset? I guess that line 4-6 is going through each sample in the unlabeled dataset but what's j? Is it iterating through each label?

This line I'm especially confused about is M <- {i | pi(x) = j}. Is i being assigned the output of the model from the unlabeled sample and then being assigned to M and what is M?


Let's try to make some sense of what is going on here:

In the first 3 lines as you say, we build $n$ models, being $m$ a single model.

The second loop starts iterating over the unlabelled sample set $U$. For each unlabelled sample $x$, we iterate over the possible labels (e.g $\{0,1\}$ for binary classification, or $dog\space vs.\space cat$, etc...) from the set $C$ and we call it $j$.

$M$ thus is the set of models $m$, that predict that sample $x$ is of the class $j$

After that, if the number of models $|M|$ (where $0\leq |M|\leq n$) that agree that label is $j$ are more than half of the models available ($|M|>n/2$) and the weighted mean ($w$ is the confidence intervals of the models) of the models that agree on this prediction is higher than the models that disagree (line 7) → then the sample $x$ is included in $L$ with label $j$ (line 8).

| improve this answer | |
  • $\begingroup$ Ah I see. I'm almost with you on everything but still a little bit confused with the notation of line 6. Is pi(x) the probability that it belongs to class j? Also, shouldn't we iterate through through {1,n} again right before line 6? $\endgroup$ – zipline86 Feb 26 at 13:24
  • 1
    $\begingroup$ Yes, I think that $i|p_{i}(x) = j$ meas the model choice $i$ given the model outputted prob $p_{i}(x)$ matches the current label. About the iteration missing, I also share your concerns. I feel that indeed there should be an iteration over the trained models to predict over $x$ and generate $M$. I have been looking into the original paper and haven't found that particular preudo-code so I am not sure about how reliable it is. In any case, I think you fully got the idea behind it ;) $\endgroup$ – TitoOrt Feb 26 at 13:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.