19
$\begingroup$

When reading this paper there is a line which says "linear classifiers do not share parameters among features and classes." What is the meaning of this statement? Does it mean that linear classifiers such as logistic regression need features that are mutually independent?

$\endgroup$
23
$\begingroup$

I will try to answer this question through logistic regression, one of the simplest linear classifiers.

The simplest case of logistic regression is if we have a binary classification task ($y \in\{0,1\})$and only one input feature ($x \in R$). In this case the output of logistic regression would be:

$$ \hat y = σ(w \cdot x + b) $$ where $w$ and $b$ are both scalars. The output of the model $\hat y \in [0,1]$ corresponds to the probability that $x$ will be of class $1$.

We'll try to break down the phrase "linear classifiers do not share parameters among features and classes" into two parts. We will examine the cases of multiple features and multiple classes separately to see if logistic regression shares parameters for any those tasks:

Do linear classifiers share parameters among features?

In this case, for each example, $y$ is a scalar that takes binary values (like before), while $x$ is a vector of length $N$ (where $N$ is the number of features). Here, the the output is a linear combination of the input features (i.e. a weighted sum of these features plus the biases).

$$ \hat y = σ \left(\sum_i^N{(w_i \cdot x_i)} + b\right) \;\; or \;\; σ( \mathbf w \cdot \mathbf x + b) $$ where $ \mathbf x $ and $ \mathbf w $ are vectors of length $N$. The product $\mathbf x \cdot \mathbf w$ produces a scalar. As you can see from above there is a separate weight $w_i$ for each input feature $x_i$ and these weights are independent by all means. From this we can conclude that there is no parameter sharing among features.

Do linear classifiers share parameters among classes?

In this case $x$ is a scalar, however $y$ is a vector of length $M$ (where $M$ is the number of classes). To tackle this, logistic regression essentially produces a separate output $y_j$ for each of the $M$ classes. Each output is a scalar $y_j \in [0,1]$ and corresponds to the probability of $x$ belonging to class $j$.

$$ \mathbf{ \hat y} = w \cdot \mathbf x + \mathbf b, \;\; where \;\; \mathbf{ \hat y} = {\hat y_1, \hat y_2, ..., y_M} $$

The easiest way to think of this is as $M$ simple independent logistic regressions each with an output of:

$$ \hat y_j = σ(w_j \cdot x + b_j) $$

From the above it is obvious that no weights are shared among the different classes.

multi-feature and multi-class:

By combining the two cases above we can finally reach the most general case of multiple features and multiple classes:

$$ \mathbf{ \hat y} = σ( \mathbf W \cdot \mathbf x + \mathbf b) $$ where $\mathbf{ \hat y}$ is a vector with a size of $M$, $\mathbf x$ is a vector with a size of $N$, $\mathbf b$ is a vector with a size of $M$ and $W$ is a matrix with a size of $(N \times M)$.

In any case, linear classifiers do not share any parameters among features or classes.

To answer your second question, linear classifiers do have an underlying assumption that features need to be independent, however this is not what the author of the paper intended to say.

$\endgroup$
1

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.