31
votes
Accepted
Strange behavior with Adam optimizer when training for too long
This small instability at the end of convergence is a feature of Adam (and RMSProp) due to how it estimates mean gradient magnitudes over recent steps and divides by them.
One thing Adam does is ...
- 29.2k
12
votes
Accepted
Why must a CNN have a fixed input size?
I think the answer to this question is weight sharing in convolutional layers, which you don't have in fully-connected ones. In convolutional layers you only train the kernel, which is then convolved ...
- 1,555
9
votes
Why must a CNN have a fixed input size?
It's actually not true. CNN's don't have to have a fixed-size input. It is possible to build CNN architectures that can handle variable-length inputs. Most standard CNNs are designed for a fixed-...
- 3,411
6
votes
Perceptron learning rate
I agree with Dawny33, choosing learning rate only scales w.
While training of Perceptron we are trying to determine minima and choosing of learning rate helps us determine how fast we can reach that ...
- 61
6
votes
How does combining two linear perceptrons create non-linear boundaries?
You are correct that stacking two layers with a linear activation function on top of each other does not do anything that a single layer could not do (i.e. it is still a linear combination of terms).
...
- 393
5
votes
What is the difference between Perceptron and ADALINE?
The Adaline (Adaptive Linear Element) and the Perceptron are both linear classifiers when considered as individual units. They both take an input, and based on a threshold, output e.g. either a 0 or a ...
- 15.1k
5
votes
Accepted
Is the percepetron algorithm's convergence dependent on the linearity of the data?
Yes, the perceptron learning algorithm is a linear classifier. If your data is separable by a hyperplane, then the perceptron will always converge. It will never converge if the data is not linearly ...
- 66
5
votes
Accepted
Is the prediction algorithm absolutely the same for all linear classifiers?
Is it true that linear classifiers differ only in the Learning
algorithm, but do they do the same during Prediction y = w1*x1 + w2*x2
+ ... + c?
Yes, all parametric linear classifiers try to ...
- 2,273
5
votes
difference between empirical risk minimization and structural risk minimization?
SVMs were developed by Vapnik (1995,1998), based on the structural risk minimization principle (Vapnik, 1982) from statistical learning theory.
The complexity of the class of functions performing ...
- 4,098
4
votes
How should the hyper parameters be defined for the other algorithms defined for sklearn_crfsuite.CRF
C1 and C2 are coefficients for L1 and L2 regularization, respectively. You can use the same definition of ...
- 141
4
votes
Whether add bias or not in a perceptron
Suppose bias as a threshold. Using threshold, your activation function moves across the $x$ axis which may get complicated. Consequently, people usually use the bias term and always centre the ...
- 14.2k
4
votes
Accepted
Ambiguity in Perceptron loss function (C. Bishop vs F. Rosenblatt)
I managed to find the Bishop's version by unearthing! the Rosenblatt's 1962 Principles of neurodynamics, Page 110 book, so the Wikipedia's version must be the alternative one.
It is worth noting ...
- 9,442
4
votes
Accepted
Conceptual questions on MLP and Perceptrons
When the data is linearly inseparable, we use MLP. Here what is meant by "data"--is it the response or the input feature that is linearly inseparable?
This means that a linear function of the input ...
- 2,782
4
votes
What's wrong with my implementation of the Adaline algorithm?
First of all, let me add this schema which I think is quite nice to understand the transition and improvement from the initial Rosenblatt's perceptron and the Adaline algorithm:
In Adaline, provided ...
- 2,734
3
votes
Learning rate of perception
After some study, I figured out the answer and want to share with people if someone also finds it helpful. The loss function of Perceptron is hinge loss or
$J(w) = max(0, -yw^Tx)$.
Adding a ...
- 49
3
votes
What is the difference between Perceptron and ADALINE?
The differences between the Perceptron and Adaline:
The Perceptron uses the class labels to learn model coefficients.
Adaline uses continuous predicted values (from the net input) to learn the model ...
- 31
3
votes
Accepted
Perceptron weight vector update
The algorithm works by adding or subtracting the feature vector to/from the weight vector. If you only add/subtract parts of the feature vector your a not guaranteed to always nudge the weights in the ...
- 6,452
3
votes
Perceptron learning rate
With regard to the single-layered perceptron (e.g. as described in wikipedia), for every initial weights vector $w_0$ and training rate $\eta>0$, you could instead choose $w_0'=\frac{w_0}{\eta}$ ...
- 674
3
votes
Accepted
How to find bias for perceptron algorithm?
Rather than viewing your data as
X = [[0.8, 0.1], [0.7, 0.2], [0.9, 0.3], [0.3, 0.8], [0.1, 0.7], [0.1, 0.9]]
Y = [-1, -1, -1, 1, 1, 1]
You could have treat the ...
- 3,128
3
votes
Accepted
Multilayer Perceptron: What is the value used to update the weights in the hidden layers?
A Multilayer Perceptron changes your weights by an algorithm called "backpropagation". This algorithm uses gradient descent combined with a learning rate to change every weight in your MLP.
Basically ...
- 672
3
votes
Accepted
Understanding computations of Perceptron and Multi-Layer Perceptrons on Geometric level
The two pictures you show illustrate how to interprete one perceptron and a MLP consisting of 3 layers.
Let us discuss the geometry behind one perceptron first, before explaining the image.
We ...
- 1,044
2
votes
Perceptron learning rate
The choice of learning rate m does not matter because it just changes
the scaling of w.
I agree that it is just the scaling of w which is done by the learning ...
- 8,416
2
votes
Is there a relationship between LDA, linear SVMs and Perceptron?
The brief answers are:
1, 2. No. They depend on different subsets of examples.
3, 4. Yes, but only if it separates classes linearly and you are extremely lucky. Otherwise no.
5, 6. No, because ...
- 1,551
2
votes
Perceptron learning rate
Some of the answers on this page are misleading. In the perceptron algorithm, the weight vector is a linear combination of the examples on which an error was made, and if you have a constant learning ...
- 21
2
votes
Accepted
How to implement gradient descent for a tanh() activation function for a single layer perceptron?
How can I circumvent this problem?
TLDR : Normalize your input data.
Why?
Notice how tanh actually works on the input data:
...
- 454
2
votes
Accepted
Normalizing the final weights vector in the upper bound on the Perceptron's convergence
The Perceptron's output $f$ is
$$
f(\overline\theta \cdot \overline{x}) = \begin{cases} 1 &\text{ if } \overline{\theta }\cdot \overline{x} > 0 \\ 0 &\text{ if } \overline{\theta }\cdot \...
- 1,646
2
votes
Accepted
Why perceptron does not converge on data not linearly separable
The behavior appears to actually depend on the learning rate $\eta$; a smaller $\eta$ affects which points are misclassified in the next iteration, which affects the weight update more than just by ...
- 12.3k
2
votes
number of neurons for mnist dataset using mlp?
If you train long enough and have "too many" hidden layer units, then you will eventually have over-training. Usually, the goal is to find the smallest number of hidden units that can do reasonably ...
- 301
2
votes
Accepted
Perceptron learning rate irrelevant in convergence
Your conclusion is correct.
Note that the classifier is of the form of
$$f(x; \theta) = \operatorname{sign}(\theta^Tx)$$
while the update rule is
$$\theta^{(k+1)}=\theta^{(k)}+\eta y_tx_t$$ ...
- 3,128
2
votes
Learning rate in the Perceptron Proof and Convergence
(My answer is with regard to the well known variant of the single-layered perceptron, very similar to the first version described in wikipedia, except that for convenience, here the classes are $1$ ...
- 674
Only top scored, non community-wiki answers of a minimum length are eligible