5
$\begingroup$

I have created a neural network to classify the MNIST handwritten numbers dataset. It is using softmax as the activation function for the output layer and various other functions for the hidden layer.

My implementation with the help of this question seems to be passing the gradient checks for all activation functions but when it comes to the actual run with my training data for an exemplary run of 10 iterations I get an accuracy of about 87% if I use sigmoid or tanh as the activation function for the hidden layer, but if I use cosine it returns an accuracy of 9%. Training the network with more iterations (100, 200, 500) does not have any effect either and in fact my minimization function does not manage to move below 2.18xxx for the cost function no matter how many epochs pass.

Is there some pre-processing step that I need to perform before using cosine if not why is it that this activation function works so badly?

Improve this question
$\endgroup$
8
  • 2
    $\begingroup$ Compare them after convergence, not a fixed number of iterations. $\endgroup$
    – Emre
    May 16, 2017 at 22:16
  • $\begingroup$ @Emre I edited the question to show that more test were done, but in general I suspect this is some issue I cannot yet imagine. $\endgroup$
    – ealiaj
    May 17, 2017 at 5:30
  • 3
    $\begingroup$ Although sigmoid or tanh is well known for slower convergence, it shouldn't be that bad. I suspect you have a bug. Unfortunately, I don't see a way for us to answer it. $\endgroup$
    – SmallChess
    May 17, 2017 at 6:04
  • $\begingroup$ From your linked question, have you changed the line sigma2 = (sigma3*Theta2).*activationGradient([ones(m, 1) z2], 'sigmoid'); to match the activation function you are actually using for each test? If you have started to run lots of tests and switching between versions of your script, worth reviewing that, make sure you are running the code that you think you are $\endgroup$
    – Neil Slater
    May 17, 2017 at 7:07
  • $\begingroup$ @NeilSlater yes, in fact I use a variable for it so it computes the correct derivative each time. $\endgroup$
    – ealiaj
    May 17, 2017 at 11:10

1 Answer 1

Reset to default
5
$\begingroup$

Cosine is not a commonly used activation function.

Looking at the Wikipedia page describing common activation functions, it is not listed.

And one of the desirable properties of activation functions described on that page is:

Approximates identity near the origin: When activation functions have this property, the neural network will learn efficiently when its weights are initialized with small random values. When the activation function does not approximate identity near the origin, special care must be used when initializing the weights.

$cos(0) = 1$, a basic cosine function does not have this property. Combined with its periodic nature, this makes it look like it could be particularly tricky to get correct starting conditions and other hyper-parameters in order to have a network learn whilst using it.

In addition, cosine is not monotonic, which means that error surface is likely to be more complex than for e.g. sigmoid.

I suggest trying with a low learning rate, and initialising all the bias values to $-\frac{\pi}{2}$. Maybe reduce the variance in initial weights a little too, just to start off with things close to zero. Essentially this is starting with $sin()$. Caveat: not that I have tried this myself, just an educated guess, so I would be interested to know if that helps at all with stability.

Improve this answer
$\endgroup$
4
  • $\begingroup$ I did saw this particular activation function from this answer, your suggestions are indeed interesting though I will try them and report back. $\endgroup$
    – ealiaj
    May 17, 2017 at 18:36
  • 3
    $\begingroup$ If you follow the link from the answer, it seems like cosine activations are not used along with gradient descent. The title of the paper is "Replacing minimization with randomization in learning" and does not once use the word gradient (nor back propagation) $\endgroup$
    – Neil Slater
    May 17, 2017 at 18:37
  • $\begingroup$ I found this answer after looking up cosine activation as the authors of the book Deep Learning (Goodfellow, et al.) made the claim: "many unpublished activation functions perform just as well as the popular ones. To provide a concrete example, the authors tested a feedforward network using h = cos(W x + b) on the MNIST dataset and obtained an error rate of less than 1%, which is competitive with results obtained using more conventional activation functions." Does anyone know what the authors of that book did differently to achieve good performance with cosine? $\endgroup$
    – David Etler
    Oct 11, 2020 at 18:16
  • 2
    $\begingroup$ @DavidEtler You may be better off asking a new question as opposed to a comment here. I make a suggestion at the end of this answer, initialise b to $-\frac{\pi}{2}$. I do not know if the authors did this, or just found it worked ok without that. Probably depends on the problem being solved $\endgroup$
    – Neil Slater
    Oct 11, 2020 at 18:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.