4
$\begingroup$

Let's say I start with a standard conv net architecture capable of 99% accuracy on MNIST such as this one, but let's say I merge in an "easy" feature to the fully-connecter layer such as a vector of length 10 that encodes the correct output digit 95% of the time and a random digit otherwise.

  1. Will my network ever reach 99% accuracy
  2. Will it take longer to get there?

I think the answer the question 1 is yes, because we will should always be able to find a downhill in the error before the conv net path has reached its full potential. In fact I think it may surpass the accuracy of the original architecture since we are leaking information about the correct output labels.

However I am not so sure about question 2. Does this make the shape of error function unfavorable in any way? I can't decide whether reaching the same accuracy of the original architecture will be faster, slower, or exactly the same.

Hopefully there is an easy answer, otherwise I will run an experiment and report back!

$\endgroup$
  • $\begingroup$ Why don't you go ahead and do the experiment? $\endgroup$ – Valentas Dec 13 '17 at 10:20
  • 1
    $\begingroup$ Because the neural network in my question isn't the only thing that is lazy! $\endgroup$ – Imran Dec 13 '17 at 11:13
  • $\begingroup$ I will if this doesn't get an answer in a day or so. $\endgroup$ – Imran Dec 13 '17 at 11:13
2
$\begingroup$

Interesting theoretical question. While I cannot answer this with 100% certainty, my own intuition and experience say:

1) I would be very surprised if this is not the case. If we look at different problems, where we have one very important feature that holds most of the information but some other features with some signal in there the model generally still improves by adding these lower value features. That is exactly what you are doing here, you have the MNIST pixels that individually hold little value but combined tell you quite a bit. The value of these pixels goes down quite a bit by adding such a strong feature, but the value is absolutely not reduced to 0. I think in theory this should lead to a strictly better model.

2) I think it will take around the same time to converge on average. Fitting to the one-hot encoded feature will be very fast which means most of the time will be spent on fitting to the features, which is a similar problem as without the feature. However, instead of going from 10% to 99% accuracy, we are now kind of going from 95% accuracy to 99.7% accuracy.

I'm less certain about the second question but I think this is what you would find if you do the (interesting) experiment. Let us know if you do implement it!

| improve this answer | |
$\endgroup$
2
$\begingroup$

This should also depend on the network architecture:

  • If the cost-function causes a high penalty for the remaining 5%, then they should be learned as well.
  • If the architecture uses drop-out, then this will force the NN to not rely on the one easy feature.
| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.