I am currently working on a classification problem:

The dataset (2000 features, 25 labels) I am using is seperable by using a 2-layered Multi-Layer-Perceptron (1 Input + 1 OutputLayer = 1 Weight Matrix) with achieving great accuracy.

Note: The Data I am using are spectrograms.

Now what I am curious about is the following: Can I use a deep network (more than 2 layers) to even increase the accuracy?

What I think about: Increasing the network structure would make the network interpret the data in a more complex way and therefore should be able to even correctly classify little "outbrakers" and "noisy elements" of the dataset.


If you are using two hidden layers, it may mean that your data is not linearly separable. If you just use one unit in a single hidden layer then you can claim that your data is linearly separable, which in your case you may say that there exist a hyper-plane that separates your data linearly.

As the answer to your question, yes. It is possible to reduce the amount of error by adding layers or adding extra units to the current architecture but there are some points that are necessary to be considered.

  • Whenever you add more layers, there will be vanishing and exploding gradients which may cause your network not to learn, or learning may happen so slowly. To avoid, you should use ReLU activation in order to avoid saturation of gradients. Moreover you have to use He or Xavier initialization techniques to avoid having bad random weights. There are other techniques for solving this problem which are called skip connections but at least I've never seen the use of them in MLPs Although they are really helpful for solving the mentioned problem.
  • Covariat shift is the problem of learning for deeper layers. As a solution you have to use Batch Normalization to somehow normalize the activations of the deeper layers.
  • Overfitting is a problem that happens for large architectures that are not fed with enough training data. You should use regularization techniques. There are so many papers about this but in your case which is using MLP I highly recommend you using Dropout technique which is invented by the so called God Father of deep learning.
  • 1
    $\begingroup$ Good answer, but there are few problems with your first paragraph. Whether or not the data is linearly separable is independent of what network architecture you choose to use. It only affects whether you'll be able to achieve 100% training accuracy. $\endgroup$
    – Imran
    Dec 18 '17 at 0:31
  • $\begingroup$ @Imran between machine-learning practitioners, it is common and usual to have learning scenarios. whenever the input features of the problems are too large because it is hard to imagine the shape of decision boundary, they start with a simple architecture and step by step they try to make it complex. suppose that your problem is to classify data of AND operation. In this case you know that you have to start with a linear model and if it does not work you make it complex. reason you know to use linear model is you see the data and is exactly dependent to your data which is separable linearly. $\endgroup$ Dec 18 '17 at 3:32
  • $\begingroup$ Sure, but @Luftbaum says "The dataset I am using is linear seperable". This is an inherent property of the data, and training a multi-layer network on that data cannot possibly change the data itself as you suggest. It's like saying my dog is a cat because he's sleeping in the cat bed. $\endgroup$
    – Imran
    Dec 18 '17 at 3:48
  • $\begingroup$ I guess I figured out what are saying. I have told that because he has not reached 100 percent accuracy. If your data is linearly separable, then it can easily be solved with just a single neuron. Using too layer nets in such cases, will cause high variance problem and surly the model would be overfitted which implies to have 100 percent accuracy. $\endgroup$ Dec 18 '17 at 3:52
  • $\begingroup$ Ok this one is my fault (I still have some trouble with the terminology): I have edited my question therefore. Really Sorry! Maybe someone can explain how you call data that is separable by two layers? $\endgroup$
    – Luftbaum
    Dec 18 '17 at 19:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.