How to choose architecture of neural network for concrete task?

Question

How to systematically choose the architecture of a neural network (NN) for a concrete task?

For example, I am solving classification task with 3 classes (NN should recognize pandas, dogs and cats). What type of architecture I should choose? Is there a general rule of thumb?

So far, I have simply picked an architecture, which has 90% accuracy on CIFAR-10 dataset, and been trying to apply it to my task. Results are not very good (on 30 epoch, I got only about 40% accuracy on train set and 57% on val set).

Answer 1

The architecture of any neural network can be thought of as a hyperparameter. It can be tricky to choose the 'right' number of layers and the 'right' number of neurons that works for you. As suggested by Shubham, you should go through the implementation of the other researchers working on a similar problem. It will give you a good starting point. Additionally, there are libraries (grid-search based of course!) to tune some of the NN architectures like KerasTuner, AutoKeras, etc which can help you. Another good starting point is to start with a simple architecture and keep adding the layers and neurons until you encounter overfitting. However, the final architecture should also make 'sense'. Note that the architecture of NN depends on the quality of features you feed the model. If your features are 'good', you may not need a complicated architecture to get a 'good' generalization. Pretrained models and transformers as feature extractors can help you reduce the complexity of the architecture.

Answer 2

There's no thumb rule for designing a NN architecture. Every architecture is designed to perform well on specific problems, so you need to choose one for yourself, which best suites your problem.

You may consider these points while implementing a NN architecture for your problem,

• Check if other researchers have worked on similar problems. If yes, review their approach and the NN architecture they designed or used ( with some modifications ).

• For image classification, pretrained models like VGG, Inception, ResNet, MobileNets etc. are available which are trained on huge datasets like ImageNet. These models are used a "backbone" model for other problems, for better feature extraction and also to reduce the no. of trainable parameters. Consider using these models as a whole or use some layers only.