I have the about 43 different categories of traffic signs images data. If I am using the small data of 3 categories the maximum accuracy I am getting is around 65% and I have tried a lot of different layer values as well as losses and optimizers. When I am using the complete dataset I am only getting accuracy of about 5%. Please give some pointers on how to improve my accuracy. Please tell me should I add more layers and if somebody has any good example somewhere, please provide the link.

import cv2
import numpy as np
import os
import sys
import tensorflow as tf
import glob

from sklearn.model_selection import train_test_split

TEST_SIZE = 0.4     
path = []
data = []
label = []

def main():

    # Check command-line arguments
    if len(sys.argv) not in [2, 3]:
        sys.exit("Usage: python traffic.py data_directory [model.h5]")

    # Get image arrays and labels for all image files
    images, labels = load_data(sys.argv[1])

    # Split data into training and testing sets
    labels = tf.keras.utils.to_categorical(labels)
    x_train, x_test, y_train, y_test = train_test_split(
        np.array(images), np.array(labels), test_size=0.2
    # Get a compiled neural network
    model = get_model()

    # Fit model on training data
    model.fit(x_train, y_train, epochs=EPOCHS)

    # Evaluate neural network performance
    model.evaluate(x_test,  y_test, verbose=2)

    # Save model to file
    if len(sys.argv) == 3:
        filename = sys.argv[2]
        print(f"Model saved to {filename}.")

def load_data(data_dir):
    Load image data from directory `data_dir`.

    Assume `data_dir` has one directory named after each category, numbered
    0 through NUM_CATEGORIES - 1. Inside each category directory will be some
    number of image files.

    Return tuple `(images, labels)`. `images` should be a list of all
    of the images in the data directory, where each image is formatted as a
    numpy ndarray with dimensions IMG_WIDTH x IMG_HEIGHT x 3. `labels` should
    be a list of integer labels, representing the categories for each of the
    corresponding `images`.
    subdirs = [x[0] for x in os.walk('C:\\Users\\LENOVO\\traffic\\gtsrb')]
    for i in range(1,NUM_CATEGORIES+1):
        subdirs[i] = subdirs[i] + "\*.*"
    for i in range(1,NUM_CATEGORIES+1):
        for file in glob.glob(subdirs[i]):
            a = cv2.imread(file)
            a = cv2.resize(a,(IMG_WIDTH,IMG_HEIGHT))


def get_model():
    Returns a compiled convolutional neural network model. Assume that the
    `input_shape` of the first layer is `(IMG_WIDTH, IMG_HEIGHT, 3)`.
    The output layer should have `NUM_CATEGORIES` units, one for each category.
    model = tf.keras.models.Sequential([
        32, (3, 3), activation="relu", input_shape=(30,30,3)
    tf.keras.layers.MaxPooling2D(pool_size=(2, 2)),
    tf.keras.layers.Dense(128, activation='relu'),

    # Add an output layer with output units for all 10 digits
    tf.keras.layers.Dense(NUM_CATEGORIES, activation="softmax")])
    return (model)
    raise NotImplementedError

if __name__ == "__main__":
  • $\begingroup$ Your model currently consists of only 1 convolutional layer, so you should definitely try increasing the depth of your model. $\endgroup$
    – Oxbowerce
    Commented May 2, 2020 at 13:04
  • $\begingroup$ Yes, I have added 2 more convolutional layers but I cant seem to increase the accuracy. Please tell what should I do. $\endgroup$ Commented May 2, 2020 at 14:02
  • $\begingroup$ That is still quite a shallow model, look at models such a VGG where they use 16 (or 19) blocks consisting of 2 convolutional layers and a max pooling layer (i.e. 32 convolutional layers). $\endgroup$
    – Oxbowerce
    Commented May 2, 2020 at 17:49

2 Answers 2


Adding more images in the training set is one way to increase accuracy. You can also do transfer learning, i.e. using layers trained by a larger dataset if you are classifying common things such as animals which have networks that have already been trained such as the ResNet50.

When you have enough images that the accuracy does not increase anymore, you can try adding more layers, with convolutional and ReLu layers alternating. If you train for many epochs and the model starts to overfit i.e. training set accuracy still trends up but test set accuracy trends down, you can add layers that reduce overfitting, such as the dropout layer. You should first get the model to overfit and then try reducing overfitting.

If you don't have enough training data you could also do augmentation, e.g. adding rotated, shifted and horizontally flipped images to the training set.

The thing with traffic signs is that if you are classifying images with also the landscape instead of just the cropped traffic signs, you are going to have problems. You should then also detect the traffic signs (YOLO v3 could be used as transfer learning) instead of just classifying them.

Here's also a good example of traffic sign classification using transfer learning on Kaggle: https://www.kaggle.com/valentynsichkar/traffic-signs-classification-with-cnn

  • $\begingroup$ Yes I think I needed more data because I downloaded the same dataset from kaggle and got like 2000 images per category for almost all categories. In my original dataset I had fever images per category and sometimes there was a big difference between number of images in certaon categories. Also I used the LeNet architecture and I got better accuracy. $\endgroup$ Commented May 3, 2020 at 12:40

In order to improve the accuracy, you first have to understand the current issues. Without having looked into the dataset, I assume the following problems occur:

1) Your neural network is too simple in its structure.

2) Number of images is rather small

3) Number of images per classes is not balanced

Once these issues are well-understood, you have a chance of improving the accuracy.

Let's discuss these issues in detail.

Issue 1) (Your neural network is too simple in its structure):

Your defined network is a feedforward CNN. It applies 32 different convolutions, followed by the relu function to the input image. As a result you obtain 32 so-called feature-maps. The interpretation is that each feature map is attracted to some image feature which is necessary to perform the classification task. For example one such convolution could correspond to a Prewitt-Filter, which outputs horizontal or vertical edges present in the image. Another filter map could correspond to detecting round corners, arcs and so on. The final fully connected layer than classifies the image, based on these generated features.

In a deep CNN, it has been observered that the feature maps in the early layers correspond to low-level features (corners, lines, ...), while the feature maps in the later layer are attracted to high-level features (compositions of low-level features, e.g. squares, circles, faces,...) as each convolution is applied to the "image" generated by the previous layer (which is not the input image, but a feature map). Accordingly, it is essential to have multiple layers in order to achieve this hierarchical construction of features, which turned out to be very effective (and is similar to how the human brain works in some areas).

Therefore, one direction to improve accuracy is to to add more layers of convolutions (and poolings). While this turned out to be already sufficient for many task in order to achieve very high accuracies, this is probably not sufficient in your case, due to problem 2)

Issue 2) Number of images is rather small:

Theoretically, If you use a deep CNN, there should be some weights which provide very good accuracies. This probably will not work in practice in your case mainly due to:

  • all known optimization methods, which are used to obtain (train) the weights of the CNN, are not delivering a global optimal solution.

  • even in the case we had the global optimal solution and even, if we had 100% accuracy on the train set, it does not imply the CNN performs good on unseen images.

It is verly likely that you would obtain weights, which lead to bad test accuracies, even if you obtain the correct classifications on the training set. Mainly, the reason is that very likely, the learned weights do not correspond to plausible "explanations" for the general task (classification if traffic signs) but are some "shortcuts".

Example: Let's assume all images of the stop sign have been taken using camera A and all images of a "keep right" sign have been taken using camera B. Now if the sensor if camera A has some defect, e.g. the center pixel is always black, optimizing the weights will thus likely lead to the classification of stop sign by looking at the center pixel. However,once you have an image of a stop sign in the test set taken from another camera this will lead to a misclassification.

In general, if you are using a deep CNN, there are many variables that need to be optimized from the training data (among others, the kernel weights of the convolutions). In order to avoid classifications based on unplausible reasonings, it is crucial to have many training images such that no "shortcuts" are possible. In the example, it would be crucial to take images from different cameras such that hopefully the "explanations" or the features learned by the CNN are not some special characteristics of the training data but focus on a plausible model (traffic sign is an octagon with red background and text..).

In case that you do not have enough training data, one option is to use transfer learning, that means we use a pre-trained CNN model (trained on another task, but with a hugh dataset), such that the features in the early layers are attractive to low-level features. Then you use the training only the adapt the high-level features to your specific classification task. This is less prone to short-cuts as there are already meaningful features in the early layers with are feed forward in the network.

Another option is to use another ML model, e.g. Random Forest, which does not need so many training samples as the features are not learned. In this case, features have to be defined or learned in another way.

Issue 3) There are several ways to handle class imbalances for Random Forest and Neural network. Most of them affect how you sample your data.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.