# Neural network accuracy for simple classification

I am trying to develop a NN for a very simple classification model with keras/tensorflow:

Ex:

• input: "Do" => class output: "Dog"
• input: "Ca" => class output: "Cat"
• input: "Mo" => class output: "Mouse"

I train the model with many "Do", "Ca", etc. (as dictionary), indexing the input to categorical arrays ([0,1,0] for example is "Cat", [1,0,0] is "Dog", [0,0,1] is "Mouse" in case of three classes).

I know that for this type of problem every other traditional classification algorithm should be used, and not DL, but I'm doing it with a purpose (I need to merge it with an image classifier).

My question is: since the classification is easy ("Do" is always "Dog", "Ca" is always "Cat") a Decision tree would have an accuracy of 100% always. Why do Neural Networks need tons of data and epochs to get an acceptable accuracy? With 10 classes, it takes 10 epochs and 8 thousand entries in order to get an accuracy higher than 90%. I'm using sparse categorical crossentropy as loss, and SGD as the optimizer. (2 Dense layer, relu - softmax). Also, I am a bit lost on how to choose the number of neurons, I guess trial and error is the way.

• Ok to be clear, the inputs can either be "Ca" or "Do" right? Nothing else? If so then why don't you vectorize that as [1,0] and [0,1] respectively. Then the output is either dog or cat. So that should vectorized the same way. Why do you have 3 output nodes? – JahKnows Mar 27 '18 at 9:09
• The model I'm working on has 10 classes, and 10 different kinds of input (every input belongs to one category). The actual case would be [1,0,0,0,0,0,0,0,0,0]. I edit the post to avoid confusion, the example for this post would be three inputs. – maurocomi Mar 27 '18 at 9:17
• Ok so ten possible inputs and 10 possible outputs right? Based on the little words you have at the input you want to predict an associated label? – JahKnows Mar 27 '18 at 9:19
• Exactly. Any traditional Ml classifier would predict with a 100% accuracy, because "Do" is always "Dog". Deep Learning work differently, and choosing the right number of neurons/hidden layers/epochs is not straightforward. – maurocomi Mar 27 '18 at 9:21
• Why do you need to use a neural network? I think it is important to pick the right tool for the job at hand. – JahKnows Mar 27 '18 at 9:22

I will add my 2 cents at the end of this answer. However, this is how it can be done using a neural network. Firstly, yes, you should expect to need more data to train even a simple neural network because their are more parameters that need tuning. Think of them like little faucets that you need to tune in order to get the right output volume based on an input. If you have millions of these faucets you an imagine that this is an arduous process.

You will need some of the following imports

from __future__ import print_function
import keras
from keras.datasets import mnist
from keras.models import Sequential
from keras.layers import Dense, Dropout, Flatten
from keras.layers import Conv2D, MaxPooling2D
from keras.callbacks import ModelCheckpoint
from keras.models import model_from_json
from keras import backend as K


But, in your case you know what your data should be so you can simulate it. I do this as follows and make a training and testing set.

import numpy as np

n = 10
m = 2

x_train = np.zeros((n, m))
y_train = np.zeros((n,))
for i in range(n):
label = np.random.randint(0,m)
y_train[i] = label
x_train[i, label] = 1

x_test = np.zeros((n//3, m))
y_test = np.zeros((n//3,))
for i in range(n//3):
label = np.random.randint(0,m)
y_test[i] = label
x_test[i, label] = 1


Now we will have a training set which contains $n$ instances and a testing set with a third as many. $m$ is the number of possible inputs. For cat vs. dog this would be $m=2$. You will be using your more general case where $m=10$. Each entry in the matrix $x$ has the vector with one-hot encoded vector where the index in accordance with the label is 1.

We need to reshape the data for it to fit with the Keras structure.

# The known number of output classes.
num_classes = m

# Channels go last for TensorFlow backend
x_train_reshaped = x_train.reshape(x_train.shape[0], m,)
x_test_reshaped = x_test.reshape(x_test.shape[0], m,)
input_shape = (m,)

# Convert class vectors to binary class matrices. This uses 1 hot encoding.
y_train_binary = keras.utils.to_categorical(y_train, num_classes)
y_test_binary = keras.utils.to_categorical(y_test, num_classes)


We then build our model

model = Sequential()

model.compile(loss=keras.losses.categorical_crossentropy,
metrics=['accuracy'])


We then train our model

epochs = 4
batch_size = 128
# Fit the model weights.
history = model.fit(x_train_reshaped, y_train_binary,
batch_size=batch_size,
epochs=epochs,
verbose=1,
validation_data=(x_test_reshaped, y_test_binary))


Epoch 4/4
10/10 [==============================] - 0s 251us/step - loss: 0.6247 - acc: 0.6000 - val_loss: 0.5311 - val_acc: 1.0000

Voila, now you have perfect classification with this network. You can play around with the model and see a summary of the model using.

model.summary()


# For $m = 10$

Due to the higher complexity that this set has you will need to increase the number of instances in your training set. I will also 2 layers to our model and make them less wide. Furthermore, we will add more epochs so we will train longer.

import numpy as np

n = 1000
m = 10

x_train = np.zeros((n, m))
y_train = np.zeros((n,))
for i in range(n):
label = np.random.randint(0,m)
y_train[i] = label
x_train[i, label] = 1

x_test = np.zeros((n//3, m))
y_test = np.zeros((n//3,))
for i in range(n//3):
label = np.random.randint(0,m)
y_test[i] = label
x_test[i, label] = 1

# The known number of output classes.
num_classes = m

# Channels go last for TensorFlow backend
x_train_reshaped = x_train.reshape(x_train.shape[0], m,)
x_test_reshaped = x_test.reshape(x_test.shape[0], m,)

input_shape = (m,)

# Convert class vectors to binary class matrices. This uses 1 hot encoding.
y_train_binary = keras.utils.to_categorical(y_train, num_classes)
y_test_binary = keras.utils.to_categorical(y_test, num_classes)

model = Sequential()

model.compile(loss=keras.losses.categorical_crossentropy,
metrics=['accuracy'])

epochs = 10
batch_size = 128
# Fit the model weights.
history = model.fit(x_train_reshaped, y_train_binary,
batch_size=batch_size,
epochs=epochs,
verbose=1,
validation_data=(x_test_reshaped, y_test_binary))


Epoch 10/10
1000/1000 [==============================] - 0s 49us/step - loss: 1.5977 - acc: 1.0000 - val_loss: 1.5235 - val_acc: 1.0000

# My suggestions

I would not use a NN for such a case. Most of the frameworks allow you to add information throughout your model. Such that if you have images you can run a CNN over them, then when you are ready to convert your layers to a densely connected layer you can in additional information, such as vectorized text.

You can thus use a random forests approach or something even simpler to get your 100% classification even faster. Then you can feed the output of this model to your deep learning framework which has already "extracted the features" from the images and concatenate these additional features to that tensor. Then you will pass this larger tensor through the subsequent Dense layers to get your final output.

• Thank you so much! (I cannot upvote unfortunately). The one I did is slighty different, I see that using Adadelta instead of SGD boosts the accuracy to almost 99%. Also, is there a reason why you use 64 as the output for the first layer? – maurocomi Mar 27 '18 at 10:08
• My choice of number of neurons is kind of random. You just want to make sure it's wide enough to accommodate enough combinations of the input features such that information is properly utilized. For this case it can probably be lowered significantly. I like to make them 2^ though out of habit. I heard once that it was faster, I never looked into that claim, I just like nice looking numbers. – JahKnows Mar 27 '18 at 10:12
• The more neurons per layer the higher the complexity, thus you will have to train with more data or for a longer time, or both. So you want to keep it as constrained as possible while not making it too small that you are losing potential information. – JahKnows Mar 27 '18 at 10:13
• Now it's clear, thanks. So you are suggesting to create a CNN model, and before the fully connected layers 8so after MaxPooling), give an input which will be the output of a random forest? Do I understand that right? This was my idea originally, but I don't know if that can work – maurocomi Mar 27 '18 at 10:50

For every problem there is proportional solution.

I understand that you want to build your classifier inside an image recognition algorithm.

You may need a NN for the image (lots of inputs + lots of possibilities) but not for the whole system.

What is usually done for this kind of problematic is to build a "pipe". A sequence of ML algorithms taking as input the output of the previous one:

• First block is a text zone detection within the image
• Second block read the zones into text
• Third block do the actual classification.

This method also make the debugging process easier, as you can evaluate your prediction rate at every step making easy to know which part can be optimized/fixed if a problem is encountered.

Hope this helps.