Keras model producing same output

Question

I've seen a couple questions that have a similar problem, but none of them solved mine. I'm trying to fit a neural network in Keras to a dataset with 22 input features for binary classification. The problem is that I only have 195 training samples. I know it's a small dataset, but I don't know if it's possible to fit a model with reasonable accuracy (I'm aiming for >95% accuracy). The problem I'm having is that my model is only outputting 1 and getting 75% accuracy because my dataset is 75% positive cases. Here's the code I have:

data = pd.read_csv("") #filename omitted, but it loads properly
scaler = MinMaxScaler()
X = scaler.fit_transform(X)
Y = data['status']
X = data.drop(['status', 'name'], axis = 1)
xTrain, xTest, yTrain, yTest = train_test_split(X, Y, train_size = 0.8)
model = Sequential()
model.add(Dense(48, input_shape=(22,), activation = 'relu'))
model.add(Dropout(0.5))
model.add(Dense(1, activation = 'softmax'))
optim = keras.optimizers.adam(lr=0.0001)
model.compile(optimizer = optim, loss = 'binary_crossentropy', metrics = ['accuracy'])
model.fit(xTrain, yTrain, epochs = 20, batch_size = 5, validation_data = (xTest, yTest))

I've tried adding more hidden layers, increasing the number of training epochs, and increased and lowered the optimizer's learning rate, but the accuracy stays the same. Here's the link to the dataset: https://www.dropbox.com/s/c4td650b4z7aizc/fixed.xlsx?dl=0

EDIT: I fixed my problem by applying SMOTE to balance my dataset. The accuracy went up to 96% after I did this and added more hidden layers.

Answer 1

A reason might be that you are running a single-layer neural net. Ideally, you should have more than one layer, and use the sigmoid activation function.

Answer 2

It happens often, that some model only predicts one class. The reason usually is, that the model is unable to distinguish the two classes well, and resorts to one (often the majority) class.

With your data, you may have a hard time to fit a NN with reasonable results. I suggest you check boosting which usually works okay with small data. Make sure you tune parameters well.

Here is an application of lightgbm to the iris data.

You could also check Logit with L1 regularization.

Answer 3

Probably because you are using a single hidden layer which is unable to learn that many good parameters to differentiate between the classes.

try adding 1-2 more Dense layers.

you can try out this configuration as a baseline

model.add(Dense(128, input_shape=(22,), activation='relu'))

model.add(Dense(64, input_shape=(22,), activation='relu'))

at the output layer sigmoid will be fine because it is a binary classification.

model.add(Dense(1, activation = 'sigmoid')

Share your results so that we can explore more on this.

Answer 4

Pin pointing exact reason would be impossible without looking at the data. Few things you can try:

1. Add more layers, neurons in layers to make the model more expressive. Other has pointed this out already.
2. As you have only 195 data points, try to see if there is a scope to use transfer learning.
3. Look at the data, if imbalanced you can do following:
   • Upsampling/Downsampling
   • Use weighted cost, misclassification of less represented class will be penalised heavily compared to the other class.
   • Be careful with your error metric. Accuracy may not be the right choice.
4. Play with the learning rate.

Answer 5

You are using a softmax function with only one neuron in your last layer.

It is normal that your output is the same, one property of softmax function is that the sum of output across neurons equals 1 (1 neuron = 1 always).

Use a sigmoid function or set your last layer to 2 neuron.