2
$\begingroup$

I've seen a couple questions that have a similar problem, but none of them solved mine. I'm trying to fit a neural network in Keras to a dataset with 22 input features for binary classification. The problem is that I only have 195 training samples. I know it's a small dataset, but I don't know if it's possible to fit a model with reasonable accuracy (I'm aiming for >95% accuracy). The problem I'm having is that my model is only outputting 1 and getting 75% accuracy because my dataset is 75% positive cases. Here's the code I have:

data = pd.read_csv("") #filename omitted, but it loads properly
scaler = MinMaxScaler()
X = scaler.fit_transform(X)
Y = data['status']
X = data.drop(['status', 'name'], axis = 1)
xTrain, xTest, yTrain, yTest = train_test_split(X, Y, train_size = 0.8)
model = Sequential()
model.add(Dense(48, input_shape=(22,), activation = 'relu'))
model.add(Dropout(0.5))
model.add(Dense(1, activation = 'softmax'))
optim = keras.optimizers.adam(lr=0.0001)
model.compile(optimizer = optim, loss = 'binary_crossentropy', metrics = ['accuracy'])
model.fit(xTrain, yTrain, epochs = 20, batch_size = 5, validation_data = (xTest, yTest))

I've tried adding more hidden layers, increasing the number of training epochs, and increased and lowered the optimizer's learning rate, but the accuracy stays the same. Here's the link to the dataset: https://www.dropbox.com/s/c4td650b4z7aizc/fixed.xlsx?dl=0

EDIT: I fixed my problem by applying SMOTE to balance my dataset. The accuracy went up to 96% after I did this and added more hidden layers.

$\endgroup$
  • $\begingroup$ try setting activation='tanh' instead of 'relu'. There's a problem called "dying relu" in which weights using that activation function can get "stuck" on zero, the effect of which is that your network will always report a single class. $\endgroup$ – Dan Scally Sep 23 '19 at 21:27
1
$\begingroup$

A reason might be that you are running a single-layer neural net. Ideally, you should have more than one layer, and use the sigmoid activation function.

| improve this answer | |
$\endgroup$
1
$\begingroup$

It happens often, that some model only predicts one class. The reason usually is, that the model is unable to distinguish the two classes well, and resorts to one (often the majority) class.

With your data, you may have a hard time to fit a NN with reasonable results. I suggest you check boosting which usually works okay with small data. Make sure you tune parameters well.

Here is an application of lightgbm to the iris data.

You could also check Logit with L1 regularization.

| improve this answer | |
$\endgroup$
0
$\begingroup$

Probably because you are using a single hidden layer which is unable to learn that many good parameters to differentiate between the classes.

try adding 1-2 more Dense layers.

you can try out this configuration as a baseline

model.add(Dense(128, input_shape=(22,), activation='relu'))

model.add(Dense(64, input_shape=(22,), activation='relu'))

at the output layer sigmoid will be fine because it is a binary classification.

model.add(Dense(1, activation = 'sigmoid')

Share your results so that we can explore more on this.

| improve this answer | |
$\endgroup$
  • $\begingroup$ Initially, adding more layers didn't help. After I applied SMOTE to balance my dataset, adding more layers did increase the accuracy. $\endgroup$ – achandra03 Sep 26 '19 at 2:17
  • $\begingroup$ yes that will increase accuracy, but in your case as you are having imbalanced class distribution, you can try other matrices to validate your model. like F1-score. Then you can go for confusion matrix to check where exactly it is going wrong for more debugging. $\endgroup$ – Desmond Sep 26 '19 at 3:25
  • $\begingroup$ I had an F1 score of .96875, so I'm pretty satisfied with how it is $\endgroup$ – achandra03 Sep 26 '19 at 18:34
  • $\begingroup$ how about the training and validation losses? $\endgroup$ – Desmond Sep 26 '19 at 18:39
  • $\begingroup$ Training loss - .1562 $\endgroup$ – achandra03 Sep 26 '19 at 18:41
0
$\begingroup$

Pin pointing exact reason would be impossible without looking at the data. Few things you can try:

  1. Add more layers, neurons in layers to make the model more expressive. Other has pointed this out already.
  2. As you have only 195 data points, try to see if there is a scope to use transfer learning.
  3. Look at the data, if imbalanced you can do following:
    • Upsampling/Downsampling
    • Use weighted cost, misclassification of less represented class will be penalised heavily compared to the other class.
    • Be careful with your error metric. Accuracy may not be the right choice.
  4. Play with the learning rate.
| improve this answer | |
$\endgroup$
0
$\begingroup$

You are using a softmax function with only one neuron in your last layer.

It is normal that your output is the same, one property of softmax function is that the sum of output across neurons equals 1 (1 neuron = 1 always).

Use a sigmoid function or set your last layer to 2 neuron.

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.