2
$\begingroup$

I am currently training an ANN using Sequential(a class from Keras API within tensorflow), and I am optimizing the model's architecture and came across something I have not seen before.

The graph of the test accuracy seems a bit odd. The graph appears that it is not a smooth curve, but different.

Model

import pandas as pd
from tensorflow import keras
from tensorflow.keras import layers


from sklearn.model_selection import train_test_split 
from tensorflow.keras.models import Sequential 
from tensorflow.keras.layers import Dense

features = data[['Vcom', 'T', 'Vair']]
labels = data[['COP', 'CC']]

features_train, features_test, labels_train, labels_test = train_test_split(features, labels, test_size=0.2, random_state=42)

from tensorflow.keras.layers import  Dropout , BatchNormalization
from tensorflow.keras import regularizers


model = Sequential() 
l2_regularizer = regularizers.l2(0.01)


model.add(Dense(64, activation='relu', input_shape=(3,), kernel_regularizer=l2_regularizer))
model.add(BatchNormalization())

model.add(Dropout(0.6))
model.add(Dense(32, activation='relu', kernel_regularizer=l2_regularizer))

model.add(BatchNormalization())

model.add(Dropout(0.6))
model.add(Dense(32, activation='relu'))
model.add(BatchNormalization())


model.add(Dense(2,))  # 2 output neurons for output1 and output2

model.compile(optimizer='adam', loss='mean_squared_error', metrics=['accuracy'])

model_history=model.fit(features_train, labels_train, epochs=150, batch_size=32, validation_data=(features_test, labels_test))


print(model_history.history.keys())
# summarize history for accuracy
plt.plot(model_history.history['accuracy'])
plt.plot(model_history.history['val_accuracy'])
plt.title('model accuracy')
plt.ylabel('accuracy')
plt.xlabel('epoch')
plt.legend(['train', 'test'], loc='upper left')
plt.show()

My question is:

  1. Why the test accuracy showing odd patterns ?

  2. How can I fix this problem?

enter image description here

$\endgroup$
6
  • $\begingroup$ Looks like a problem with the validation data. Have you checked it for "strange things"? Also, is the training data preprocessing also applied to the validation data? $\endgroup$
    – noe
    Commented Nov 25, 2023 at 19:05
  • $\begingroup$ @noe I have split the data of 20% for validation set. No, data preprocessing is not applied specially for validation data . $\endgroup$
    – Aach_copro
    Commented Nov 25, 2023 at 19:12
  • 1
    $\begingroup$ Can you share both your training and test dataset? Looks like you're applying your ANN to a fairly simple problem and thus overfit occurs just after 30 epochs. $\endgroup$
    – cinch
    Commented Nov 26, 2023 at 8:02
  • $\begingroup$ @mohottnad I have just one dataset, then I have split the dataset into training and test dataset $\endgroup$
    – Aach_copro
    Commented Nov 26, 2023 at 9:36
  • $\begingroup$ To understand further what is happening, have you (1) looked at the mean-squared-error plots (i.e. plotting the metric you are using as the loss function) for the training and validation data and (2) tried with a different train/test split (i.e. use a different random state in the call to train_test_split)? $\endgroup$
    – Lynn
    Commented Nov 26, 2023 at 10:17

1 Answer 1

3
$\begingroup$

Why the test accuracy showing odd patterns ?

As @mohottnad mentioned in the comment, it appears your model overfits. It means that it doesn't generalise well and works badly on testing data. I don't know the details of you data, but this strange pattern of accuracy might be explained as follows:

  1. after many epochs your model is so overtrained that it return just one class.
  2. in epoch 100 this class is A and it covers correctly almost 60% of data
  3. in epoch 115 decision threshold changes slightly and the model's guess is B that is correct in +- 1/3 of data
  4. and so on, and so on

It's my hypothesis you can dissect, your situation probably isn't as simple as I described, but it might be very similar. You can also check if you have an imbalanced dataset that might be an issue.

How can I fix this problem?

I don't know how complex your variables are, but you have just 3 input variables and you feed them into 64->32->32 neurons. In many cases, it's too much. I would try with a way simpler model like 16->8 neurons or even smaller. Your demend on model complexity is associated with complexity of relationships in your data. Maybe you need just a logistic regression, who knows? It's up to you to experiment and offset it. The value=0.6 in dropout for this data and model also appears to be too high.

$\endgroup$
1
  • 1
    $\begingroup$ Now I understand that what was the problem in my model, after experimenting with model for different values , I get optimised values which solved the issues. $\endgroup$
    – Aach_copro
    Commented Nov 27, 2023 at 11:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.