High model Accuracy and very low validation accuracy

Question

I'm building a neural network model in python using Keras l for deep learning

my data is having vital information , age and other health symptoms about 1385 individuals and the output label predict about the liver condition: output class

Cirrhosis :362 Few Septa :332 Many Septa :355 Portal Fibrosis :336

I wrote a sequential model script to make the discrete classification as follows:

1- while importing data it is randomized as below

pd.options.display.max_rows = 10
pd.options.display.float_format = '{:.1f}'.format

liver_dataframe = pd.read_csv("LiverData.csv", sep=",")

liver_dataframe = liver_dataframe.reindex(
    np.random.permutation(liver_dataframe.index))

2- The data contains categorical data .so I did the clean up as below -

cleanup_nums = {"Gender":     {"Male": 1, "Female": 0},
                "Fever": {"Present": 1, "Absent" : 0},
                "Nausea/Vomting" : {"Present": 1, "Absent": 0},
                "Headache " :{"Present": 1, "Absent": 0},
                "Diarrhea " :{"Present": 1, "Absent": 0},
                "Fatigue & generalized bone ache ":{"Present": 1, "Absent": 0},
                "Jaundice " :{"Present": 1, "Absent": 0},
                "Epigastric pain ":{"Present": 1, "Absent": 0},
                 
                "Class" :{"Cirrhosis" : 0,"Many Septa" :1,"Portal Fibrosis" :2,"Few Septa" :3}
                 }

3- DataFrame info

liver_dataframe.replace(cleanup_nums, inplace=True)
liver_dataframe.head(10)

4 -PreProcess of Data

def preprocess_features(liver_dataframe):
  """Prepares input features from concrete slump test data set.

  Args:
    liver_dataframe: A Pandas DataFrame expected to contain data
      from the liver_dataframe.
  Returns:
    A DataFrame that contains the features to be used for the model. 
  """
  selected_features = liver_dataframe[
    ["Age " ,"Gender","Fever","Nausea/Vomting","Headache ","Diarrhea ",
     "Fatigue & generalized bone ache ","Jaundice ","Epigastric pain ",
     
     "BMI","WBC","RBC","HGB","Plat","AST 1","ALT 1","ALT4","ALT 12","ALT 24","ALT 36","ALT 48","ALT after 24 w","RNA Base",
     "RNA 4","RNA 12","RNA EOT","RNA EF"]]
    
  processed_features = selected_features.copy()
  
  return processed_features

def preprocess_targets(liver_dataframe):
  """Prepares target features (i.e., labels) from  liver data set.

  Args:
    dataframe: A Pandas DataFrame expected to contain data
      from the data set.
  Returns:
    A DataFrame that contains the target feature.
  """
  output_targets = liver_dataframe["Class"]
  return output_targets 

5- Choose the first 1104 examples for training.

training_examples = preprocess_features(liver_dataframe.head(1104))
training_targets = preprocess_targets(liver_dataframe.head(1104))
scaler = StandardScaler().fit(training_examples.values)
scaledf = scaler.transform(training_examples.values)
training_examples = pd.DataFrame(scaledf, index=training_examples.index, columns=training_examples.columns)

Choose the 281 examples for validation.

validation_examples = preprocess_features(liver_dataframe.tail(281))
vscaled = scaler.transform(validation_examples.values)
validation_examples = pd.DataFrame(vscaled, index=validation_examples.index, columns=validation_examples.columns)
validation_targets = preprocess_targets(liver_dataframe.tail(281))

6-Build Model

baseline_model = keras.Sequential([
    keras.layers.Dense(54, activation=tf.nn.relu,
                       input_shape=(training_examples.shape[1],)),
    keras.layers.Dense(80, activation=tf.nn.relu,),
    keras.layers.Dense(4,activation=tf.nn.softmax)

  ])

baseline_model.compile(loss = 'sparse_categorical_crossentropy',
                optimizer="adam",
                metrics=['accuracy'])
baseline_model.summary()

7- Fit Model

class PrintDot(keras.callbacks.Callback):
  def on_epoch_end(self, epoch, logs):
    if epoch % 100 == 0: print('')
    print('.', end='')

EPOCHS = 500
b_history = baseline_model.fit(training_examples, training_targets, epochs=EPOCHS,
                    validation_data= (validation_examples, validation_targets),
                    )

after 500 epochs my output is as below

500
1104/1104 [==============================] - 0s 75us/sample - loss: 1.0799e-06 - accuracy: 1.0000 - val_loss: 12.9441 - val_accuracy: 0.2242

baseline_model.evaluate(validation_examples, validation_targets)

281/281 [==============================] - 0s 58us/sample - loss: 12.9441 - accuracy: 0.2242
[12.944118489574283, 0.2241993]

With the model, the training accuracy is 100 pct while the validation accuracy is only 22 pct.

As a beginner, I am not sure how to improve performance. How can I proceed?

Answer 1

Your model probably overfits. This articles provides an easy to read intro to the topic.

As a very first step I suggest to plot your learning curves and look for epochs with lower validation loss. Also, this helps to properly diagnose your model in terms of model capacity. Training a model for less epochs is one way to reduce model capacity and avoid overfitting (others are, for example, to increase the number of training samples or introduce dropout but that is not the first thing to do here).

Section 11.4.1 of the Deep Learning Book provides a good overview on this, too.

Moreover, I would double check the training and validation sets for a similar class distribution to make sure it is not related to your data split.