As the title states, my validation accuracy isn't changing when I try to train my model. I've built an NVIDIA model using tensorflow.keras in python. I have absolutely no idea what's causing the issue.
Here is a link to the google colab I'm writing this in. But I've also put all my code below, below the model summary and Epoch history.
Please help. I'm slowly going insane.
Model Summary:
Model: "sequential_3"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
conv2d_15 (Conv2D) (None, 31, 98, 24) 1824
_________________________________________________________________
conv2d_16 (Conv2D) (None, 14, 47, 36) 21636
_________________________________________________________________
conv2d_17 (Conv2D) (None, 5, 22, 48) 43248
_________________________________________________________________
conv2d_18 (Conv2D) (None, 3, 20, 64) 27712
_________________________________________________________________
conv2d_19 (Conv2D) (None, 1, 18, 64) 36928
_________________________________________________________________
dropout_8 (Dropout) (None, 1, 18, 64) 0
_________________________________________________________________
flatten_3 (Flatten) (None, 1152) 0
_________________________________________________________________
dense_13 (Dense) (None, 100) 115300
_________________________________________________________________
dropout_9 (Dropout) (None, 100) 0
_________________________________________________________________
dense_14 (Dense) (None, 50) 5050
_________________________________________________________________
dropout_10 (Dropout) (None, 50) 0
_________________________________________________________________
dense_15 (Dense) (None, 10) 510
_________________________________________________________________
dropout_11 (Dropout) (None, 10) 0
_________________________________________________________________
dense_16 (Dense) (None, 1) 11
=================================================================
Total params: 252,219
Trainable params: 252,219
Non-trainable params: 0
_________________________________________________________________
None
Epoch History:
Epoch 1/30
18/18 [==============================] - 14s 716ms/step - loss: 0.4594 - accuracy: 0.1236 - val_loss: 0.1029 - val_accuracy: 0.1287
Epoch 2/30
18/18 [==============================] - 13s 702ms/step - loss: 0.1315 - accuracy: 0.1369 - val_loss: 0.0946 - val_accuracy: 0.1287
Epoch 3/30
18/18 [==============================] - 13s 701ms/step - loss: 0.1042 - accuracy: 0.1397 - val_loss: 0.0842 - val_accuracy: 0.1287
Epoch 4/30
18/18 [==============================] - 13s 697ms/step - loss: 0.0972 - accuracy: 0.1392 - val_loss: 0.0753 - val_accuracy: 0.1287
Epoch 5/30
18/18 [==============================] - 13s 699ms/step - loss: 0.0887 - accuracy: 0.1397 - val_loss: 0.0692 - val_accuracy: 0.1287
Epoch 6/30
18/18 [==============================] - 13s 698ms/step - loss: 0.0837 - accuracy: 0.1392 - val_loss: 0.0653 - val_accuracy: 0.1287
Epoch 7/30
18/18 [==============================] - 13s 699ms/step - loss: 0.0775 - accuracy: 0.1397 - val_loss: 0.0604 - val_accuracy: 0.1287
Epoch 8/30
18/18 [==============================] - 13s 700ms/step - loss: 0.0721 - accuracy: 0.1392 - val_loss: 0.0593 - val_accuracy: 0.1287
Epoch 9/30
18/18 [==============================] - 13s 698ms/step - loss: 0.0741 - accuracy: 0.1397 - val_loss: 0.0603 - val_accuracy: 0.1287
Epoch 10/30
18/18 [==============================] - 13s 700ms/step - loss: 0.0730 - accuracy: 0.1397 - val_loss: 0.0573 - val_accuracy: 0.1287
Epoch 11/30
18/18 [==============================] - 13s 705ms/step - loss: 0.0705 - accuracy: 0.1403 - val_loss: 0.0558 - val_accuracy: 0.1287
Epoch 12/30
18/18 [==============================] - 13s 701ms/step - loss: 0.0691 - accuracy: 0.1403 - val_loss: 0.0544 - val_accuracy: 0.1287
Epoch 13/30
18/18 [==============================] - 13s 705ms/step - loss: 0.0619 - accuracy: 0.1397 - val_loss: 0.0538 - val_accuracy: 0.1287
Epoch 14/30
18/18 [==============================] - 12s 694ms/step - loss: 0.0653 - accuracy: 0.1403 - val_loss: 0.0541 - val_accuracy: 0.1287
Epoch 15/30
18/18 [==============================] - 13s 695ms/step - loss: 0.0612 - accuracy: 0.1409 - val_loss: 0.0516 - val_accuracy: 0.1287
Epoch 16/30
18/18 [==============================] - 13s 696ms/step - loss: 0.0594 - accuracy: 0.1403 - val_loss: 0.0500 - val_accuracy: 0.1287
Epoch 17/30
18/18 [==============================] - 13s 701ms/step - loss: 0.0577 - accuracy: 0.1403 - val_loss: 0.0483 - val_accuracy: 0.1287
Epoch 18/30
18/18 [==============================] - 13s 701ms/step - loss: 0.0552 - accuracy: 0.1420 - val_loss: 0.0539 - val_accuracy: 0.1287
Epoch 19/30
18/18 [==============================] - 13s 695ms/step - loss: 0.0563 - accuracy: 0.1415 - val_loss: 0.0488 - val_accuracy: 0.1287
Epoch 20/30
18/18 [==============================] - 13s 699ms/step - loss: 0.0556 - accuracy: 0.1409 - val_loss: 0.0502 - val_accuracy: 0.1287
Epoch 21/30
18/18 [==============================] - 13s 709ms/step - loss: 0.0534 - accuracy: 0.1403 - val_loss: 0.0475 - val_accuracy: 0.1287
Epoch 22/30
18/18 [==============================] - 13s 704ms/step - loss: 0.0579 - accuracy: 0.1397 - val_loss: 0.0488 - val_accuracy: 0.1287
Epoch 23/30
18/18 [==============================] - 13s 697ms/step - loss: 0.0514 - accuracy: 0.1420 - val_loss: 0.0467 - val_accuracy: 0.1287
Epoch 24/30
18/18 [==============================] - 13s 696ms/step - loss: 0.0490 - accuracy: 0.1415 - val_loss: 0.0479 - val_accuracy: 0.1287
Epoch 25/30
18/18 [==============================] - 13s 695ms/step - loss: 0.0472 - accuracy: 0.1403 - val_loss: 0.0477 - val_accuracy: 0.1287
Epoch 26/30
18/18 [==============================] - 13s 698ms/step - loss: 0.0495 - accuracy: 0.1415 - val_loss: 0.0455 - val_accuracy: 0.1287
Epoch 27/30
18/18 [==============================] - 13s 711ms/step - loss: 0.0473 - accuracy: 0.1415 - val_loss: 0.0463 - val_accuracy: 0.1287
Epoch 28/30
18/18 [==============================] - 13s 703ms/step - loss: 0.0466 - accuracy: 0.1415 - val_loss: 0.0451 - val_accuracy: 0.1287
Epoch 29/30
18/18 [==============================] - 13s 700ms/step - loss: 0.0439 - accuracy: 0.1426 - val_loss: 0.0448 - val_accuracy: 0.1287
Epoch 30/30
18/18 [==============================] - 13s 698ms/step - loss: 0.0446 - accuracy: 0.1415 - val_loss: 0.0482 - val_accuracy: 0.1287
Code:
# Utility
import numpy as np
import matplotlib.pyplot as plt
import matplotlib.image as mpimg
import pandas as pd
from sklearn.utils import shuffle
from sklearn.model_selection import train_test_split
import cv2
# For saving the model and reading in data
import os
import ntpath
from tensorflow.keras.models import load_model
from tensorflow.keras.utils import plot_model
# Copy the repository of images
from git import Repo
if not os.path.isdir("track"):
Repo.clone_from("https://github.com/Grant-Allan/Track", "track")
# For the model
from tensorflow.keras.models import Sequential
from tensorflow.keras.optimizers import Adam
from tensorflow.keras.layers import Conv2D
from tensorflow.keras.layers import Dense
from tensorflow.keras.layers import Flatten
from tensorflow.keras.layers import Dropout
# Split off everything that isn't the name of the image from the path
def path_leaf(path):
head, tail = ntpath.split(path)
return tail
# Read in the driving data
def read_data(datadir):
# Name the columns of data
columns = ['center', 'left', 'right', 'steering', 'throttle', 'reverse', 'speed']
# Read in the data from the cvs while naming the columns
data = pd.read_csv(os.path.join(datadir, "driving_log.csv"), names=columns)
# Fix the paths to only have the images
data['center'] = data['center'].apply(path_leaf)
data['left'] = data['left'].apply(path_leaf)
data['right'] = data['right'].apply(path_leaf)
# Set the width of cells in the table to show the entire name
pd.set_option('display.max_colwidth', 12)
# Display the top five rows of the data
# print(data.head())
return data
# Get the intervals of the steering angles
def intervals(data, num_bins):
# Balance the data in order to prevent the data from being overly
# biased towards one angle or another.
samples_per_bin = 300
# The values and bins of the steering angles
hist, bins = np.histogram(data['steering'], num_bins)
# Balance the data, then redo the histogram
data = balance_data(data, bins, num_bins, samples_per_bin)
hist, bins = np.histogram(data['steering'], num_bins)
# Centers the center bin by adding the two bins next to it into one bin
# This doubles all the other values, requiring us to divide by 2
center = (bins[:-1] + bins[1:])/2
# Graph it
plt.title('Balanced Data Histogram')
plt.bar(center, hist, width=0.05)
return data
def balance_data(data, bins, num_bins, samples_per_bin):
remove_list = []
# For each bin
for j in range(num_bins):
list_ = []
# For each steering angle
for i in range(len(data['steering'])):
# If the steering angle is in the current bin
if data['steering'][i] >= bins[j] and data['steering'][i] <= bins[j+1]:
list_.append(i)
# Shuffle the data so that when we cut off part of it, we don't cut off
# an entire section of data (since the input is ordered, not random).
list_ = shuffle(list_)
# Cut off the tail end of the list; all elements beyond the samples_per_bin element
# We use a new list since we use list_ for a number of more loops
list_ = list_[samples_per_bin:]
remove_list.extend(list_)
data.drop(data.index[remove_list], inplace=True)
return data
def load_img_steering(datadir):
# Add the IMG folder to the directory path
datadir = os.path.join(datadir, "IMG")
# Create lists for later
image_path = []
steering = []
for i in range(len(data)):
# iloc allows us to perform a selection on a row of data in the data
# frame based on a specified index. In this case, we go row by row.
indexed_data = data.iloc[i]
# The indexes correspond to center, left, right (0, 1, 2)
# We only want the center values for the moment.
# strip gets rid of any spaces in the name
image_path.append(os.path.join(datadir, indexed_data[0].strip()))
# Get the steering angle corresponding to the image
steering.append(float(indexed_data[3]))
# Convert image_path and steering into arrays for easy manipulation
image_paths = np.asarray(image_path)
steerings = np.asarray(steering)
return image_paths, steerings
def set_histograms(y_train, y_val, num_bins):
fig, axes = plt.subplots(1, 2, figsize=(12, 4))
axes[0].hist(y_train, bins=num_bins, width=0.05, color='blue')
axes[0].set_title('Training Set')
axes[1].hist(y_val, bins=num_bins, width=0.05, color='red')
axes[1].set_title('Validation Set')
plt.show()
# Preprocess the images
def preprocessing(img):
# Use the image path to get an actual image
img = mpimg.imread(img)
# Get rid of noise chunks
# Crop the top of the image to remove the sky. Crop the
# bottom of the image to get rid of the hood of our car.
img = img[60:135, :, :]
# Convert the images to YUV (recommended for the NVIDIA neural network model)
# Y is the luminance/brightness/grayscale
# UV are chrominance (color for the image) (they're color difference)
img = cv2.cvtColor(img, cv2.COLOR_RGB2YUV)
# Gaussian blur the image to reduce noise
# kernel size and standard deviation.
img = cv2.GaussianBlur(img, (3, 3), 0)
# Resize the image to decrease size to allow for faster computations
# (200, 66) matches the input size of the NVIDIA architecture, which
# is a nice bonus
img = cv2.resize(img, (200, 66))
# Normalize the data
img = img/255
return img
# Create the NVIDIA model
# The model is discussed in detail in "End to End Learning For Self-Driving Cars"
def create_model():
# Define the model using Sequential
model = Sequential()
# We use elu instead of relu to avoid the dead relu problem, which we had an issue with.
model.add(Conv2D(24, kernel_size=(5,5), strides=(2,2), activation='elu', input_shape=(66,200,3)))
model.add(Conv2D(36, kernel_size=(5,5), strides=(2,2), activation='elu'))
model.add(Conv2D(48, kernel_size=(5,5), strides=(2,2), activation='elu'))
model.add(Conv2D(64, kernel_size=(3,3), activation='elu'))
model.add(Conv2D(64, kernel_size=(3,3), activation='elu'))
model.add(Dropout(0.5))
model.add(Flatten())
# model.add(Dense(1164, activation='elu'))
model.add(Dense(100, activation='elu'))
model.add(Dropout(0.5))
model.add(Dense(50, activation='elu'))
model.add(Dropout(0.5))
model.add(Dense(10, activation ='elu'))
model.add(Dropout(0.5))
model.add(Dense(1))
# Learning rate of 0.0001
model.compile(optimizer=Adam(learning_rate=1e-3), loss='mse', metrics=['accuracy'])
return model
# Either load or create the model and model history
def get_model(X_train, y_train, X_val, y_val):
if os.path.isfile('model.h5') and os.path.isfile('history.npy'):
# Load the saved model
model = load_model('model')
# Print the architecture summary
print("\n=================================================================")
print(model.summary())
print("=================================================================\n")
# Load the saved model history
history = np.load('history.npy', allow_pickle='True').item()
else:
# Create the model
model = create_model()
# Print the architecture summary
print("")
print(model.summary())
print("")
# Train the model and get the History() object
history_obj = model.fit(x=X_train, y=y_train, validation_data=(X_val, y_val), batch_size=100, epochs=30, verbose=1, shuffle=1)
print("\n")
# Save the model
model.save('model.h5')
# Save the model summary visualization
plot_model(model, to_file='model.png', show_shapes=True, show_dtype=False, show_layer_names=True, rankdir='TB', expand_nested=False, dpi=96)
# Set and save the history of the loss and accuracy values of the model
history = history_obj.history
np.save('history.npy', history)
return model, history
# Plot the model loss and accuracy
def loss_and_accuracy(history):
fig, axes = plt.subplots(1, 2, figsize=(12, 4))
axes[0].set_title("Loss")
axes[0].plot(history["loss"])
axes[0].plot(history["val_loss"])
axes[0].legend(["Loss", "Validation Loss"])
axes[0].set_xlabel ("Epoch")
axes[1].set_title("Accuracy")
axes[1].plot(history["accuracy"])
axes[1].plot(history["val_accuracy"])
axes[1].legend(["Accuracy", "Validation Accuracy"])
axes[1].set_xlabel ("Epoch")
plt.show()
# Main driver code
if __name__ == "__main__":
# Set parent directory
datadir = "track"
# Get the driving data
data = read_data(datadir)
# Plot the intervals
num_bins = 25
data = intervals(data, num_bins)
# Load the image paths and corresponding steering angles
image_paths, steerings = load_img_steering(datadir)
# Split into training and validation sets
# test_size is the proportion of test to valid; this splits 80% into test and 20% into valid
# random_state is the seed for the random number generator
X_train, X_val, y_train, y_val = train_test_split(image_paths, steerings, test_size=0.2, random_state=6)
print("Training Samples: {}\nValidation Samples: {}".format(len(X_train), len(X_val)))
# Show the steering angle histograms of each set
set_histograms(y_train, y_val, num_bins)
# Display a sample image to make sure the preprocessing is working
image = image_paths[100]
original_image = mpimg.imread(image)
prep_img = preprocessing(image)
fig, axes = plt.subplots(1, 2, figsize=(15, 10))
fig.tight_layout()
axes[0].imshow(original_image)
axes[0].set_title("Original Image")
axes[1].imshow(prep_img)
axes[1].set_title("Preprocessed Image")
plt.show()
# Preprocess the images
X_train = np.array(list(map(preprocessing, X_train)))
X_val = np.array(list(map(preprocessing, X_val)))
# Get the model and history
model, history = get_model(X_train, y_train, X_val, y_val)
# Plot model loss and accuracy
loss_and_accuracy(history)
