# Model Not Learning with Sparse Dataset (LSTM with Keras)

This classification problem is apparently simple and I have no idea why it's not working, perhaps I'm doing a conceptual mistake. I'm trying to make a predictor which will classify minutes on a clock as 0 for no requests or 1 for a request in that time instant. So far, the model can't seem to learn past giving everything 0's or putting zeros before all the requests happen. The best case scenario I've got so far is:

I've been trying to implement all the things I've learned related to Neural Networks and my code works in the following manner:

1. Create the Clock.
2. Create some Requests (I've made them last longer so the dataset is less sparse).
3. Scale (MinMax) the independent variable (clock).
4. Create a custom loss function for the sparse dataset (I've tried it with a regular logits, but it can't go much further than giving zeros to almost everything; so I penalized mistakes made on the instants with requests).
5. Create the model, composed of 3 stacked LSTM layers (Both a regular NN and an LSTM have been tried).

Other techniques that have also been tried are Dropout, Gradient Clipping (for exploding gradients) and extremely small learning rates with ADAM (on the order of $$10^{-5}$$). Basically, I just can't go much further than an accuracy of around 73.33%.

The code below is indeed longer than I would like to believe I'm allowed to post here, but it is also, hopefully, rather simple to read. Thank you for trying to help me.

# Data Pre-processing

## The Clock

clock = []
for i in range(0,24):
for j in range(0,60):
if i < 10:
if j < 10:
clock.append('0' + str(i) + '0' + str(j))
else:
clock.append('0' + str(i) + str(j))
else:
if j < 10:
clock.append(str(i) + '0' + str(j))
else:
clock.append(str(i) + str(j))

## Requests

levels = 2
request_full = [0]*len(clock)
request_1300 = []
request_1500 = []
request_1800 = []
request_2100 = []
request_2300 = []
for i in range(0,60):
if i < 10:
request_1300.append(['13' + '0' + str(i), 1])
request_1500.append(['15' + '0' + str(i), 1])
request_1800.append(['18' + '0' + str(i), 1])
request_2100.append(['21' + '0' + str(i), 1])
request_2300.append(['23' + '0' + str(i), 1])
else:
request_1300.append(['13' + str(i), 1])
request_1500.append(['15' + str(i), 1])
request_1800.append(['18' + str(i), 1])
request_2100.append(['21' + str(i), 1])
request_2300.append(['23' + str(i), 1])

request_list = [i for i in
request_1300 +
request_1500 +
request_1800 +
request_2100 +
request_2300]

for i,j in request_list:
idx = clock.index(i)
request_full[idx] = j

import numpy as np
import pandas as pd

df = pd.DataFrame({'clock': clock, 'requests': request_full})

df.to_csv('test1.csv', sep = ',')

clock = df['clock'].values
clock = np.reshape(clock, (len(clock), 1))
requests = df['requests'].values
requests = np.reshape(requests, (len(requests), 1))

## Feature Scaling
from sklearn.preprocessing import MinMaxScaler

sc = MinMaxScaler(feature_range = (0, 1))
training_set_scaled = sc.fit_transform(clock)

X_train = []
y_train = []
for i in range(60, len(clock)):
X_train.append(training_set_scaled[i-60:i, 0])
y_train.append(requests[i, 0])
X_train, y_train = np.array(X_train), np.array(y_train)

X_train = np.reshape(X_train, (X_train.shape[0], X_train.shape[1], 1))

from keras.models import Sequential
from keras.layers import Dense
from keras.layers import LSTM
from keras.layers import Dropout
from keras import optimizers
import tensorflow as tf

def sparse_penalty_logits(y_true, y_pred):
penalty = 10

loss = tf.where(tf.greater(y_true, 0),
penalty*tf.nn.sigmoid_cross_entropy_with_logits(logits = y_pred,
labels = y_true),
tf.nn.sigmoid_cross_entropy_with_logits(logits = y_pred,
labels = y_true))

return loss

# Initialising the RNN
classifier = Sequential()

# Adding the first LSTM layer and some Dropout regularisation
return_sequences = True,
input_shape = (X_train.shape[1], 1)))

# 2nd Layer
return_sequences = True))

# 3rd Layer
return_sequences = False))

activation = 'sigmoid'))

# Compiling the RNN
clipnorm = 1,
clipvalue = 0.5)

loss = sparse_penalty_logits,
metrics = ['binary_accuracy'])

# Fitting the RNN to the Training set
classifier.fit(X_train,
y_train,
epochs = 100)

# Making the predictions with the dataset
y_pred = classifier.predict(X_train)
y_pred_bin = (y_pred > 0.5)

from sklearn.metrics import confusion_matrix

cm = confusion_matrix(y_train, y_pred_bin)

# Plotting
import matplotlib.pyplot as plt

plt.plot(y_pred, color = 'green', label = 'model')
plt.plot(y_pred_bin, color = 'red', label = 'model binary')
plt.plot(y_train, color = 'blue', label = 'training')
plt.legend()
plt.show()
plt.close()