Model Not Learning with Sparse Dataset (LSTM with Keras)

Question

This classification problem is apparently simple and I have no idea why it's not working, perhaps I'm doing a conceptual mistake. I'm trying to make a predictor which will classify minutes on a clock as 0 for no requests or 1 for a request in that time instant. So far, the model can't seem to learn past giving everything 0's or putting zeros before all the requests happen. The best case scenario I've got so far is:

I've been trying to implement all the things I've learned related to Neural Networks and my code works in the following manner:

1. Create the Clock.
2. Create some Requests (I've made them last longer so the dataset is less sparse).
3. Scale (MinMax) the independent variable (clock).
4. Create a custom loss function for the sparse dataset (I've tried it with a regular logits, but it can't go much further than giving zeros to almost everything; so I penalized mistakes made on the instants with requests).
5. Create the model, composed of 3 stacked LSTM layers (Both a regular NN and an LSTM have been tried).

Other techniques that have also been tried are Dropout, Gradient Clipping (for exploding gradients) and extremely small learning rates with ADAM (on the order of $10^{-5}$). Basically, I just can't go much further than an accuracy of around 73.33%.

The code below is indeed longer than I would like to believe I'm allowed to post here, but it is also, hopefully, rather simple to read. Thank you for trying to help me.

# Data Pre-processing

## The Clock

clock = []
for i in range(0,24):
    for j in range(0,60):
        if i < 10:
            if j < 10:
                clock.append('0' + str(i) + '0' + str(j))
            else:
                clock.append('0' + str(i) + str(j))
        else:
            if j < 10:
                clock.append(str(i) + '0' + str(j))
            else:
                clock.append(str(i) + str(j))

## Requests

levels = 2
request_full = [0]*len(clock)
request_1300 = []
request_1500 = []
request_1800 = []
request_2100 = []
request_2300 = []
for i in range(0,60):
    if i < 10:
        request_1300.append(['13' + '0' + str(i), 1])
        request_1500.append(['15' + '0' + str(i), 1])
        request_1800.append(['18' + '0' + str(i), 1])
        request_2100.append(['21' + '0' + str(i), 1])
        request_2300.append(['23' + '0' + str(i), 1])
    else:
        request_1300.append(['13' + str(i), 1])
        request_1500.append(['15' + str(i), 1])
        request_1800.append(['18' + str(i), 1])
        request_2100.append(['21' + str(i), 1])
        request_2300.append(['23' + str(i), 1])

request_list = [i for i in 
                request_1300 + 
                request_1500 + 
                request_1800 + 
                request_2100 + 
                request_2300]

for i,j in request_list:
    idx = clock.index(i)
    request_full[idx] = j

import numpy as np
import pandas as pd

df = pd.DataFrame({'clock': clock, 'requests': request_full})

df.to_csv('test1.csv', sep = ',')

clock = df['clock'].values
clock = np.reshape(clock, (len(clock), 1))
requests = df['requests'].values
requests = np.reshape(requests, (len(requests), 1))

## Feature Scaling
from sklearn.preprocessing import MinMaxScaler

sc = MinMaxScaler(feature_range = (0, 1))
training_set_scaled = sc.fit_transform(clock)

X_train = []
y_train = []
for i in range(60, len(clock)):
    X_train.append(training_set_scaled[i-60:i, 0])
    y_train.append(requests[i, 0])
X_train, y_train = np.array(X_train), np.array(y_train)

X_train = np.reshape(X_train, (X_train.shape[0], X_train.shape[1], 1))

from keras.models import Sequential
from keras.layers import Dense
from keras.layers import LSTM
from keras.layers import Dropout
from keras import optimizers
import tensorflow as tf

def sparse_penalty_logits(y_true, y_pred):
    penalty = 10

    loss = tf.where(tf.greater(y_true, 0),
                    penalty*tf.nn.sigmoid_cross_entropy_with_logits(logits = y_pred,
                                                                    labels = y_true),
                    tf.nn.sigmoid_cross_entropy_with_logits(logits = y_pred,
                                                            labels = y_true))

    return loss

# Initialising the RNN
classifier = Sequential()

# Adding the first LSTM layer and some Dropout regularisation
classifier.add(LSTM(units = 60, 
                    return_sequences = True,
                    input_shape = (X_train.shape[1], 1)))
classifier.add(Dropout(0.2))


# 2nd Layer
classifier.add(LSTM(units = 60, 
                    return_sequences = True))
#classifier.add(Dropout(0.2))

# 3rd Layer
classifier.add(LSTM(units = 60, 
                    return_sequences = False))
#classifier.add(Dropout(0.2))

# Adding the output layer
classifier.add(Dense(units = 1, 
                     activation = 'sigmoid'))

# Compiling the RNN
adam = optimizers.Adam(lr = 10**(-5), 
                       clipnorm = 1, 
                       clipvalue = 0.5)

classifier.compile(optimizer = adam, 
                   loss = sparse_penalty_logits,
                   metrics = ['binary_accuracy'])

# Fitting the RNN to the Training set
classifier.fit(X_train, 
               y_train, 
               epochs = 100)


# Making the predictions with the dataset
y_pred = classifier.predict(X_train)
y_pred_bin = (y_pred > 0.5)

from sklearn.metrics import confusion_matrix

cm = confusion_matrix(y_train, y_pred_bin)

# Plotting
import matplotlib.pyplot as plt

plt.plot(y_pred, color = 'green', label = 'model')
plt.plot(y_pred_bin, color = 'red', label = 'model binary')
plt.plot(y_train, color = 'blue', label = 'training')
plt.legend()
plt.show()
plt.close()

Answer 1

First of all, this has been quite a journey to solve. I had to ask for a friend's help who, in turn, presented me to a friend of his. The solution is something that appears somewhat intuitive to me, but, nevertheless, was not at all on my radar.

Basically, since we only have one row of data to look at, the fit is going to be too sharp for the NN, so it will most likely always plateau and stick to spitting out only zeros. To make things easier for the NN, the suggested solution is to apply some noise to our data, so that the system won't think that it needs to get a very sharp fit in order to do its job.

Despite this little trick, it is still very hard for the NN to get a good fit because it's still one row, so, now, if we use ADAM (which dynamically reduces the learning rate), it might get stuck in local minima and never come out. The last tip then is to use incredibly high learning rates, in my case, between 0.2 and 0.3. The learning process will not be stable either and, thus, lastly, it's necessary to use an absurd amount of epochs (20,000) with callbacks to register the most optimal weights. This way, my friend's friend solution got us to 99.8% accuracy.

Despite the good performance, quite frankly, he and I are both in agreement that the most reliable and fast solution to this problem is actually to use a Decision Tree type of algorithm.

I am not able to provide the source code of the solution here due to a request of its author.

Answer 2

I would suggest you try neural poisson processes. What you are try to predict is a point process with a fixed (or variable in hawkes processes) density function. As such I am not sure simply modeling it as a sequence can give you reasonable performance.