Neural network model for sparse multi-class classifier on Tensorflow

Question

The problem I'm trying to solve is the following:

the data is Movielens with N_users=6041 and N_movies=3953, ~1 million ratings.

For each user, a vector of size N_movies is defined, and the values of the vector are 1 if the user rated the movie before time T, 0 if not. For instance if the user rated movie 3 and 5 the input vector is [0,0,1,0,1].

The goal is to predict the movie the user will rate in the future (between time T and T+delta T). The labels are vector of size N_movies, and if the user rate the movie 4 the labels vector is [0,0,0,1,0].

I'm currently trying to get some initial results based on Fully connected layers, but it seems that it cannot optimize the loss at all. The representation might be too sparse, but it seems that the neural network should be able to learn at least some features.

Is it possible to make this model work, is there any problem with the loss function, or the optimizer?

from __future__ import print_function
import tensorflow as tf
from tensorflow.contrib import rnn
import csv
import bisect
import glob
import re
import numpy as np
import random
import data
import config

cfg = config.Config()
graph_data = data.Graph_data(cfg)

X = tf.placeholder("float", [None, cfg.N_movies])
Y = tf.placeholder("float", [None, cfg.N_movies])

def Dense(x):
    hidden_layer_1 = tf.layers.dense(inputs=x, units=500, activation=tf.nn.relu)
    hidden_layer_2 = tf.layers.dense(inputs=hidden_layer_1, units=50, activation=tf.nn.relu)
    output_layer = tf.layers.dense(inputs = hidden_layer_2, units= cfg.N_movies, activation=tf.nn.softmax)
    return output_layer

logits = Dense(X)
cross_entropy = tf.reduce_sum(- Y * tf.log(logits), 1)
loss_op = tf.reduce_mean(cross_entropy)
optimizer = tf.train.GradientDescentOptimizer(learning_rate=cfg.learning_rate)
train_op = optimizer.minimize(loss_op)

init = tf.global_variables_initializer()

with tf.Session() as sess:
  sess.run(init)

  for step in range(1, cfg.training_steps+1):
    batch_x, batch_y = graph_data.train_next_batch(cfg.batch_size)
    sess.run(train_op, feed_dict={X: batch_x, Y: batch_y})
    if step % cfg.display_step == 0 or step == 1:
      loss = sess.run(loss_op, feed_dict={X: batch_x,
                                                                  Y: batch_y})
      print("loss = ",loss)

Answer 1

The model seems fine to me. Did you try to optimize the hyperparameters? You could use trial and error methods for optimiring the parameters such as,

• Learning rate
• Number of epochs
• Batch size etc,.

And instead of vanilla gradient descent optimizer, you could try other optimizers such as Adagrad, Adam or RMSProp etc,.

Best, Sangathamilan Ravichandran.

Answer 2

Given your data is sparse it may be worth trying a loss function for that type of representation.

As an example Keras has a SparseCategoricalCrossentropy class:

tf.keras.losses.SparseCategoricalCrossentropy(
    from_logits=False, reduction="auto", name="sparse_categorical_crossentropy"
)

More information on this is available here.

Finally, it is always worth it to try running your model with mock data, just to be sure your computational graph is initiated as expected.