# Neural network model for sparse multi-class classifier on Tensorflow

The problem I'm trying to solve is the following:

the data is Movielens with N_users=6041 and N_movies=3953, ~1 million ratings.

For each user, a vector of size N_movies is defined, and the values of the vector are 1 if the user rated the movie before time T, 0 if not. For instance if the user rated movie 3 and 5 the input vector is [0,0,1,0,1].

The goal is to predict the movie the user will rate in the future (between time T and T+delta T). The labels are vector of size N_movies, and if the user rate the movie 4 the labels vector is [0,0,0,1,0].

I'm currently trying to get some initial results based on Fully connected layers, but it seems that it cannot optimize the loss at all. The representation might be too sparse, but it seems that the neural network should be able to learn at least some features.

Is it possible to make this model work, is there any problem with the loss function, or the optimizer?

from __future__ import print_function
import tensorflow as tf
from tensorflow.contrib import rnn
import csv
import bisect
import glob
import re
import numpy as np
import random
import data
import config

cfg = config.Config()
graph_data = data.Graph_data(cfg)

X = tf.placeholder("float", [None, cfg.N_movies])
Y = tf.placeholder("float", [None, cfg.N_movies])

def Dense(x):
hidden_layer_1 = tf.layers.dense(inputs=x, units=500, activation=tf.nn.relu)
hidden_layer_2 = tf.layers.dense(inputs=hidden_layer_1, units=50, activation=tf.nn.relu)
output_layer = tf.layers.dense(inputs = hidden_layer_2, units= cfg.N_movies, activation=tf.nn.softmax)
return output_layer

logits = Dense(X)
cross_entropy = tf.reduce_sum(- Y * tf.log(logits), 1)
loss_op = tf.reduce_mean(cross_entropy)
optimizer = tf.train.GradientDescentOptimizer(learning_rate=cfg.learning_rate)
train_op = optimizer.minimize(loss_op)

init = tf.global_variables_initializer()

with tf.Session() as sess:
sess.run(init)

for step in range(1, cfg.training_steps+1):
batch_x, batch_y = graph_data.train_next_batch(cfg.batch_size)
sess.run(train_op, feed_dict={X: batch_x, Y: batch_y})
if step % cfg.display_step == 0 or step == 1:
loss = sess.run(loss_op, feed_dict={X: batch_x,
Y: batch_y})
print("loss = ",loss)


## 2 Answers

The model seems fine to me. Did you try to optimize the hyperparameters? You could use trial and error methods for optimiring the parameters such as,

• Learning rate
• Number of epochs
• Batch size etc,.

And instead of vanilla gradient descent optimizer, you could try other optimizers such as Adagrad, Adam or RMSProp etc,.

Best, Sangathamilan Ravichandran.

• Thanks for your suggestions, I tried the Optimizers, as well as many different Hyper parameters. The loss function never goes down. What surprises me however, is that the precision on the other hand is quite well optimized. Oct 19, 2018 at 12:16
• That has once happened to me. But for me the problem was I was using simple MSE(fluctuates mightly with noisy data) which then I changed to cross_entropy , which got it working. But I could see you already are using cross entropy. Strange. Oct 19, 2018 at 12:22

Given your data is sparse it may be worth trying a loss function for that type of representation.

As an example Keras has a SparseCategoricalCrossentropy class:

tf.keras.losses.SparseCategoricalCrossentropy(
from_logits=False, reduction="auto", name="sparse_categorical_crossentropy"
)


More information on this is available here.

Finally, it is always worth it to try running your model with mock data, just to be sure your computational graph is initiated as expected.