# How to predict tf-Idf vector with Neural Network?

I designed a neural network in Torch for prediction of articles. I want to predict, which article will be read by user based on your reading history.

Input to the network is tf-idf vector of article and another information about user. Output from the network should be tf-idf vector of next article, which user read.

Tf-idf vector is normalized by SoftMax function.

I know the best solution would be, if a output was a representation of classes but there are a lot of articles and by that there will be a lot of classes. So I represent articles with tf-idf vector on the output.

I tried train with DistKLDivCriterion ( Kullback–Leibler divergence ) but it didn't converge. How to design a network for prediction tf-idf vectors? Which criterion should I use?

tmodel = nn.Sequential()

criterion = nn.SequencerCriterion(nn.DistKLDivCriterion())


I hope you do understand, that trying to predict some abstract TF-IDF vector based on read article you are trying to teach a neural network to produce a new article for a given user. It would be a major step in artificial intelligence if achieved, so keep us posted on your progress.

However, if you would think of alternative approaches, I would suggest following steps to make your neural network to work:

1. You need a classification/clustering model for your articles (which can be based on your TF*IDF, or on some tags assigned to articles). The more classes you could squeeze from your model, the better would your resulting recommender be. On that step, you should be at least able to provide user with articles similar by topic.
2. You will need at least two-hop data set of user reading current article and hopping to the next one which would be used as a training set for actual predictor. Here, you will train a neural network (or any other model) which will try to predict a next-article class (from step one) based on current article - that would be a recommender system.

Of course, you may, given you have data from (2), train a neural network that would answer the question of "If user reads article A, will the next article contain word B", but that would require you to have the size of output variables of your TF*IDF matrix (for any reasonable dataset that would be at least 20-30k words unless you work with books for toddlers) which makes your model overcomplicated (that would be, actually, a TF*IDF matrix size number of models for every word).

You might want to check out algorithm from recommendation system perspective. It would be more appropriate to use something like matrix factorization (assuming you have more than one user in your dataset), rather than neutral network. TF-IDF is often a very fat matrix, with a lot more columns than rows, so your target variable is a very long vector. Your model eventually suffers from the curse of dimensionality due to lack of training example.

Even if you have enough training examples to train the model, the output is the TF-IDF of some document. So you would still have a non-trivial problem of matching this TF-IDF matrix with some document in your corpus.

You should not use neural networks for this kind of problem. Neural network works well for classification problems. Here, your problem is somewhat like recommender system problem. You should search some other algorithm for this problem.