# Using Neural Networks To Predict Sets

I'm building a neural network for data analysis, however I'm stuck on how many output neurons I need and what they should represent. The neural network tries to predict peoples choices for certain objects. There are 75 possible objects, but here's the catch, they choose 6 of them at a time. These six can (theoretically, though this won't happen realistically) be any combination of the 75 objects, in rare cases there might even be duplicates of a certain object in this set of six.

I naturally considered creating an output neuron for each possible set, but since that would lead to 75^6th output neurons (or more if we consider duplicates) I imagine that that would have disadvantages for the learning speed. Another option I considered is just taking the six highest ratest items as a set, but I'm unsure if this would work since the choice of the second, third, etc. objects depend on which earlier objects were chosen. I wonder if there are any better, faster or more accurate ways of doing this?

Edit, some more information about the trainingdata: The neural network will be used to analyse the ingame item choices in the computergame called league of legends. There players can choose up to six items each game. These six items are most likely based on some factors that change each game such as the ingame characters they and their enemies play, but also things such as what the other 5 items are they have. However, the reason for this may be universal to all games. For instance, the (likely) biggest reason why items depend on eachother is the multiplicativity of item stats, if you already have an item that makes your attacks hit harder then getting an item that allows you to attack more often is going to do more for you than getting an item that makes your attacks hit even harder. There are also some links between item choices though, while probably more limited, there is for instance an item that returns damage to the enemy when he attacks you, while it's not detrimental it is probably not a great idea to combine that with an item that lowers the amount of attacks that an enemy can do to you.

• I think the update to the question makes my answer less usable, although you should maybe try it. If choice of items is flexible enough and driven mostly by the character stats or other data you have, then it may still work. The alternative might be to treat the item selection as a sequence and use RNN, but unless you have label data that actually shows 1st, 2nd 3rd items chosen, then this could be applying too much structure to the problem. It does make your problem interesting, I'm not aware of any classifiers that deal with this kind of structure. – Neil Slater Jun 5 '16 at 6:26
• Out of interest, is there a natural way to sequence item decisions, e.g. is there a in-game cost to each item so they could be ranked, or do they take "slots" with big decisions such as weapon type or armour type needing to be covered first? – Neil Slater Jun 5 '16 at 6:42
• I would have to collect some extra data, but it would be possible to determine in which order the items were bought. As for your second comment, I'm unsure what you're asking. Item's have an ingame cost in gold and are bought over time, you usually build the most important items first for this reason. – Hasse Iona Jun 5 '16 at 10:35
• I was just wondering if there was some other way to determine a sequence of choices based purely on item stats. The stricter that items can be sequenced in a way that matches how decisions are made in the game, the better match this could be to a sequence-predicting network like an RNN. RNNs are complex though, so maybe first try the simplest per-item classifier and see if it is good enough for your needs. – Neil Slater Jun 5 '16 at 16:39
• I don't think so, it's not like you always build defence before offence or something like that. I'll definitely give your suggestions a try! – Hasse Iona Jun 7 '16 at 11:38

The 75^6 option is not only bad for speed, but it is a very difficult representation to train, because the NN doesn't "understand" that any of the output categories are related. You would need an immense amount of data to train such a network, because ideally you need at least a few examples in any category that you expect the network to predict. Unless you had literally billions of examples to train from, the chances are certain combinations will never occur in your training set, thus could never be predicted with any confidence.

Therefore I would probably use 75 outputs, one for each object representing the probability that it would be chosen. This is easy to create training data for, if you have training examples with the 6 favoured objects - just a 1 for the objects chosen and 0 for all others as a 75-wide label.

For prediction, select the 6 objects with the highest probabilities. If these choices are part of a recommender system (i.e. may be presented to same person as being predicted for), then you can select items randomly using the outputs as weights. You may even find that this weighted Monte Carlo selection works well for predicting bulk user behaviour as well (e.g. for predictions fed into stock purchases). In addition, this stochastic approach can be made to predict duplicates (but not accurately, except perhaps averaged over many predictions).

A sigmoid transfer function on the output layer is good for representing non-exclusive probability. The logloss objective function can be used to generate the error values and train the network.

If you want to accurately predict duplicate choices out of the 6 items chosen, then you will need plenty of examples where duplicates happened and have some way to represent that in the output layer. For example, you could have double the number of output neurons, with two assigned to each object. The first probability would then be probability of selecting the item once, and the second probability would be for selecting it twice.

The question has since been updated, and it appears there are strong relationships between items making the choice of a set of items potentially very recipe-like. That may reduce the effectiveness of the ideas outlined above in this answer.

However, using 75 outputs may still work better than other approaches, and is maybe the simplest setup, so I suggest still giving it a try, even if just to establish a benchmark for other ideas. This will work best when decisions are driven heavily by the feature data available, and when in practice there are lots of valid choices for combining items so there is a strong element of player preference. It will work less well if there is a large element of game mastery and logic in player decisions in order to combine items.

I don't think you can split the combinations and use it for prediction. E.g: I like fish n chips as my first choice but neither fish nor chips might be in my top 10 choices.

A better idea is to use what's called "support". Do some preprocessing to find out which of the $(75)^6$ combinations occur at least say 100 times in your dataset. Only use these combinations for training and prediction. You want to identify generalizable patterns in people's choices not events which might have occurred just by chance.

• I hadn't thought of this, it is a good point, and the food example highlights it. This depends very much on what the set, that the OP is trying to predict, represents. In my answer I am assuming something like "set of favourite items", which are independent enough that you won't get the recipe problem. But if the sets are recipe-like, then your answer would be closer to the mark. – Neil Slater May 29 '16 at 8:42
• Only the OP can say what they're thinking about ! I agree that your answer applies if the individual objects were to be ranked for each individual – wabbit May 30 '16 at 14:37
• Sorry for the late reply! I updated my original post with the answer to your questions. I think a lot of the 'recipyness' comes from universal rules about not taking too much of the same stat in all your items, would this be something a neural network would be able to learn? Even so, there is likely a part of the 'recipyness' that comes from certain combinations not working as well and I'm not sure how small or substantial that is. – Hasse Iona Jun 4 '16 at 19:31
• @HrishikeshGanu thanks for your suggestion, I will start by trying out Neil Slater's suggestion, just because that would seem like a more convenient option IF it works for my dataset, but if it doesn't then I like your idea and I agree that chance occurences are indeed not interesting and it may even be favorable in this case to filter them out (because angry players might intentionally build something bad just because they want to destroy the game for other players) – Hasse Iona Jun 7 '16 at 11:41