I'm having some trouble describing in one line what I want, which is probably why I haven't had much luck with Google.

Say I have a game like 2048 where the possible actions each step are fixed (and more than two). I want to train a neural network that chooses a move, so I have 4 neurons in output layer and I make the move with the highest prediction. The output vector is normalized (softmax layer).

However, the training data I have is just the state, the move that was made, and whether that had good or bad results. If the chosen move is bad, I don't know which of the other ones was better (if any).

How should I train this? My current thought is like this:

  • Good move? -> chosen action gets positive error (so prediction goes up)
  • Bad move? -> chosen step gets negative error (so prediction goes down)

But I haven't found literature supporting this guess. There are alternatives:

  1. Maybe I should also update the options that weren't chosen (in the opposite direction)?
  2. Is it a good idea to set error directly instead of using goal predictions?
  3. The error for correct and incorrect could be different, maybe to preserve normalization?
  4. ...

(I'm doing 2048 and using neural networks, but I think this is not limited to this game or this method.)

  • $\begingroup$ "...the training data I have is just for the move that was made, and whether that had good or bad results." So you don't have the state of the game? You can't train your network with just the move and its effect on the error. You have to connect those moves to the game. $\endgroup$
    – AN6U5
    Commented Jul 27, 2015 at 5:58
  • $\begingroup$ Ow yeah, sorry, of course I have that. But I don't have results for any other possible moves. $\endgroup$
    – Mark
    Commented Jul 27, 2015 at 19:25
  • 2
    $\begingroup$ "Reinforcement Learning" might be a good topic to Google whilst waiting for an answer, although there are a lot of variants that won't apply to your specific game. Also, for a small network, you could look into combining neural networks with genetic algorithms to search for good NN weights as opposed to backpropagation of error terms (where precise error values may not be known, or are significantly delayed) $\endgroup$ Commented Jul 27, 2015 at 19:49
  • $\begingroup$ Good that you have the state. 2048 has 1. initial game state, 2. move then resulting game state, 3. resulting game state + stochastic addition of extra tiles. If you are just connecting 1 to 2, then this piece is deterministic. Why use a learning algorithm at all? Really you want to use the learning algo for what is the best state before adding random 2 tiles. This seems better because there is invarriance in the system under rotation, so left, right, up, down loose meaning for different game states. e.g. don't have your ANN choose between 4 moves, have it choose between 4 final states. $\endgroup$
    – AN6U5
    Commented Jul 27, 2015 at 21:52
  • 1
    $\begingroup$ @Mark Please, read this answer on StackOverFlow stackoverflow.com/a/22498940/2309097...You will love it :) $\endgroup$
    – Tasos
    Commented Jul 28, 2015 at 21:22

1 Answer 1


One way to frame your problem is with reinforcement learning (RL). Reinforcement learning (RL) train an agent to accomplish a goal in environment. In your case, the environment is 2048, the goal is to solve the game, and the agent is the model you are training.

If the chosen move is bad, I don't know which of the other ones was better.

That trade-off is frequently called explore-exploit. Does an agent do the move it predicts is the best possible (exploit)? Or does the agent look for better possible moves (explore)?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.