3
$\begingroup$

I'm having some trouble describing in one line what I want, which is probably why I haven't had much luck with Google.

Say I have a game like 2048 where the possible actions each step are fixed (and more than two). I want to train a neural network that chooses a move, so I have 4 neurons in output layer and I make the move with the highest prediction. The output vector is normalized (softmax layer).

However, the training data I have is just the state, the move that was made, and whether that had good or bad results. If the chosen move is bad, I don't know which of the other ones was better (if any).

How should I train this? My current thought is like this:

  • Good move? -> chosen action gets positive error (so prediction goes up)
  • Bad move? -> chosen step gets negative error (so prediction goes down)

But I haven't found literature supporting this guess. There are alternatives:

  1. Maybe I should also update the options that weren't chosen (in the opposite direction)?
  2. Is it a good idea to set error directly instead of using goal predictions?
  3. The error for correct and incorrect could be different, maybe to preserve normalization?
  4. ...

(I'm doing 2048 and using neural networks, but I think this is not limited to this game or this method.)

$\endgroup$
10
  • $\begingroup$ "...the training data I have is just for the move that was made, and whether that had good or bad results." So you don't have the state of the game? You can't train your network with just the move and its effect on the error. You have to connect those moves to the game. $\endgroup$
    – AN6U5
    Jul 27 '15 at 5:58
  • $\begingroup$ Ow yeah, sorry, of course I have that. But I don't have results for any other possible moves. $\endgroup$
    – Mark
    Jul 27 '15 at 19:25
  • 2
    $\begingroup$ "Reinforcement Learning" might be a good topic to Google whilst waiting for an answer, although there are a lot of variants that won't apply to your specific game. Also, for a small network, you could look into combining neural networks with genetic algorithms to search for good NN weights as opposed to backpropagation of error terms (where precise error values may not be known, or are significantly delayed) $\endgroup$ Jul 27 '15 at 19:49
  • $\begingroup$ Good that you have the state. 2048 has 1. initial game state, 2. move then resulting game state, 3. resulting game state + stochastic addition of extra tiles. If you are just connecting 1 to 2, then this piece is deterministic. Why use a learning algorithm at all? Really you want to use the learning algo for what is the best state before adding random 2 tiles. This seems better because there is invarriance in the system under rotation, so left, right, up, down loose meaning for different game states. e.g. don't have your ANN choose between 4 moves, have it choose between 4 final states. $\endgroup$
    – AN6U5
    Jul 27 '15 at 21:52
  • 1
    $\begingroup$ @Mark Please, read this answer on StackOverFlow stackoverflow.com/a/22498940/2309097...You will love it :) $\endgroup$
    – Tasos
    Jul 28 '15 at 21:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.