How to structure and train an LSTM for real-time predictions in Keras?

Question

I am trying to create an AI to play a 3D melee fighting game using an LSTM. The NN receives as input the relative positions of the enemies' joints (e.g. head, hands, legs) and should output the correct movements for each timestep. Currently, I am using a vanilla feed-forward NN made with Keras and it performs well, but it needs information from multiple timesteps to be more effective.

I am now trying to implement an LSTM for this problem. I want to receive an output at each timestep rather than feeding in a whole sequence of data before receiving output.

I believe that a stateful LSTM is suited for this task, but I am unsure of how to train it and when to reset the state. Ideally, the AI would have no fixed number of timesteps of memory and would just be able to continuously look back as it's making predictions in real time for each timestep, without having its state reset abruptly during fights.

My training data is one long, continuous sequence of enemy positions (input) and keystrokes (output) for each timestep. How would I structure and train an LSTM to learn from this data to make real time predictions at each timestep using Keras?

Answer 1

The tough part about this problem is evaluating what moves are "correct" per se. In a fighting game sequence, there may be 2 or more moves which both work in theory in the current frame. If you are optimizing for knocking the other character out of frame, it would be useful to build a reward set which optimizes for this.

It may be worthwhile to build a data set that takes into account corresponding stick positional data and then labeling past and future time locations as advantageous/disadvantageous based on simulation, similar to a chess computer.

First you have to build something that can identify perfect combos- situations in which the enemy, regardless of their inputs, has no reliable way to escape the combo which results in high % or death. And then the problem becomes easier because the ai can just identify positions to set up this combo, and then executes them.

Train a cpu to combo via reinforcement, and then integrate with the identifier, and you'll have a pretty damn good bot.