# Deep reinforcement learning on changing data sizes

I have a game that I want to build a model that will learn to play the game. Yet, the environment output is two lists that represent the location and number of soldiers of the user and Opponent.

The lists lengths are changing with each step, so to know the input size to the network is a problem.

How can this be dealt with?

• since your environment changes maybe it's better to use RNN and feed them deltas? – quester Oct 12 at 19:26

Although this problem can happen quite frequently in Reinforcement Learning (RL) projects, it is not a RL issue. The issue is due to neural network requirements, and also applies to many other parametrised machine learning solutions.

You can vary the number of parameters of a neural network in a limited fashion. However, it can only usually be done before training (or re-training), and is not a suitable approach to deal with changes happening over a small number of time steps, or during post-training prediction (e.g. when playing the game with the trained agent).

There are a few different approaches you can take. Which is suitable will depend on details of the game:

This is the simplest approach, and you can do this without needing much additional research. It relies on there being a workable maximum number of inputs you could need at any time step.

Practical considerations:

• Have a simple mechanism to identify whether a particular set of inputs are present. E.g. add a $${0,1}$$ flag to show present of a unit, leaving rest of inputs that represent the unit at some neutral value such as $$0$$.

• If there are only rare occasions when the last few inputs are used, you should expect the neural network to learn about those situations very slowly. It will not generalise between units in different input "slots", unless you alter the architecture to link them.

### Invert the representation from list of units to "world map"

This works nicely for a game where movement is discrete and there are rules that limit the amount of stacking possible in any single space. It also allows you to use a CNN architecture.

For example, this is how AlphaZero represents Go and Chess, which both allow varied number of pieces on the board at any one time, but where the board itself is strictly defined. In the case of Chess, this is also how AlphaZero identifies which piece to move, and how, by output on a matching grid to select the piece and a movement type (each selection/movement type combination in that case is a separate output channel, the output reprsents 1000s of potential moves all at once).

Practical considerations:

• If your units have statistics that you need to track, then each of these can go in a separate instance of the grid (sometimes called a "plane" or a "channel"). So you might have one grid showing presence of a piece, another identifying its type, another its attach strength etc, all stacked into a 3D structure assuming the playing board is 2D.

• If the map is very large or continuous, then it may be possible to discretise it into a grid.

• Using a CNN allows for good generalisation based on similarity between pieces and their stats. However, the approach becomes limited if the grid is very large or stacks of pieces at any one grid position can become large, in addition to requiring a much larger memory footprint to represent the game state.

### Feature engineering

Technically, representing the game as a map/board is just one common form of feature engineering in RL. Whether or not it will work, you know the game and may have knowledge of what factors are important to playing it well. You can use this knowledge to simplify the rerpesentation and summarise key details that are independent of the number of units.

For instance, it may be important to prioritise play based on proximity of strongest units in play. So you could sort the available units based on this heuristic and present a smaller fixed number of key units.

Practical considerations:

• This can be done in combination with any other approach.

• This relies heavily on your creativity and a good technical understanding of both the game and what works well in features used for neural networks.

If you want advice on possible feature engineering approaches, I suggest you open a new question with more details of the game being played.

### Recurrent Neural Networks (RNNs)

A recurrent neural network architecture, such as LSTM or GRU, can work directly with mixed length sequences. This can work really well when each item in the sequence is the same kind of object but with different feature values.

Practical considerations:

• RNNs are harder to understand and train than fully connected feed forward networks or CNNs.

• If the two lists of units are just one part of the game representation, you will need a hybrid system that combines a RNN architecture with other neural network designs.