3
$\begingroup$

Is there a CONSTANT limit in DNC external memory usage? Like a human brain, 7 chunks of information? Or is it some sort of a hyperparameter? How much data it can associate using that Short-Term memory?

$\endgroup$
1
$\begingroup$

If the reference to 7 chunks of information for humans is a reference to the 'Magical Number Seven' Paper, which is the human working memory, then:

Consider the high level operation of DNC at a timestep. An input comes in, and is processed by an LSTM to generate a 'Interface vector' which interacts with memory with Write, Erase and Read operations. The operations are then executed, and a final layer produces an output based on the retrieved information and the input.

Where is the analogy to human 'working memory'?

1). If you consider the working memory is what is retrieved on read, then there is a hyperparameter for number of memory reads. This applies generally to Write and Erase as well as those are all hyperparameters which determine the size of the Interface Vector. In the paper they used a write size of 1 in all cases for purposes of controlling the experiments. Note that the 'size' of what is read/written in a single operation is a separate parameter, which just depends on how the problem is encoded (see the methods section of the paper).

2). If you consider working memory as external memory, then this is another hyperparameter. I think it is a conceptual mistake to view the external memory as short term memory, as the addition of this and all the gating was specifically developed to store information long term and then recall when needed.

3). There is a part of the DNC's operation which queries a link matrix (storing what was written to long term memory and when) during the read phase. This can be seen as being related to human working memory, as numerous behavioral economics studies showed that what we recall and how we process it is effected by what we have just seen. This is the only part in the paper where the computation is O(N^2) w.r.t to the size of the memory, (though they approximate with an O(NlogN) method). This portion of retrieval/bias of what is retrieved to most recently accessed information can then be said to have a limit, since for large memories it will not scale with rest of network, but this is still not constant.

4). Probably the best candidate for the working memory analogy IMO is the hidden state of the controller LSTM, since this part of the algorithm is responsible for storing what is needed over time for retrieval, and on its own is used for storing a state over time. The reason that this is short term memory in this analogy is just that the external memory is an explicit external memory, and this is the only other place where a state is maintained.

So in short no, there is no Constant, as just about every conceivable analogy to 'working memory' and the entire architecture is controlled by parameters. It is worth checking out the code and paper for more details on exactly how the parameters are set for different experiments.

Let me know if this makes sense.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.