Implementing Neural Turing Machines


Abstract: Neural Turing Machines (NTMs) are an instance of Memory Augmented Neural
Networks, a new class of recurrent neural networks which decouple computation
from memory by introducing an external memory unit. NTMs have demonstrated
superior performance over Long Short-Term Memory Cells in several sequence
learning tasks. A number of open source implementations of NTMs exist but are
unstable during training and/or fail to replicate the reported performance of
NTMs. This paper presents the details of our successful implementation of a
NTM. Our implementation learns to solve three sequential learning tasks from
the original NTM paper. We find that the choice of memory contents
initialization scheme is crucial in successfully implementing a NTM. Networks
with memory contents initialized to small constant values converge on average 2
times faster than the next best memory contents initialization scheme.

Submission history

From: Mark Collier [
view email]

[v1] Mon, 23 Jul 2018 10:35:18 GMT (703kb,D)

[v2] Wed, 25 Jul 2018 11:48:54 GMT (703kb,D)

Read More

This site uses Akismet to reduce spam. Learn how your comment data is processed.