Goal
One of the latest/best regularisation techniques for training RBMs is dropout. Unfortunately, the original Boltzmann.jl package does not implement this technique, so we should undertake this ourselves.
Technique
During the training phase of the RBM, each hidden node is present with only probably $p$. Training is performed for this reduced model and then the resulting trained models are combined. The pertinent section from (Srivasta 2014) reads,
8.2 Learning Dropout RBMs
Learning algorithms developed for RBMs such as Contrastive Divergence (Hinton et al., 2006) can be directly applied for learning Dropout RBMs. The only difference is that r is first sampled and only the hidden units that are retained are used for training. Similar to dropout neural networks, a different r is sampled for each training case in every minibatch. In our experiments, we use CD-1 for training dropout RBMs.
References
Srivasta et al, "Dropout: A simple way to prevent neural networks from overfitting," JMLR, vol. 15, 2014, pp. 1929-1958.
Goal
One of the latest/best regularisation techniques for training RBMs is dropout. Unfortunately, the original Boltzmann.jl package does not implement this technique, so we should undertake this ourselves.
Technique
During the training phase of the RBM, each hidden node is present with only probably$p$ . Training is performed for this reduced model and then the resulting trained models are combined. The pertinent section from (Srivasta 2014) reads,
References
Srivasta et al, "Dropout: A simple way to prevent neural networks from overfitting," JMLR, vol. 15, 2014, pp. 1929-1958.