Expressive leaky memory neuron (`ELMCell`)

[MartinuzziFrancesco]

Paper

https://arxiv.org/abs/2306.16922

Equations

$$\begin{aligned} \boldsymbol{\kappa}_m(t) &= \exp\left(-\frac{\Delta t}{\tau_m}\right), \\\ \boldsymbol{\kappa}_s(t) &= \exp\left(-\frac{\Delta t}{\tau_s}\right), \\\ \mathbf{s}(t) &= \boldsymbol{\kappa}_s(t) \circ \mathbf{s}(t-1) + \mathbf{w}_s \circ \mathbf{x}(t), \\\ \Delta \mathbf{m}(t) &= \tanh\left( \mathrm{MLP}_{\mathbf{w}_p}\!\left[ \mathbf{s}(t),\; \boldsymbol{\kappa}_m(t) \circ \mathbf{m}(t-1) \right] \right), \\\ \mathbf{m}(t) &= \boldsymbol{\kappa}_m(t) \circ \mathbf{m}(t-1) + \lambda \left(1 - \boldsymbol{\kappa}_m(t)\right) \circ \Delta \mathbf{m}(t), \\\ \mathbf{y}(t) &= \mathbf{w}_y \cdot \mathbf{m}(t). \end{aligned}$$

Official implementation

https://github.com/AaronSpieler/elmneuron (Jax)