We need to implement the model. In the reference paper there are two architectures (NAR/AR), both of the same architecture. Some key notes:
- transformer
- 12 layers
- 16 attention heads
- embedding dimension of 1024
- feed forward layer with dimension of 4096
- dropout 0.1
For further details we'll need to consult some other implementation and the original paper
We need to implement the model. In the reference paper there are two architectures (NAR/AR), both of the same architecture. Some key notes:
For further details we'll need to consult some other implementation and the original paper