k,v caching implementation in streamAtt and alignAtt

[coder001-rgb]

❓ Questions and Help

Before asking:

1. search the issues.
2. search the docs.

What is your question?

If we are using seq2seq model, and use streamAtt or AlignAtt then in streaming mode (audio coming in chunked fashion ) can we use k, v caching?

If yes do you think the cached values remain valid when a new audio chunk is added / concatted to previous accumulated audio?

If no any optimization suggestion to improve inference speed?

Code

What have you tried?

What's your environment?

• fairseq Version (e.g., 1.0 or master):
• PyTorch Version (e.g., 1.0)
• OS (e.g., Linux):
• How you installed fairseq (pip, source):
• Build command you used (if compiling from source):
• Python version:
• CUDA/cuDNN version:
• GPU models and configuration:
• Any other relevant information:

[mgaido91]

Dear @coder001-rgb, I'd say in general k, v caches would not be safe in general, as the encoder output may change (due to the new audio) and also the representations of previously generated tokens (after the first layer) can therefore change. The question may become, then, if these changes are big enough to need the full recomputation and it becomes a trade-off between speed and quality, which may also depend on the underlying model. Does this answer your question?

[coder001-rgb]

Yes it is clear..

Thanks for the quick response