_strip_incomplete_words cuts complete words

[azziko]

Hello,

I saw this and wondered, wouldn't this cut complete words too? Assume, nothing is in the alignatt threshold, the hypothesis is:

['▁U', 'ser', '▁Inter', 'ac', 'tion', '.']

Then the whole Interaction is cut, even though it was not in the frame_threshold

simulstream/simulstream/server/speech_processors/base_streamatt.py

Line 232 in f6688fe

selected_tokens = self._strip_incomplete_words(selected_tokens)

If this is not intended, I would put it like this:

        # Truncate tokens up to the first invalid alignment (if any)
        if len(invalid_tok_ids) > 0:
            selected_tokens = selected_tokens[:invalid_tok_ids[0]]
            if self.word_level_postprocess:
                selected_tokens = self._strip_incomplete_words(selected_tokens)

But maybe that's intended for the models that output partials words

[sarapapi]

Hi, thanks for pointing it out. I'll make a PR to correct this.

[mgaido91]

I am not sure this is a bug to be honest. This is rather a design choice. It also depends on how BPEs are handled. In the past, when BPE would put a @@ after tokens that have a continuation (i.e. not followed by a space), rather than a _ in front of the leading BPE, you could know earlier when a word is over.

What we can do here is rely also on the punctuation, but I would consider as a different variant of the current method. We can do that, but I would not remove the previous behavior. Also, @azziko feel free to open PRs if you prefer to have your contributions acknoledged and be listed in the contributors of the repo.

[azziko]

I see, thank you

Then maybe it would make sense to inject the _strip_incomplete_words method as it's done with text_history_method? And provide some of the implementation for the stripping. Because now even with the punctuation handling _strip_incomplete_words still cuts complete words if punctuation is not emmited. I'm not sure if this is the intended behaviour to cut the last words even if they fall outside of the alignatt threshold.

@mgaido91 I thought I would open an issue first to have an anchor for the PR. I will implement the approach above today and open a PR into fix_strip_incomplete_words

[sarapapi]

@azziko Could you please make an example for the following one?

Because now even with the punctuation handling _strip_incomplete_words still cuts complete words if punctuation is not emmited.

A word, at the moment, is considered complete only if _ is emitted after it, as @mgaido91 was saying. Have you found a case in which this is not the case?

[mgaido91]

@mgaido91 I thought I would open an issue first to have an anchor for the PR. I will implement the approach above today and open a PR into #33

yes, sure, you did great. I just want to make sure that we can properly acknowledge your contributions :)

[azziko]

@sarapapi I see, thanks

Consider the following complete hypothesis:

output[0].tokens=['▁commun', 'ities', '▁accord', 'ing', '▁to', '▁cur', 'rent']

With the current implementation, the word "current" would be stripped from the output, even thought theoretically it's a complete word.

I just find it counterintuitive, since with this approach even 0 frame_threshold becomes > 0, even though I realize now that the only other way to know if it's a complete word is to expect what part of the hypothesis fell into the frame_threshold (Or none if a model allows for incomplete words output ?)

[sarapapi]

@sarapapi I see, thanks

Consider the following complete hypothesis:

output[0].tokens=['▁commun', 'ities', '▁accord', 'ing', '▁to', '▁cur', 'rent']

With the current implementation, the word "current" would be stripped from the output, even thought theoretically it's a complete word.

I just find it counterintuitive, since with this approach even 0 frame_threshold becomes > 0, even though I realize now that the only other way to know if it's a complete word is to expect what part of the hypothesis fell into the frame_threshold (Or none if a model allows for incomplete words output ?)

I see, but it is indeed an expected behavior, because the model might produce a punctuation or something else that is not separated by a space in the next space (i.e., ▁ with SentencePiece), and which should be emitted together with the word (i.e., current, while note current ,). Therefore, our definition of a complete word is a word separated by ▁, with the only exception of the first and last ones of the entire input stream (not within steps).

[azziko]

Understood, thank you for the clarification. Then no fix is needed?

[mgaido91]

Yes, I think here there is nothing to fix. We can think of ways to improve this, but those would be imprevements/new methods rather than a bug fix.