ASS renderer refresh rate seems too low

[BlackstormFirst]

Hello,
In first thanks a lot for this module. Its really apprecated.

I'm currently using your libass with Jellyfin AndroidTV master debug release + the integration of the libass from "nielsvanvelzen" (with your RC5 release) that was actually disabled for the 0.19.X branch due to the devellopement state of this feature.

Usually, your plugin work as well as possible with basic and complexe ASS subtitles with media3[1.8.0] and avoid to have a transcoded content.
A lot of features are working, but in some cases, there are still some troubles with complex ASS renderer.

For exemple, i have a subtitle with 4 or 5 sings witch are on the same time, on different position, and that should move (all at the same time) smoothly through the screen, but in that case, the signs just jump from a position to another one.

It's like the content were not refreshed one every frame during theres display times.

Another thing, the subtitles seems to be cleared a little too late, but this trouble maybe on the jellyfin side, or not.

If possible, could you take a look please ? and tell me if i am right or maybe if it's on the Jellyfin app side ?

Best regards

Edit: in addition the renderer mode is : OVERLAY

[peerless2012]

Can you provie the example resource or assets? and if there is a video capture for this issue it better.

[BlackstormFirst]

Hello, sorry for delay, i had no times this week and last week-end, but i will share you next days an example with, if possible, short samples.

For the second case, the subs appear a little time after the normal sync and end a little time after (other Jellyfin access types, android, web browser have a correct sync for the same media), it's not really important for dialogues, but when you have signs it's clearly visible no matter what you draw.

It's likely, the OVERLAY miss an offset (apparently as the same as the old LEGACY mode) but maybe the changes miss to provide the offset parameter on the Jellyfin side to solve this issue...

For the "re-implementation" of the piece of code on the Jellyfin side, I only restored the recently deleted parts and applied them to the current master release. (with an update to your rc05 libs) : jellyfin/jellyfin-androidtv@ef53924

Thanks for your job.
Best regards,

Edit: I am on a Nvidia Shield Pro 2019, if you have the same device, i can provide you the Jellyfin debug APK that i am using.

[peerless2012]

I think there are two reasons:

1. The render time not align video frame time, see Align subtitle output with video frame timestamps (overriding explicit subtitle timestamps)
2. The OVERLAY mode draw ass on runtime, if the ass is complex or the dialplay is too large (like 4K), the draw will cost more than one frame time, the render result will delay, see render.renderFrame.

You can log the render.renderFrame to see each draw cost time to see if it is what I think above.

[BlackstormFirst]

Hello, thanks for references, i will check it as soon as possible.

For information, in fact, with my shield, the original video is streamed in 1080p, then the Shield use a medium AI 4k upscale, then it display the final content on my 8K TV (still in 4K) => I will recheck without any AI treatements.

I will also check, how to add the log, to be able to check the cost on the device.

[BlackstormFirst]

Hello, just to refresh the post... i still have no time to completly debug this wonderfull feature ;) but i read the post on media3 github... as i can read, this is very similar to the issue i found.

In fact, what i can see... is the subtitle start and finish approximatly at the good timestamp, but it’s not accurate.
If i take an anime like ”No game No Life” with a very good fansubed ass... it’s really easy to see the issue.

To update the media post... ASS/SSA is nor a text mode nor a picture mode. It’s both.
Libass is a "fork" of the vsfilter plugin for linux/unix (if i am not wrong) and now used by a lot of other program like ffmepg.
It’s a text based for usual discussions, but picture for other things (effects, draw) because ass can draw image, content. So ASS should be managed standalone in a mixed mode. ASS was especaily developped and used by fansub team to provide multiple indicators on screen, with effects, it’s not only a speech traduction, it's also a multiple form of signs translations.
ASS permit a lot of modern effects (color, size, position, fade, move, blur, clip, and what i using each time... the draw feature, that permit to completly overwrite a part of the frame, using the ”m” attribute, and insert a full block or a disigned image, content)

Maybe i am wrong with the next comment, but it’s seems that if the content have an effect (fade in my exemple) the sub seems to be more accurate... but maybe it’s only due to the effect that cleanup quickly the end of the sentence...

Frame based seems to be the only good way to integrate the ass feature. Because the sub MUST be refresh on each frame to match the correct position and effects at the appropriate time. Otherwise the effects and subs are displayed and refresh randomly and it’s not the goal ( can be quickly observed with karaoke, that are not updated correctly)
Softwares, like you said, (AEGISUB, subedit, medusa...) correctly manage ASS 4+ but documentation about the protocole and concept seems to be limited. It’s more like a convention, but standard must exist, maybe on the vsfilter side...)

I really like to see this issue solved, i think it’s the most unliked part... memory consumption will be the second part... maybe media3 will manage ass in the future natively... hope.

=> no change without IA upscaling, but still no time to integer the log part for consumption, sry.
=> i will try to : add fade effect type fade(100,100) to check, and on another side, create a ass with a sentence refreshed on each frame to see if the comment on the media3 post is revelant.

[peerless2012]

You can test your media and ass on mobile device, which ass will render to 1080p (almos), you can see the difference to tv.

As I can see, render ass to 1080p is more quick than 4k.

If so, we can config and render ass to 1080, even the movie is 4k, it will reduce the render time and also reduce the memory usage.

[BlackstormFirst]

Hello, I tried with the latest release of Jellyfin (0.19.5) and the libass 0.3.0, if i remember correctly, that was a little better... but still not fully accurate... i don't know if something change between the latest RC and the final one but difference it's hardly less noticeable

Actually, i have another problem, Jellyfin move to media 1.9.0 and now i lost all fonts attachments from my mkv.

Subtitles are still displayed, but not with the correct font...

Before opening a media3 issue to ask if something change about mkv attachments, i will wait to have all the components (including lib_ass_media) on the same release (i am still using the packaged release, imported directly from my other android studio project).

Maybe this latest issue will be solved and i will be able to switch and try the 0.4.0-alphaX and make more tests (i should have more time these next days).

NB: about vsfilter, i said a mistake, the filter was already based on the libass, so it will not help for reference/documentation.

Have a nice day.

[peerless2012]

You can use Support Advanced SubStation Alpha direct play build to test.

It has the latest jellyfin code and libass version.

Also the new render type OVERLAY_TEXTURE will have a more good experience.

[peerless2012]

Also if you can run jellyfin or libass-android, can you log the subtitle view size? In your situation, I am not sure the render size is 1080p or 8k.

[BlackstormFirst]

Hello, i tried last night, to rollout to the revision just before the Jellyfin media3 1.9.0 update but the remote system crashed on the compilation and stay frozen. I will have a direct access this night, to be sure i am able to restore the attachments for the current official release.

On my Shield pro, 1080p content is scalled by the device to 4K and sent as a 4K resolution to the 8K TV without another scalling (limited by the shield hdmi output).

I will try to add the weight/size of view asap to determine the impact of each line/effect, or if your already know where and how to add this to the current Jellyfin android tv code... i am taker to save time ;)

I will try your new mod, OVERLAY_TEXTURE in the same time, i am actually trying the 0.4.0-alpha1 with the commun OVERLAY_CANVAS (the most compliant for SDR and HDR content) but i think my first issue is linked to the media3 display method. While subtitles are not refreshed on each frame the issue will persist. Start time and end time are not enough for accurate effects.

Edit: in addition, it’s like media3 refresh its subtitle check each 1second... but it should be something like each 40ms to be more consistent.

Edit2: Thanks for the official Jellyfin+libass post, i missed it... and manually remade and adapted the content to the current build. I don’t try to ask Niels to reintegrade this feature after the 0.19.0 release, but even in experimental mode its very welcomed to avoid a lot of server side transcodes. If i need to do a choice.. i will prefere to use the libass each time as possible even if i have some delay, or even actually without the correct fonts.

Edit3: For information, i checked, the lastest Niels' code, and i have approximatively the same thing on my side, except the method, where i am still using the OVERLAY_CANVAS, the new code use the OVERLAY_OPEN_GL, and the part for the new dev preference interface where my code is smaller (i will ajust that part this night to be fully compliant with the new ui/style)... so i think we are testing the same things and we should have the same experience.

From your experience, do you think the last Niels' code is complete ? No missing parts for extracted content due to a moved/updated part ?

I suppose the main extraction part is here :
AssSubtitleParserFactory assSubtitleParserFactory = new AssSubtitleParserFactory(assHandler);
ExtractorsFactory assExtractorsFactory = AssPlayerKt.withAssMkvSupport(extractorsFactory, assSubtitleParserFactory, assHandler);
DefaultMediaSourceFactory mediaSourceFactory = new DefaultMediaSourceFactory(dataSourceFactory, assExtractorsFactory);
mediaSourceFactory.setSubtitleParserFactory(assSubtitleParserFactory);
exoPlayerBuilder.setMediaSourceFactory(mediaSourceFactory);
exoPlayerBuilder.setRenderersFactory(new AssRenderersFactory(assHandler, defaultRendererFactory));

On you side, did your mkv_exctractor/ass_extractor or parser logs say something for "mkv+ass+media3 1.9.0" on playback ?
Do you still have fonts attachements on your tests with media 1.9.0 ?

Thanks

[peerless2012]

If you are not easy to build libass, you can replace the local dependency in ass-media to maven central dependency.

[peerless2012]

Also AssHanlder will low down the ass render frame rate, see videoTime.

If you want to see high refresh rate, you can change the videoFramePeriod or remove the logic.

[BlackstormFirst]

I was able to restore a working app with attachments (Jellyfin Android TV, latest master + Libass 0.4.0-alpha01).
But:
=> If media3 1.9.0 is selected on Jellyfin side, attachments seems break, but subtittles are still displayed (maybe because libass-android is still on 1.8.0).
=> if the mkv don't have the fonts built-in, but your server have, the fonts are not OK.
=> if you have "merged" your fonts with a tools like mkvpropedit or FontsCollector, the fonts are not correctly displayed, but if you use mkvtoolnix-gui and just rebuild the same file, the fonts are correctly displayed. (=> maybe linked to this PR#24)

In addition, on my Android Studio, i never see the good fonts, but attachments working well on my Shield TV.

[peerless2012]

The library only process fonts in mkv, and only work with fonts at beging of the mkv.