clip generator with mutlithreading progress visualization support

[linzuzeng]

Currently we show percentage of file reading in clip generators, but it is super noisy and doesn't make much sense when used together with muli-threading.

I would suggest to have a block diagram to visualize the analysis progress. I have an example from AiraNg

We just need to send the starting recordid and the ending record id of a file to the Web GUI, and ask Web GUI to put green on to the corresponding block, indicating that block of file is read already.

[linzuzeng]

We could have a special logger for progress visulization, or add some javascript to parse the progress in the GUI side.
https://github.com/timetag/ETA/blob/master/etabackend/clip.py#L162

[gyger]

There is a blog entry that has a similar structure: https://towardsdatascience.com/stunning-tables-using-bokeh-and-svg-e612a6f1c541

[linzuzeng]

COOL!
Can bokeh render to a given div or we use iframe with a url pointing to a bokeh dashboard?

[Cyberfly100]

https://docs.bokeh.org/en/latest/docs/user_guide/embed.html It looks like it can target a specific <div>

…

[linzuzeng]

Great!
Now I realize maybe it's better to render the progress on the display panel (which is already bokeh) instead of embedding it on the main GUI.
It will allow users to see the progress on file with the the growing histograms on the same screen during an analysis. This may be handy for diagnosing some problem.
Also this may save one server for bokeh and probably prevents the headaches of running two bokeh dashboard simultaneously :)

[linzuzeng]

The only drawback is that the progress is not enabled by default for every analysis and the user need to manually write progress bar rendering in the display panel Python script.
But I think it's fine as the users don't always need progress bar (progress bar for realtime is meaningless ; progress bars for batch process in parallel requires extra rendering and typesetting ) and we can put something in eta.tk to minimize users' efforts.

[Cyberfly100]

Yes, this sounds like a good idea to me. We can include the toolkitted progress bar in the standard recipes, where applicable. Am Sa., 20. März 2021 um 03:30 Uhr schrieb Zuzeng Lin < ***@***.***>:

…

The only drawback is that the progress is not enabled by default for every analysis and the user need to manually write progress bar rendering in the display panel Python script. But I think it's fine as the users don't always need progress bar (progress bar for realtime is meaningless ; progress bars for batch process in parallel requires extra rendering and typesetting ) and we can put something in eta.tk to minimize users' efforts. — You are receiving this because you were assigned. Reply to this email directly, view it on GitHub <#134 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AGVYG7WWYTID5T3R6FE35Y3TEQCCPANCNFSM4LGGGPTA> .