Skip to content

Added Live Preview via overlays#20

Open
rmdocherty wants to merge 13 commits into
MIC-DKFZ:mainfrom
tldr-group:live_preview
Open

Added Live Preview via overlays#20
rmdocherty wants to merge 13 commits into
MIC-DKFZ:mainfrom
tldr-group:live_preview

Conversation

@rmdocherty

Copy link
Copy Markdown

Hi @Karol-G,

I was working on some Napari-SAM related stuff and got a 'live-view' working (where it prompts SAM at the cursor to show the label about to be placed).
napari_sam_live_preview

Main changes:

  1. Live preview - this uses a custom Napari overlay with associated custom vispy overlay that displays an ImageNode that is attached to the Image Layer. When the cursor is moved and the time between the last prediction is large enough (to avoid over-pinging) SAM is prompted with the current position of the mouse cursor and previous points. The returned mask is then converted to an image, cropped to a bounding box around the positive part of the mask (to avoid redrawing an image the size of the canvas to screen all the time) and the ImageNode's data updated. This live preview also works in bounding box mode.
  2. Bounding box selector is now an overlay using one of the default overlays, the 'selection_box' overlay. The width and colour are set as before. However, this means the bounding boxes no longer persist after a new one is added. This means one of the 'ignore_this_layer' has been removed - there is the potential to remove the second but that would probably require another custom points overlay.

Issues:

  1. The biggest is that this PR uses private attribute access to add the custom overlays to the image layer - this gives a deprecation warning for version 5.0.0. I'm assuming (hoping) that there will still be a way to add/change overlays after that update. The overlays API is probably unstable and may change - I've stuck with fairly standard stuff that is used elsewhere though.
  2. It doesn't work with 3D images - unsure how x_coord parameter really works so have left it out.
  3. The initial transformation copying in _custom_overlays is very hacky - I'm not too sure on the whole rendering pipeline but this seems to work.
  4. I had a weird bug with the 'edge_color' argument of add_points so removed it - this might just be my napari version.

@Karol-G

Karol-G commented Mar 18, 2024

Copy link
Copy Markdown
Collaborator

Hey @rmdocherty,

This looks really cool! However, I have to give you sadly the same response as on another pull request -> I am currently in a crunch time phase and won't be able to review this PR for a couple of weeks, as other projects have a higher priority right now. I will definitely take a look at it though and will set a reminder so I don't forget. I hope this is okay with you :)

On a side note, the image looks like a CT mineral particle sample? Is that correct? Coincidently, I created an automatic segmentation method for these type of images called ParticleSeg3D. Maybe this is interesting to you :)

GitHub: (https://github.com/MIC-DKFZ/ParticleSeg3D)
Paper (Arxiv version): https://arxiv.org/abs/2301.13319

Best regards,
Karol

@Karol-G Karol-G self-assigned this Mar 20, 2024
@rmdocherty

Copy link
Copy Markdown
Author

No worries! The image is a CT of catalyst particles (though I mainly do batteries), so similar to mineral samples - the framework looks interesting and I will try it out at some point

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants