Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Right click no-hold painting, ctrl+left click fast delete #54

Open
wants to merge 7 commits into
base: main
Choose a base branch
from

Conversation

kephale
Copy link

@kephale kephale commented Aug 15, 2024

@carsen-stringer ok well I guess I wanted to procrastinate on making my actual presentation material 😁 . This is testable, but I don't think it should be merged until the trigger for disabling the CellPose painting mode.

This adds:

  1. Right click to start creating a path with points, such that when the cursor returns to the initial point the path is closed and filled in the Annotations layer.

  2. (Fast delete) Ctrl + left click to delete the label under the cursor

A problem:

I'm still missing a proper connection to disable this mode once the widget is closed. This will just take some fiddling.

cellpose_hoverpaint_v03.mp4

@kephale kephale marked this pull request as ready for review September 5, 2024 17:43
@kephale
Copy link
Author

kephale commented Sep 5, 2024

@carsen-stringer points out that the initial point needs to be a large open point to work as an end zone. currently the initial point starts large and is getting reset once other points are added.

@kephale
Copy link
Author

kephale commented Sep 5, 2024

painting_with_end_zone.mp4

now with working end zone rendering.

@melonora
Copy link

Hey sorry for jumping in here. This looks very much like the polygon lasso tool in the shapes layer. Do I understand correctly it is being implemented here but for the labels layer? If there is a current request I might be able to find some time to duplicate the behaviour to the labels layer in napari core itself

@kephale
Copy link
Author

kephale commented Sep 10, 2024

@melonora yeah, there is definitely a lot of overlap w/ the polygon lasso. If you can get a labels lasso merged then we would certainly try to reuse as much of it as possible. There are at least 2 nuances about why it would still at least have a wrapper here:

  • the mouse bindings here are optimized for annotator efficiency
  • an visible "endzone" for completing the region

However, I'm not sure I would block this merge for the upstream change. We can always adjust this code accordingly.

@melonora
Copy link

I wouldn't block but will make a mental note

@kephale
Copy link
Author

kephale commented Sep 10, 2024

Awesome, thank you @melonora !

@melonora
Copy link

Just one question, given that the polygon lasso tool does not render the face of the polygon, is an end zone required?

@kephale
Copy link
Author

kephale commented Sep 10, 2024

@carsen-stringer can chime in, but for this PR we implemented the annotation mode that was currently optimized based on annotator experience for the cellpose standalone tool.

I still expect that the napari feature won't match this exactly (at least for keybinds and probably endzone as well). So I do not think it is necessary to try to replicate this PR exactly in napari. napari is hackable and should be hacked. The feature that I would argue is missing from napari is having a lasso for labels. Over time we can converge on the best UX for that.

@melonora
Copy link

Yeah ok, I am just wondering if there is experience here with the polygon lasso tool in shapes and whether that on itself fullfills the needs after refactoring of labels / shapes to use the same tooling:)

@kephale
Copy link
Author

kephale commented Sep 10, 2024

Yeah, the shapes polygon lasso is fantastic and well implemented. The end zone based trigger for closing the shape definitely accelerates annotation, and is arguably beneficial in other ways (e.g. clicking can jitter your mouse position).

From napari's perspective, I see end zones as part of cellpose's style of annotation. I don't think we need to add end zones to napari at this point in time because they are straightforward to implement (e.g. you currently close the shape on mouse click, but listening to the mouse position and triggering a shape closure is only a little more complicated). However, not everyone will want this interaction for closing shapes. We should add end zones to napari if either:
(1) cellpose becomes the only tool for annotation in napari,
(2) some tools/people other than cellpose start using the same feature, or
(3) users begin to expect this type of annotation in napari.

@kephale
Copy link
Author

kephale commented Sep 10, 2024

From some DMs w/ @melonora I wanted to clarify the feature in this PR w.r.t. napari lasso because the video doesn't capture the full UX of cellpose style annotation here.

The current polygon lasso in napari can be used 2 ways:

  • click+hold, drag to draw, release to close the shape
  • click, hover to draw, click to close the shape

The unique idea that is coming from the cellpose UI is the end zone. Annotating with the end zone is:

  • click, hover to draw and close shape when the mouse returns to the end zone (e.g. use mouse position to trigger closing of the shape)

This means the annotator does not need to hold the mouse button (which leads to fatigue), or make a second click. We know the users are drawing closed shapes so they'll need to return to the start. What isn't clear to me is that every user wants this to happen automatically, or if most people still want to be able to click to close the shape.

Otherwise, I think having the napari shapes lasso work for the labels layer would be a strict improvement (e.g. the napari lasso uses an overlay instead of creating a Points layer, and currently this PR doesn't have a line connecting the points).

@carsen-stringer
Copy link
Member

thanks so much @kephale this is amazing!

after running the segmentation, I think the output masks should themselves be a labelpainter so they are editable (I think in this case then I shouldn't have cellpose also return the outlines since they will become out-dated with the edits of the user). also the user may want to do labeling without running segmentation. should we have a button to create a labelpainter or is there another alternative?

@psobolewskiPhD
Copy link
Contributor

psobolewskiPhD commented Sep 13, 2024

This is pretty cool!
I'm going to chime in that using the Shapes lasso tool to label -- which lets you correct a bit if needed easily -- and then Layer context menu Convert to Labels is my go-to these days for these types of labeling tasks.

Edit: note that the napari Labels polygon tool has as distance threshold for double-click-to-close. It wouldn't be trivial, but could add similar stopping behavior to Shapes tools to give a landing zone-like effect.

@kephale
Copy link
Author

kephale commented Sep 25, 2024

Chatted w/ @Czaki about this. He pointed out the idea that has previously been discussed of using overlays for the Shapes lassos which would also make it easier to reuse the code for Labels.

@kephale
Copy link
Author

kephale commented Oct 3, 2024

Following up again, I met with @melonora about this yesterday and we started coordinating on the lasso implementation for the Labels layer.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants