Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make COCO metrics replicate official cocoapi better #37

Open
tadejsv opened this issue Jun 5, 2023 · 0 comments
Open

Make COCO metrics replicate official cocoapi better #37

tadejsv opened this issue Jun 5, 2023 · 0 comments

Comments

@tadejsv
Copy link
Owner

tadejsv commented Jun 5, 2023

Currently, the COCO metrics are a bit different than what we get with the official cocoapi. While the matching algorithm and other calculations should be the same, there are some differences in our implementation:

  • We do not do anything special with "iscrowd":1 ground truths
  • We do not use the area attribute from COCO json ground truths (which is obtained from the segmentation mask): instead, we multiply the width and height of the bounding box to obtain the area
  • I do not use maxDets - that is, I do not limit the maximum number of detections per image. This limiting is usually done during inference (usually "fused" with NMS), or I might additionally add it to IO constructors for Detections - but not as part of the evaluation

However, even if remove these three things (as is done in our integration example), the results still differ. So the best thing would be to narrow the differences down:

  • take only 1 size range, one IoU threshold and one class, and make sure we still have the difference.
  • Then, try to get the difference down to one image (it should be possible)
  • For this image, evaluate what went wrong - manually inspect the matches that were made

Hopefully this reveals some un-noticed difference in our matching algorithm, as compared to COCO, and then this issue will be fixed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant