Is there a way to evaluate classification/regression performance accounting of noisy annotations.
Let’s say I have a cat/dog detector and 1000 cat/dog test images. The 1000 images are human annotated, so it’s likely there are annotation errors, e.g. some bounding box may be incorrect and some class label may be wrong. Simply draw a confusion matrix or derive an IoU just compare the detector performance with the noisy data, and I don’t think such way is appropriate.
So my questions are
What’s the appropriate way to estimate the bbox and class error margin in the data set, given it’s not possible to go through each of them?
How to incorporate the above annotation error when reporting the performance of the cat/dog detector?
Thanks.Is there a way to evaluate classification/regression performance accounting of noisy annotations.
Let’s say I have a cat/dog detector and 1000 cat/dog test images. The 1000 images are human annotated, so it’s likely there are annotation errors, e.g. some bounding box may be incorrect and some class label may be wrong. Simply draw a confusion matrix or derive an IoU just compare the detector performance with the noisy data, and I don’t think such way is appropriate.
So my questions are
What’s the appropriate way to estimate the bbox and class error margin in the data set, given it’s not possible to go through each of them?
How to incorporate the above annotation error when reporting the performance of the cat/dog detector?
Thanks. Is there a way to evaluate classification/regression performance accounting of noisy annotations.
Let’s say I have a cat/dog detector and 1000 cat/dog test images. The 1000 images are human annotated, so it’s likely there are annotation errors, e.g. some bounding box may be incorrect and some class label may be wrong. Simply draw a confusion matrix or derive an IoU just compare the detector performance with the noisy data, and I don’t think such way is appropriate.
So my questions are
What’s the appropriate way to estimate the bbox and class error margin in the data set, given it’s not possible to go through each of them?
How to incorporate the above annotation error when reporting the performance of the cat/dog detector?
Thanks. annotation, classification, regression, noise, evaluation MATLAB Answers — New Questions

​