论文标题
与人类在循环排序中成对比较的注释负担减轻：在医学图像中的应用额定值
Decreasing Annotation Burden of Pairwise Comparisons with Human-in-the-Loop Sorting: Application in Medical Image Artifact Rating
论文作者
论文摘要
通过成对比较进行排名已显示出对序数分类的可靠性提高。但是，随着对成对的注释比较比较量表，当数据集很大时，这将变得不太实际。我们提出了一种减少按定量度量进行排名所需的成对比较数量的方法，以证明该方法在本概念研究证明中通过图像质量对医学图像进行排名的有效性。使用我们开发的医学图像注释软件，我们使用分类算法与循环中的人类评估者进行了成对比较。我们发现，与成对的比较相比，这种方法大大减少了全序排名所需的比较数量，而不会损害相比的比较。
Ranking by pairwise comparisons has shown improved reliability over ordinal classification. However, as the annotations of pairwise comparisons scale quadratically, this becomes less practical when the dataset is large. We propose a method for reducing the number of pairwise comparisons required to rank by a quantitative metric, demonstrating the effectiveness of the approach in ranking medical images by image quality in this proof of concept study. Using the medical image annotation software that we developed, we actively subsample pairwise comparisons using a sorting algorithm with a human rater in the loop. We find that this method substantially reduces the number of comparisons required for a full ordinal ranking without compromising inter-rater reliability when compared to pairwise comparisons without sorting.
