论文标题
品种：用于亚群移动的基准
BREEDS: Benchmarks for Subpopulation Shift
论文作者
论文摘要
我们开发了一种方法来评估模型对亚种群转移的鲁棒性 - 特别是，它们将其推广到训练过程中未观察到的新型数据亚群的能力。我们的方法利用现有数据集的基础结构来控制构成培训和测试分布的数据亚群。这使我们能够在现有的大规模数据集中综合其源可以得到精确控制和表征的现实分布变化。将此方法应用于Imagenet数据集，我们创建了一组不同的粒度偏变基准。然后，我们通过为它们获得人体基准来验证相应的偏移是可以解决的。最后，我们利用这些基准来衡量标准模型体系结构的灵敏度以及现成的火车时间鲁棒性干预的有效性。代码和数据可在https://github.com/madrylab/breeds-benchmarks上找到。
We develop a methodology for assessing the robustness of models to subpopulation shift---specifically, their ability to generalize to novel data subpopulations that were not observed during training. Our approach leverages the class structure underlying existing datasets to control the data subpopulations that comprise the training and test distributions. This enables us to synthesize realistic distribution shifts whose sources can be precisely controlled and characterized, within existing large-scale datasets. Applying this methodology to the ImageNet dataset, we create a suite of subpopulation shift benchmarks of varying granularity. We then validate that the corresponding shifts are tractable by obtaining human baselines for them. Finally, we utilize these benchmarks to measure the sensitivity of standard model architectures as well as the effectiveness of off-the-shelf train-time robustness interventions. Code and data available at https://github.com/MadryLab/BREEDS-Benchmarks .
