论文标题
通过利率延伸理论优化联邦学习中的沟通准确性权衡
Optimizing the Communication-Accuracy Trade-off in Federated Learning with Rate-Distortion Theory
论文作者
论文摘要
联合学习（FL）中的重要瓶颈是将客户设备从客户设备发送到中央服务器的网络通信成本。我们介绍了FL中模型更新的统计数据以及各种压缩技术的作用和好处。在这些观察结果的推动下，我们提出了一种新的方法来降低平均通信成本，在许多用例中，这在许多用例中几乎是最佳的，并且在堆栈溢出的Next-Word Prediction上的top-K，Drive，3LC，3LC和QSGD均超过了，这是一个现实且具有挑战性的FL基准。这是通过使用速率延伸理论来检查问题的方法，并提出变形作为模型准确性的可靠代理。可以更有效地使用失真来优化客户跨客户的模型性能和通信成本之间的权衡。我们从经验上证明，尽管非i.i.d。联合学习的性质，率延伸边界在数据集，优化者，客户和培训回合之间是一致的。
A significant bottleneck in federated learning (FL) is the network communication cost of sending model updates from client devices to the central server. We present a comprehensive empirical study of the statistics of model updates in FL, as well as the role and benefits of various compression techniques. Motivated by these observations, we propose a novel method to reduce the average communication cost, which is near-optimal in many use cases, and outperforms Top-K, DRIVE, 3LC and QSGD on Stack Overflow next-word prediction, a realistic and challenging FL benchmark. This is achieved by examining the problem using rate-distortion theory, and proposing distortion as a reliable proxy for model accuracy. Distortion can be more effectively used for optimizing the trade-off between model performance and communication cost across clients. We demonstrate empirically that in spite of the non-i.i.d. nature of federated learning, the rate-distortion frontier is consistent across datasets, optimizers, clients and training rounds.
