Communication-Efficient Federated Learning With Dataset Condensation for Vision Tasks

Internet of Things (IoT) devices are widely distributed with large and scattered data, risk of privacy leakage and high bandwidth cost make data transmission and aggregation impractical. Federated Learning (FL) is an important privacy-preserving multi-party learning paradigm, involving collaborative learning with others and local updating based on private data. Data heterogeneity is one of the main challenges in FL scenarios, which can greatly limit FLs applicability and performance. Most existing approaches tackle the heterogeneity challenge during local training or model aggregation while ignoring the performance drop caused by direct model aggregation and catastrophic forgetting in the global model. This work proposes a novel FL approach, called Federated Learning with Dataset Condensation (FedDC). It aims to relieve the issue of knowledge discrepancy among local models and catastrophic forgetting in the global model. Specifically, this work proposes a differential distribution matching method to summarize local data for each client without compromising on data privacy. It introduces collaborative knowledge distillation to encourage local models to learn from others. To mitigate interround forgetting, this work stabilizes the averaged global models training by leveraging auxiliary information from its immediately past one. Extensive experiments are conducted. The results show that FedDC significantly outperforms the state-ofthe-art methods on various vision tasks.