Federated Knowledge Expansion for Collections of Heterogeneous Devices

Federated Learning (FL) enables privacy preserving training across devices, including PCs, smartphones, and IoT devices. However, when heterogeneous devices collaborate to train a model, devices with the lowest resources throttle the model size and complexity, as FL communicates and trains the same global model across all devices. To address this limitation and train a large model while utilizing all computational power in a system with heterogeneous devices, we propose Federated Knowledge Expansion (FedKE). In FedKE, devices with similar resources form FL groups, which gradually expand a small model with additional computational layers or blocks, progressing toward a large model. Each group trains a model suitable for its resources and then passes the model to a higher-resource group for further expansion and training. FedKE uses a novel training mechanism that allows these groups to train in parallel. Inspired by parallel pipelining in distributed computing, this parallelism leverages the iterative aggregation of FL and allows higher-resource groups to start training as soon as the smaller model of the previous group is aggregated, rather than waiting for it to fully converge. Furthermore, each higher-resource group performs Adaptive Sparse Updating on the weights of the smaller model from the previous group to reduce computation overhead. The large model produced with FedKE theoretically approximates those trained under the ideal condition where all devices have the full capabilities to train this large model. The FedKE prototype is evaluated with two datasets and four model expansion configurations across three groups of real heterogeneous devices. The results show that it outperforms state-of-the-art solutions in terms of model accuracy, reduces the amount of model weights transferred and the total operation time compared to vanilla FL, and achieves good fault-tolerance and scalability.