TY - GEN
T1 - Straggler-Resilient Differentially-Private Decentralized Learning
AU - Yakimenka, Yauhen
AU - Weng, Chung Wei
AU - Lin, Hsuan Yin
AU - Rosnes, Eirik
AU - Kliewer, Jorg
N1 - Publisher Copyright:
© 2022 IEEE.
PY - 2022
Y1 - 2022
N2 - We consider straggler resiliency in decentralized learning using stochastic gradient descent under the notion of network differential privacy (DP). In particular, we extend the recently proposed framework of privacy amplification by decentralization by Cyffers and Bellet to include training latency -comprising both computation and communication latency. Analytical results on both the convergence speed and the DP level are derived for training over a logical ring for both a skipping scheme (which ignores the stragglers after a timeout) and a baseline scheme that waits for each node to finish before the training continues. Our results show a trade-off between training latency, accuracy, and privacy, parameterized by the timeout of the skipping scheme. Finally, results when training a logistic regression model on a real-world dataset are presented.
AB - We consider straggler resiliency in decentralized learning using stochastic gradient descent under the notion of network differential privacy (DP). In particular, we extend the recently proposed framework of privacy amplification by decentralization by Cyffers and Bellet to include training latency -comprising both computation and communication latency. Analytical results on both the convergence speed and the DP level are derived for training over a logical ring for both a skipping scheme (which ignores the stragglers after a timeout) and a baseline scheme that waits for each node to finish before the training continues. Our results show a trade-off between training latency, accuracy, and privacy, parameterized by the timeout of the skipping scheme. Finally, results when training a logistic regression model on a real-world dataset are presented.
UR - http://www.scopus.com/inward/record.url?scp=85144592566&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85144592566&partnerID=8YFLogxK
U2 - 10.1109/ITW54588.2022.9965898
DO - 10.1109/ITW54588.2022.9965898
M3 - Conference contribution
AN - SCOPUS:85144592566
T3 - 2022 IEEE Information Theory Workshop, ITW 2022
SP - 708
EP - 713
BT - 2022 IEEE Information Theory Workshop, ITW 2022
PB - Institute of Electrical and Electronics Engineers Inc.
T2 - 2022 IEEE Information Theory Workshop, ITW 2022
Y2 - 1 November 2022 through 9 November 2022
ER -