TY - GEN
T1 - Device-Server Collaborative Speculative Decoding for Real-Time LLM Streaming
AU - Biswas, Bishakha Rani
AU - Guan, Yongjie
AU - Yin, Mingrui
AU - Han, Tao
AU - Hou, Xueyu
N1 - Publisher Copyright:
© 2025 IEEE.
PY - 2025
Y1 - 2025
N2 - This paper addresses the challenge of real-time text streaming with large language models (LLMs) over unstable networks and server contention. To mitigate delays from packet loss and computational bottlenecks, we propose an adaptive framework that integrates speculative decoding with network-aware optimization. A lightweight draft model runs on the user's edge device to generate tokens autoregressively, while a larger target model on the server periodically verifies them. We explicitly model end-to-end latency with respect to verification acceptance rates and packet loss, and optimize the verification ratio under latency constraints. Experiments show improved resilience to high packet loss and server contention, achieving over 4× lower rebuffering and 60% higher verification ratios compared to baselines.
AB - This paper addresses the challenge of real-time text streaming with large language models (LLMs) over unstable networks and server contention. To mitigate delays from packet loss and computational bottlenecks, we propose an adaptive framework that integrates speculative decoding with network-aware optimization. A lightweight draft model runs on the user's edge device to generate tokens autoregressively, while a larger target model on the server periodically verifies them. We explicitly model end-to-end latency with respect to verification acceptance rates and packet loss, and optimize the verification ratio under latency constraints. Experiments show improved resilience to high packet loss and server contention, achieving over 4× lower rebuffering and 60% higher verification ratios compared to baselines.
KW - distributed computing
KW - large language model
KW - text streaming
KW - wireless network
UR - https://www.scopus.com/pages/publications/105036365532
UR - https://www.scopus.com/pages/publications/105036365532#tab=citedBy
U2 - 10.1109/GLOBECOM59602.2025.11432593
DO - 10.1109/GLOBECOM59602.2025.11432593
M3 - Conference contribution
AN - SCOPUS:105036365532
T3 - Proceedings - IEEE Global Communications Conference, GLOBECOM
SP - 3493
EP - 3498
BT - GLOBECOM 2025 - 2025 IEEE Global Communications Conference
PB - Institute of Electrical and Electronics Engineers Inc.
T2 - 2025 IEEE Global Communications Conference, GLOBECOM 2025
Y2 - 8 December 2025 through 12 December 2025
ER -