Skip to main navigation Skip to search Skip to main content

Device-Server Collaborative Speculative Decoding for Real-Time LLM Streaming

  • Bishakha Rani Biswas
  • , Yongjie Guan
  • , Mingrui Yin
  • , Tao Han
  • , Xueyu Hou

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Abstract

This paper addresses the challenge of real-time text streaming with large language models (LLMs) over unstable networks and server contention. To mitigate delays from packet loss and computational bottlenecks, we propose an adaptive framework that integrates speculative decoding with network-aware optimization. A lightweight draft model runs on the user's edge device to generate tokens autoregressively, while a larger target model on the server periodically verifies them. We explicitly model end-to-end latency with respect to verification acceptance rates and packet loss, and optimize the verification ratio under latency constraints. Experiments show improved resilience to high packet loss and server contention, achieving over 4× lower rebuffering and 60% higher verification ratios compared to baselines.

Original languageEnglish (US)
Title of host publicationGLOBECOM 2025 - 2025 IEEE Global Communications Conference
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages3493-3498
Number of pages6
ISBN (Electronic)9798331577810
DOIs
StatePublished - 2025
Event2025 IEEE Global Communications Conference, GLOBECOM 2025 - Taipei, Taiwan, Province of China
Duration: Dec 8 2025 → Dec 12 2025

Publication series

NameProceedings - IEEE Global Communications Conference, GLOBECOM
ISSN (Print)2334-0983
ISSN (Electronic)2576-6813

Conference

Conference2025 IEEE Global Communications Conference, GLOBECOM 2025
Country/TerritoryTaiwan, Province of China
CityTaipei
Period12/8/25 → 12/12/25

All Science Journal Classification (ASJC) codes

  • Signal Processing
  • Hardware and Architecture
  • Computer Networks and Communications
  • Artificial Intelligence

Keywords

  • distributed computing
  • large language model
  • text streaming
  • wireless network

Fingerprint

Dive into the research topics of 'Device-Server Collaborative Speculative Decoding for Real-Time LLM Streaming'. Together they form a unique fingerprint.

Cite this