A Cloud-Edge Collaborative Inference System for Data-secure LLM Serving
Wenjie Chu, Yunfeng Shao, Chunhui Du · 2025
The surge in private deployment of large language models (LLMs) driven by open-source advancements has intensified challenges in computational scalability, infrastructure costs, and data privacy. While cloud-edge collaborative inference frameworks alleviate local resource constraints through elastic cloud offloading, their efficacy in wide-area networks (WANs) is hindered by communication inefficiencies and privacy risks. This paper proposes CROSS-SEC, a novel cloud-edge collaborative inference framework integrating cross-WANs PD disaggregation with split learning (SL) for data security preservation. To mitigate transmission bottlenecks, CROSS-SEC introduces a layerwise KVCache computation-communication overlapping mechanism, coupled with asychromous concurrent transmission to eliminate ACK-induced latency. For congestion control, a dual-grained scheduling strategy is proposed: (1) KVCache-level priority scheduling across multi-user/multi-prefill requests ensures first-come-first-serve processing, and (2) latency-sensitive prioritization of latent variables over KVCache transfers guarantees TPOT compliance with SLAs. Experimental validation demonstrates that CROSS-SEC reduces TTFT by 19.83% and improves throughput by 3.63% compared to state-of-the-art frameworks, while maintaining data privacy through SL-based input/output confinement.