Phase 8: LangGraph state machine, full pipeline, retry loop, session loader

data/cache/1e5d6ab4fdb2fb277f3fd6083f3ea165.json

@@ -0,0 +1 @@
+[{"url": "https://arxiv.org/html/2502.19732v4", "snippet": "October 8, 2025 -Moving forward, significant challenges persist inconstructing solid theoretical foundations to grasp the balance between parallelism and quality, as well as in developing comprehensive approaches that span different modalities\u2014efforts that could narrow the divide between the capabilities ...", "title": "Speculative Decoding and Beyond: An In-Depth Survey of Techniques", "inferred_year": 2025, "hybrid_score": 0.0, "source": "duckduckgo"}, {"url": "https://arxiv.org/abs/2603.03251", "snippet": "1 week ago -If the actual verification outcome is then in the predicted set, a speculation can be returned immediately, eliminating drafting overhead entirely. We identify three key challenges presented by speculative speculative decoding, and suggest principled methods to solve each.", "title": "[2603.03251] Speculative Speculative Decoding", "inferred_year": null, "hybrid_score": 0.0, "source": "duckduckgo"}, {"url": "https://arxiv.org/html/2411.13157v1", "snippet": "November 20, 2024 -If a significant number of tokens are rejected after the drafting phase, this can lead to inefficiencies in a batch, as the errors must be corrected while still processing the remainder of the batch. Moreover, not all sequences require the same amount of GPU usage, since more complex sequences require more compute power. Additional research on such challenges of batching will be beneficial to maximize model parallelism. In real world applications, people do not have the comput", "title": "Closer Look at Efficient Inference Methods: A Survey of Speculative Decoding", "inferred_year": 2024, "hybrid_score": 0.0, "source": "duckduckgo"}]

data/cache/4c24dca2d7482d3b1bde1eeca342e442.json

@@ -0,0 +1 @@
+[{"url": "https://arxiv.org/html/2510.02128v1", "snippet": "Oct 2, 2025\u00b7This paper conducts an analysis of speculative decoding through the lens of its potential disparate speed-up rates across tasks. Crucially, the ...", "title": "The Disparate Impacts of Speculative Decoding - arXiv", "inferred_year": 2025, "hybrid_score": 0.0, "source": "duckduckgo"}, {"url": "https://arxiv.org/html/2502.19732v4", "snippet": "Oct 8, 2025\u00b7To address this, recent research has focused on optimizing decoding efficiency to accelerate recommendation generation. [105] propose DARE ...Missing:paper| Show results with:paper", "title": "Speculative Decoding and Beyond: An In-Depth Survey of Techniques", "inferred_year": 2025, "hybrid_score": 0.0, "source": "duckduckgo"}, {"url": "https://arxiv.org/pdf/2603.03251", "snippet": "Mar 3, 2026\u00b7Speculative decoding (Leviathan et al., 2023; Chen et al., 2023) (SD) is a technique introduced to alleviate this problem. Instead of slow, ...", "title": "[PDF] Speculative Speculative Decoding - arXiv", "inferred_year": 2026, "hybrid_score": 0.0, "source": "duckduckgo"}]

data/cache/7fe27046f57c042b4663b61225a39f78.json

@@ -0,0 +1 @@
+[{"title": "Accelerating LLM Inference with Staged Speculative Decoding", "abstract": "Recent advances with large language models (LLM) illustrate their diverse capabilities. We propose a novel algorithm, staged speculative decoding, to accelerate LLM inference in small-batch, on-device scenarios. We address the low arithmetic intensity of small-batch inference by improving upon previous work in speculative decoding. First, we restructure the speculative batch as a tree, which reduces generation costs and increases the expected tokens per batch. Second, we add a second stage of speculative decoding. Taken together, we reduce single-batch decoding latency by 3.16x with a 762M parameter GPT-2-L model while perfectly preserving output quality.", "year": 2023, "citation_count": 165, "paper_id": "43e624ddeed82df944a6cae0dedec3372438e243", "authors": ["B. Spector", "Christal Re"], "references": ["d50f023fe0921cabdd6d053c377cdd26c715994c", "42a14d824caa3348046eb34c37e2ab7985faa7a3", "a1f8082505c7e90b0a033e1b9da0a97d67aad66c", "d8e9f8c8a37cb4cd26b92ad0d942d641cd512644", "f3a6115e5fb2237df938976e005468f0b18da797", "7da0f2501034522e3d50af7e9b8fa7ec9d7b65b6", "4be7d1524edb0137599a5cc95f72844b85a52fe1", "87c5b281fa43e6f27191b20a8dd694eda1126336", "094ff971d6a8b8ff870946c9b3ce5aa173617bfb", "acbdbf49f9bc3f151b93d9ca9a06009f4f6eb269", "db1afe3b3cd4cd90e41fbba65d3075dd5aebb61e", "df7d26339adf4eb0c07160947b9d2973c24911ba", "90abbc2cf38462b954ae1b772fac9532e2ccd8b0", "6c4b76232bb72897685d19b3d264c6ee3005bc2b", "204e3073870fae3d05bcbc2f6a8e263d9b72e776", "a6cb366736791bcccc5c8639de5a8f9636bf87e8", "07563644cae0d3d03b37724efe084b6510220103", "c1787db25af5614f41e56938aa594f2dbb1dca07", "3efcb97c1de1c87832a7a1d99e91801992a938ec", "b0130277677e5b915d5cd86b3afafd77fd08eb2e", "9405cc0d6169988371b2755e573cc28650d14dfe"], "hybrid_score": 0.6804, "source": "semantic_scholar"}, {"title": "Tutorial Proposal: Speculative Decoding for Efficient LLM Inference", "abstract": "This tutorial presents a comprehensive introduction to Speculative Decoding (SD), an advanced technique for LLM inference acceleration that has garnered significant research interest in recent years. SD is introduced as an innovative decoding paradigm to mitigate the high inference latency stemming from autoregressive decoding in LLMs. At each decoding step, SD efficiently drafts several future tokens and then verifies them in parallel. This approach, unlike traditional autoregressive decoding, facilitates the simultaneous decoding of multiple tokens per step, thereby achieving promising 2x-4x speedups in LLM inference while maintaining original distributions. This tutorial delves into the latest techniques in SD, including draft model architectures and verification strategies. Additionally, it explores the acceleration potential and future research directions in this promising field. We aim for this tutorial to elucidate the current research landscape and offer insights for researchers interested in Speculative Decoding, ultimately contributing to more efficient LLM inference.", "year": 2025, "citation_count": 4, "paper_id": "bd05866c656b6cbe8fd3a6782a0ed1cbcb788185", "authors": ["Heming Xia", "Cunxiao Du", "Yongqing Li", "Qian Liu", "Wenjie Li"], "references": ["160924af0791331ec8fa5a3d526ea125355f3b8b", "cab58a0263d454604896dce6b8fbf4df1dd99ff0", "f1a9e0830bc36c048fa4659beaa62609869895b5", "382d3b37f53fd6118ab979cb56f7f3d13eb0951d", "1b5db3170c195508ff24fee8eda0d4987e806f0b", "a8b66565cdb2b8c90556bb98a7fc58ac679c2cec", "57e7af0b69325fafb371ef5d502e39ef9c90ef7e", "0cee098244c9978032702862a43a09f468f691a4", "532c2c7a247d9e97d20abec1b2f4612984fdab93", "83b90f4a0ae4cc214eb3cc140ccfef9cd99fac05", "00e889fcfaf4396a20f37f681cf8b14f3e878879", "104b0bb1da562d53cbda87aec79ef6a2827d191a", "3556722b4703a21abafd2f9388743202943f4503", "f0c31511134abdd23f990310e8a2f2eb3a629b62", "57e849d0de13ed5f91d086936296721d4ff75a75", "b7d12aec8a0152ec4921dfa43ab525a63b334385", "a1f8082505c7e90b0a033e1b9da0a97d67aad66c", "d8e9f8c8a37cb4cd26b92ad0d942d641cd512644", "218c5c69f3cf0c158e9b6af239a2cc62a688c6de", "3efd4b048dd7544333092332bccc3f0aea79f5c7", "2c2c466be651951014e9a97518bd078877f8212c", "3c8a456509e6c0805354bd40a35e3f2dbf8069b1", "dc52b09089704ebd6f471177474bc29741c50023", "5e04881e91bff952d102d967c4ffb498ec30d4af", "843a1567b056c8a1d0deddc8b699e1725194f85c"], "hybrid_score": 0.6552, "source": "semantic_scholar"}, {"title": "SpecVLM: Enhancing Speculative Decoding of Video LLMs via Verifier-Guided Token Pruning", "abstract": "Video large language models (Vid-LLMs) have shown strong capabilities in understanding video content. However, their reliance on dense video token representations introduces substantial memory and computational overhead in both prefilling and decoding. To mitigate the information loss of recent video token reduction methods and accelerate the decoding stage of Vid-LLMs losslessly, we introduce SpecVLM, a training-free speculative decoding (SD) framework tailored for Vid-LLMs that incorporates staged video token pruning. Building on our novel finding that the draft model's speculation exhibits low sensitivity to video token pruning, SpecVLM prunes up to 90% of video tokens to enable efficient speculation without sacrificing accuracy. To achieve this, we performs a two-stage pruning process: Stage I selects highly informative tokens guided by attention signals from the verifier (target model), while Stage II prunes remaining redundant ones in a spatially uniform manner. Extensive experiments on four video understanding benchmarks demonstrate the effectiveness and robustness of SpecVLM, which achieves up to 2.68$\\times$ decoding speedup for LLaVA-OneVision-72B and 2.11$\\times$ speedup for Qwen2.5-VL-32B. Code is available at https://github.com/zju-jiyicheng/SpecVLM.", "year": 2025, "citation_count": 7, "paper_id": "6b53ef2981c053deafc9267ac5d4404bd5dc80b1", "authors": ["Yicheng Ji", "Jun Zhang", "Heming Xia", "Jinpeng Chen", "Lidan Shou", "Gang Chen", "Huan Li"], "references": ["38d00f8474a120c0748bbd6697481ea37e2fa390", "c23490f57fe0dd1ee77d3acbdf5689667ff3452e", "933e1246c24fed43eeb44e354796aa76c21746ca", "1daf2c6e5deb9f71275e8affba25aa07c245dd11", "32ab8c59adcf1b5b5fdee2f5ddd8addfcba3bf67", "9e39eb548f1ddd9a98b73ed03b35ca9991da8046", "260f54bc42934cb85c71318863b5f3b2c042d92f", "bacdf9671fb872287201b53d768df89b4d6630a3", "a05c0dd5a8bc70814517fb424cf55e94b4208e67", "28ea43cfad0ad2a367096581865114b6b94a9a6e", "3f03876b23b491bdc161816024044e13b02b46e5", "1a71f7b216b710b936da666027014adb83af8e7a", "2f9bcfe03ed3c5827036e7a7e672f952e2d1a382", "c4da87efe7ff962b327d8aad409cecab7a51e79a", "cab58a0263d454604896dce6b8fbf4df1dd99ff0", "d02b9420df66330620f8853d19de610c98d2e1c1", "382d3b37f53fd6118ab979cb56f7f3d13eb0951d", "1b5db3170c195508ff24fee8eda0d4987e806f0b", "57e7af0b69325fafb371ef5d502e39ef9c90ef7e", "107fb6eec2febbae12db29bf3e311aaf5680027c", "fdc53c2c10742464087c0525f77e32604827a21d", "2a09ebbfcca1a6994eeb472cd4159f5f3858dbf9", "b7d12aec8a0152ec4921dfa43ab525a63b334385", "d8e9f8c8a37cb4cd26b92ad0d942d641cd512644", "218c5c69f3cf0c158e9b6af239a2cc62a688c6de", "204e3073870fae3d05bcbc2f6a8e263d9b72e776", "c735320ee0e5b2e580043b0a1504e592c0840e24"], "hybrid_score": 0.6063, "source": "semantic_scholar"}, {"title": "Speculative Decoding and Beyond: An In-Depth Survey of Techniques", "abstract": "Sequential dependencies present a fundamental bottleneck in deploying large-scale autoregressive models, particularly for real-time applications. While traditional optimization approaches like pruning and quantization often compromise model quality, recent advances in generation-refinement frameworks demonstrate that this trade-off can be significantly mitigated. This survey presents a comprehensive taxonomy of generation-refinement frameworks, analyzing methods across autoregressive sequence tasks. We categorize methods based on their generation strategies (from simple n-gram prediction to sophisticated draft models) and refinement mechanisms (including single-pass verification and iterative approaches). Through systematic analysis of both algorithmic innovations and system-level implementations, we examine deployment strategies across computing environments and explore applications spanning text, images, and speech generation. This systematic examination of both theoretical frameworks and practical implementations provides a foundation for future research in efficient autoregressive decoding.", "year": 2025, "citation_count": 15, "paper_id": "adb0c91669f211d3e99590d8efb1a6d06aabf016", "authors": ["Yunhai Hu", "Zining Liu", "Zhenyuan Dong", "Tianfan Peng", "Bradley McDanel", "Sai Qian Zhang"], "references": ["621ba0b763e89a4e82fac4b4e264ca8fb6c04fb6", "50af31e0aa76103dfb0ed901acea0bddbdbaef8d", "cb32b7087239b445aec1f2b9882dc88e8fbac482", "6008a7676b3cae015670217c263ec0fdfc31e38b", "e7f3cb962ea084300be6394662a054dc43799e9c", "62c35559859ad9bd0a14891a2901f4f47d50ccaf", "3230ed476488a459d27efc22e8cc5eb4d0298c4f", "27848e58b581e66db0b21c170ef61d62165f3d93", "0b4e28534319802e2a28db384c22f9c4b37867c6", "e7be26cedfbbcb68bff4d44bc9c66b74ace8205b", "a58cf38e7d57da6fda25422ecd1005c26613b566", "4b29c01ba5854381f09ee8c99f6e9b146e0f2854", "90d4a025f872062772c4eac7213a03bae9b1ffcf", "69d995c14c46eeda5c3761b5cb4e094816052926", "139aea58a13e615a7e466fe0c74273804372b746", "9a494717e6fef23fe0a941fa3a0f8f3779d83fa0", "4385314ade0ee492d99bc03c1e8ea98ec577c44c", "bacdf9671fb872287201b53d768df89b4d6630a3", "2d06b3601b4841a464929558d1d25b0db385f91f", "79172e64a8b7abb8523b0fcf44802b9e1142aede", "56ff54c5fe085b68bf533fd1322fcf1431af18c8", "b41ae48f3389d7aa1f7ef39b823ea587b3232c50", "5370aa35ab6fc216284402435af6fc7940ec9677", "d4e25f1a01580d7b1d3be2ca500fb6a03b2042b6", "3c71cf8939d8956a08c215c00b0b6981f531c820", "16be68af143ed87af00d9f52bf0d945780a270e5", "88a8d366eaf6a63829ad1f4ca1a9ce161427ef83", "81039ad704a0c23059677d0bbf097b9bc90326ba", "230762f388c4b6f0e8af4554e2df5fd4248b522d", "28ea43cfad0ad2a367096581865114b6b94a9a6e", "ace3d2bcecc658a97632c9be4ce461450271eb5b", "40e8af970329135ec95057d73e239dab805ad128", "72759823ffb8591a1e82eef860babe0b2c2b738f", "4ee9bd94575535ea25d71ccf9c914f8861982039", "6ecbac9ffce24f2d643755baba3ef114faa552b6", "160924af0791331ec8fa5a3d526ea125355f3b8b", "853a593768d0128b7104017e296028b9ae87ec27", "540209de237977a2fb16d3a2c4021618e8ea8b47", "cab58a0263d454604896dce6b8fbf4df1dd99ff0", "e46a79cf7c2c9192639f9e1d12280650e06b9d56", "d613e92a30f19029d842d8ef7d73f8ad4dbbd3ae", "7e206ea4c8b42f67d66c5205891b06d2af612667", "05bde17d7cfe69fff3ab574d2521b9d806fc901e", "a541d6399f3e2703e1faf6c10bcec3ff22d6bb8c", "050d66cab131c8e10c230df9469d897622ccf7d2", "b9c289f141c54ab465a885fb7be7936d98e8c853", "d50c7d1287d93539f5c4cc312ae9cfaaaf570fe6", "2774fb46e2078396e556704c4b325f027e9d3bf9", "eee108b3e51e89d160d1116935e062c4d169b475", "c17d6af40861a705a8e5e60aaf54762f43b787d8", "d0deaec3e1f74701ac43600d9e64c5c969be7391", "840ea9e7883ccd276ba26f97b76072686ce6be30", "0d81434f6a3fca88774ab058ebabd2206be70016", "76e8218f657c77c38da44daaed5bb54ab727a8fc", "f4fff56d412ab1319daf07cda14b6e2becf86d6c", "a8dcaddc541b3f54a513eb1a4d3310d26fd9a761", "ab16bf2223fcb8c749a52ee3cc0496dc5bb0b4ec", "a1849a77644ff411a03833b5aa7a65ff57158c50", "582d172769d5774774c1dfd7546b829248d5adc1", "4623901ec1f8311b32e27bd627b2bce56161c6bb", "4d742b55892f071b7a8c52af6e5da4a0c529ff5e", "26128d9ec47f044de6f0186f86c0c7fb3ccef77f", "df6239d0c9acac4d8a700946aa323a998daedbc3", "25218efe864ee83bea43f8d1b0e0ed4ed81a0581", "d0b049018c9e21b7b95c179d33e1e2ac9113c85b", "f1a9e0830bc36c048fa4659beaa62609869895b5", "1b5db3170c195508ff24fee8eda0d4987e806f0b", "57e7af0b69325fafb371ef5d502e39ef9c90ef7e", "e30666ed82670463aa47686e744f0c6f2a0e083d", "4067a6f57f708dec4459d3d4322373e06c2b168c", "70ded1d6e83a1cbeecec256a070c4b9ebfc6085f", "532c2c7a247d9e97d20abec1b2f4612984fdab93", "ea1f648988c632a6dbab6d8b88432456aa021cfb", "f206d34afed6a5705757f96ea97c7bfb9e1a83cd", "56767c18bb5aaa2b6377624168bed1b6dcc4b94d", "ba5261e729c181e28a98dee2c08d7cf5fc7127a2", "564855d475ed9197dd7516594557ff886ff623e5", "43e624ddeed82df944a6cae0dedec3372438e243", "104b0bb1da562d53cbda87aec79ef6a2827d191a", "d988eb48e8b4e471f5df9d081bfc32db0781e6bf", "c12b80b44d9acfe6cd92fdf965264c4b706c367c", "3556722b4703a21abafd2f9388743202943f4503", "dfd8944d39b378489b878d6e105d040fa0e524db", "eb3415db1b322b1f7bf95f8697aed701b0d40f88", "57e849d0de13ed5f91d086936296721d4ff75a75", "b7d12aec8a0152ec4921dfa43ab525a63b334385", "a1f8082505c7e90b0a033e1b9da0a97d67aad66c", "1367dcff4ccb927a5e95c452041288b3f0dd0eff", "d8e9f8c8a37cb4cd26b92ad0d942d641cd512644", "e342165a614588878ad0f4bc9bacf3905df34d08", "1243e13254bb4ea1f71b4be8a3e4e54ffd02d2fe", "218c5c69f3cf0c158e9b6af239a2cc62a688c6de", "1197ae4a62f0e0e4e3f3fb70396b5ff06ef371aa", "e3a3e85c5a32af29e13b3561f6cf070de70651de", "5c126ae3421f05768d8edd97ecd44b1364e2c99a", "90abbc2cf38462b954ae1b772fac9532e2ccd8b0", "4c7730d6227f8b90735ba4de7864551cb8928d92", "5e04881e91bff952d102d967c4ffb498ec30d4af", "b587ee7c802a5bd222a69090f59285e0dfdb29f1", "896de8418884f4aab1ae4a60027500c9e8baffc3", "de5e7320729f5d3cbb6709eb6329ec41ace8c95d", "712237ba4a33a2426e622465f4cd4e9e8ffb4587", "7c00f228c6e926baf86b2c29b32702937af911d1", "5b6fdb2aea424d02c141da81d62b04d739d62b96", "99dbeffeb423b08109ee21c341020ff261cc5839", "5cbdc8fd22d054d37c5d6baff0c35d2a04979973", "9405cc0d6169988371b2755e573cc28650d14dfe", "cd18800a0fe0b668a1cc19f2ec95b5003d0a5035", "3e2734025037fc626873c56e05ddb43ccccd3858"], "hybrid_score": 0.5437, "source": "semantic_scholar"}]

data/cache/bf42e0ac174449dddbcc6b211ff7ee33.json

@@ -0,0 +1 @@
+[{"title": "DSSD: Efficient Edge-Device LLM Deployment and Collaborative Inference via Distributed Split Speculative Decoding", "abstract": "Large language models (LLMs) have transformed natural language processing but face critical deployment challenges in device-edge systems due to resource limitations and communication overhead. To address these issues, collaborative frameworks have emerged that combine small language models (SLMs) on devices with LLMs at the edge, using speculative decoding (SD) to improve efficiency. However, existing solutions often trade inference accuracy for latency or suffer from high uplink transmission costs when verifying candidate tokens. In this paper, we propose Distributed Split Speculative Decoding (DSSD), a novel architecture that not only preserves the SLM-LLM split but also partitions the verification phase between the device and edge. In this way, DSSD replaces the uplink transmission of multiple vocabulary distributions with a single downlink transmission, significantly reducing communication latency while maintaining inference quality. Experiments show that our solution outperforms current methods, and codes are at: https://github.com/JasonNing96/DSSD-Efficient-Edge-Computing", "year": 2025, "citation_count": 9, "paper_id": "0646bf8db2b75dd1d45e0f04104547710d02aa1b", "authors": ["Jiahong Ning", "Ce Zheng", "Tingting Yang"], "references": ["8d75c83ce75759e233c493b77c435371d33a4877", "c174f23f0c4d326c690a71603693a26171e7aec5", "82c686fa2eee0eab7f92e101244552e84fa028a6", "12099c1e2fb7a0c8d0153adaa48d189cb3c50cad", "c80a3847fd13b06e1b0ff4e59c4bb3df06bd088d", "97b6f4357d1e3ab40a7ee60acb5260a948e3641d", "4ff8a215899299617efeeb288410c3c7169c160c", "a1f8082505c7e90b0a033e1b9da0a97d67aad66c", "d8e9f8c8a37cb4cd26b92ad0d942d641cd512644", "f40d75978e76702c5e3a20adb149694788b53fc0"], "hybrid_score": 0.5941, "source": "semantic_scholar"}, {"title": "Multi-Candidate Speculative Decoding", "abstract": "Large language models have shown impressive capabilities across a variety of NLP tasks, yet their generating text autoregressively is time-consuming. One way to speed them up is speculative decoding, which generates candidate segments (a sequence of tokens) from a fast draft model that is then verified in parallel by the target model. However, the acceptance rate of candidate tokens receives limitations from several factors, such as the model, the dataset, and the decoding setup. This paper proposes sampling multiple candidates from a draft model and then organising them in batches for verification. We design algorithms for efficient multi-candidate verification while maintaining the distribution of the target model. Our approach shows significant improvements in acceptance rates on multiple datasets and models, consistently outperforming standard speculative decoding.", "year": 2024, "citation_count": 32, "paper_id": "1c0a0ec50a639efe9569d0c57f73c8e1b47acbcd", "authors": ["Sen Yang", "Shujian Huang", "Xinyu Dai", "Jiajun Chen"], "references": ["ea1f648988c632a6dbab6d8b88432456aa021cfb", "56767c18bb5aaa2b6377624168bed1b6dcc4b94d", "83b90f4a0ae4cc214eb3cc140ccfef9cd99fac05", "163b4d6a79a5b19af88b8585456363340d9efd04", "57e849d0de13ed5f91d086936296721d4ff75a75", "a1f8082505c7e90b0a033e1b9da0a97d67aad66c", "e965e93e76a9e6c4e4863d145b5c007b540d575d", "d8e9f8c8a37cb4cd26b92ad0d942d641cd512644", "cdbd4f9b6ab2e2fd1ddf5400d5ed2c18960635d1", "4be7d1524edb0137599a5cc95f72844b85a52fe1", "13a0d8bb38f739990c8cd65a44061c6534f17221", "da0d38cf2ac7e2a6908e0d9e1fff07058daab2ed", "4a8964ea0de47010fb458021b68fa3ef5c4b77b2", "90abbc2cf38462b954ae1b772fac9532e2ccd8b0", "dc52b09089704ebd6f471177474bc29741c50023", "6c4b76232bb72897685d19b3d264c6ee3005bc2b", "5e04881e91bff952d102d967c4ffb498ec30d4af", "204e3073870fae3d05bcbc2f6a8e263d9b72e776", "0c908739fbff75f03469d13d4a1a07de3414ee19", "5ec85a0d88adcc4344bb5cc81b0d1aef9bcd8dcc", "ee34b5cc38241ed5eb39d08f2eea322469103471", "d1a6b3a5efde3783b53f822dc8dd00aaac934b95"], "hybrid_score": 0.5922, "source": "semantic_scholar"}, {"title": "Fuzzy Speculative Decoding for a Tunable Accuracy-Runtime Tradeoff", "abstract": "Speculative Decoding (SD) enforces strict distributional equivalence to the target model when accepting candidate tokens. While it maintains the target model's generation quality, this strict equivalence limits the speedup achievable by SD and prevents users from trading deviations from the target distribution in exchange for further inference speed gains. To address these limitations, we introduce Fuzzy Speculative Decoding (FSD) - a decoding algorithm that generalizes SD by accepting candidate tokens based on the divergences between the target and draft model distributions. By allowing for controlled divergence from the target model, FSD enables users to flexibly trade generation quality for inference speed. Across several benchmarks, our method is able to achieve significant runtime improvements of over 5 tokens per second faster than SD at only an approximate 2% absolute reduction in benchmark accuracy. In many cases, FSD is even able to match SD benchmark accuracy at over 2 tokens per second faster, demonstrating that distributional equivalence is not necessary to maintain target model performance. Furthermore, FSD can be seamlessly integrated into existing SD extensions; we demonstrate this by applying FSD to EAGLE-2, greatly enhancing this existing extension's efficiency while allowing it to leverage FSD's tunable quality-speed trade-off.", "year": 2025, "citation_count": 4, "paper_id": "c110e21884b7167b0ac07cdee9bd77e42ec68bdb", "authors": ["Maximilian Holsman", "Yukun Huang", "Bhuwan Dhingra"], "references": ["6008a7676b3cae015670217c263ec0fdfc31e38b", "88aa6b1f37d1fd8e0a40499ce9bb87873f03aaa8", "81039ad704a0c23059677d0bbf097b9bc90326ba", "ace3d2bcecc658a97632c9be4ce461450271eb5b", "ec2ce4e38af8bc82f1b8928ba51a84911bad0cc6", "cab58a0263d454604896dce6b8fbf4df1dd99ff0", "7e206ea4c8b42f67d66c5205891b06d2af612667", "050d66cab131c8e10c230df9469d897622ccf7d2", "eee108b3e51e89d160d1116935e062c4d169b475", "df6239d0c9acac4d8a700946aa323a998daedbc3", "f5c83f5156904bdf92ff2f169871e320394f7c0a", "f1a9e0830bc36c048fa4659beaa62609869895b5", "382d3b37f53fd6118ab979cb56f7f3d13eb0951d", "1b5db3170c195508ff24fee8eda0d4987e806f0b", "57e7af0b69325fafb371ef5d502e39ef9c90ef7e", "3556722b4703a21abafd2f9388743202943f4503", "f0c31511134abdd23f990310e8a2f2eb3a629b62", "b7d12aec8a0152ec4921dfa43ab525a63b334385", "a1f8082505c7e90b0a033e1b9da0a97d67aad66c", "d8e9f8c8a37cb4cd26b92ad0d942d641cd512644", "d6045d2ccc9c09ca1671348de86d07da6bc28eea", "acbdbf49f9bc3f151b93d9ca9a06009f4f6eb269", "814a4f680b9ba6baba23b93499f4b48af1a27678", "c21a4d70d83e0f6eb2a9e1c41d034842dd561e47", "00e6ca75129dedd275c272ac934ef553ba1bc8a1"], "hybrid_score": 0.5752, "source": "semantic_scholar"}, {"title": "C2T: A Classifier-Based Tree Construction Method in Speculative Decoding", "abstract": "The growing scale of Large Language Models (LLMs) has exacerbated inference latency and computational costs. Speculative decoding methods, which aim to mitigate these issues, often face inefficiencies in the construction of token trees and the verification of candidate tokens. Existing strategies, including chain mode, static tree, and dynamic tree approaches, have limitations in accurately preparing candidate token trees for verification. We propose a novel method named C2T that adopts a lightweight classifier to generate and prune token trees dynamically. Our classifier considers additional feature variables beyond the commonly used joint probability to predict the confidence score for each draft token to determine whether it is the candidate token for verification. This method outperforms state-of-the-art (SOTA) methods such as EAGLE-2 on multiple benchmarks, by reducing the total number of candidate tokens by 25% while maintaining or even improving the acceptance length.", "year": 2025, "citation_count": 6, "paper_id": "dfc4715a02fe52360d7de95f9119ad6949a1c599", "authors": ["Feiye Huo", "Jianchao Tan", "Kefeng Zhang", "Xunliang Cai", "Shengli Sun"], "references": ["eee1086f23ea9f14d90fe9365ffabb28db7e2d0f", "3c71cf8939d8956a08c215c00b0b6981f531c820", "88a8d366eaf6a63829ad1f4ca1a9ce161427ef83", "cab58a0263d454604896dce6b8fbf4df1dd99ff0", "05bde17d7cfe69fff3ab574d2521b9d806fc901e", "050d66cab131c8e10c230df9469d897622ccf7d2", "ab16bf2223fcb8c749a52ee3cc0496dc5bb0b4ec", "1b5db3170c195508ff24fee8eda0d4987e806f0b", "57e7af0b69325fafb371ef5d502e39ef9c90ef7e", "a0a79dad89857a96f8f71b14238e5237cbfc4787", "f0c31511134abdd23f990310e8a2f2eb3a629b62", "163b4d6a79a5b19af88b8585456363340d9efd04", "57e849d0de13ed5f91d086936296721d4ff75a75", "a1f8082505c7e90b0a033e1b9da0a97d67aad66c", "d8e9f8c8a37cb4cd26b92ad0d942d641cd512644", "d6045d2ccc9c09ca1671348de86d07da6bc28eea", "acbdbf49f9bc3f151b93d9ca9a06009f4f6eb269", "90abbc2cf38462b954ae1b772fac9532e2ccd8b0", "dc52b09089704ebd6f471177474bc29741c50023", "10eda4521c032adabaa8e70d6569e17370b29dcd", "17dbd7b72029181327732e4d11b52a08ed4630d0", "f37076f426023241f19cdc2fb0a0fd733a6fa7fa", "843a1567b056c8a1d0deddc8b699e1725194f85c", "da6f13c10f1675fce96398da0c83c39f798aef66", "9405cc0d6169988371b2755e573cc28650d14dfe"], "hybrid_score": 0.5058, "source": "semantic_scholar"}]

data/cache/c31d37d8cf517dcbeac07b62c0ef2a9d.json

@@ -0,0 +1 @@
+[{"title": "Spiffy: Multiplying Diffusion LLM Acceleration via Lossless Speculative Decoding", "abstract": "Diffusion LLMs (dLLMs) have recently emerged as a powerful alternative to autoregressive LLMs (AR-LLMs) with the potential to operate at significantly higher token generation rates. However, currently available open-source dLLMs often generate at much lower rates, typically decoding only a single token at every denoising timestep in order to maximize output quality. We present Spiffy, a speculative decoding algorithm that accelerates dLLM inference by $\\mathbf{2.8{-}3.1\\times}$ while provably preserving the model's output distribution. This work addresses the unique challenges involved in applying ideas from speculative decoding of AR-LLMs to the dLLM setting. Spiffy proposes draft states by leveraging the dLLM's distribution itself in an auto-speculative manner. This approach is efficient and effective, and eliminates the overheads of training and running an independent draft model. To structure the candidate draft states, we propose a novel directed draft graph which is uniquely designed to take advantage of the bidirectional, block-wise nature of dLLM generation and can be verified in parallel by the dLLM. To further optimize the structure of these draft graphs, we introduce an efficient, offline calibration algorithm that procedurally determines high-quality graph configurations. These optimized draft graphs, enabling increased acceptance rates, lead to a significant boost in the overall speedup achieved by the system. Crucially, Spiffy is also complementary to other recent innovations in improving dLLM generation speeds such as KV-caching and multi-token unmasking. We demonstrate that when combined with such parallel decoding algorithms, Spiffy is able to effectively multiply the benefits of these methods leading to total speedups of up to $\\mathbf{7.9\\times}$.", "year": 2025, "citation_count": 12, "paper_id": "2a9c37efd3b943e58f0cf56ee91c9ff7894546cb", "authors": ["Sudhanshu Agrawal", "Risheek Garrepalli", "Raghavv Goel", "Mingu Lee", "Christopher M. Lott", "F. Porikli"], "references": ["b376f5b0b65061b621120acdaad75318fdb747d8", "b04ee64c23a26254b63c4833946becbe36d1ad8e", "6de03206638d7d43c4142a1dfc891849fa0ea696", "81244a5f36534b41d6265d259082933d951c503e", "f8dcb5c2f1a90806459d7bed4410a27b475c78ec", "5e9ffdd179df49be1129e32ae75fc89a2b68e676", "c6f896aa698b2d65160372bce057ea5f081904de", "773fdb9a909abe5065262c94b873572dc9eb7e82", "a24b410204822d4faf61b9f3135be6ef39c5a617", "868c3143efe5c974025fb887359ee4eff0c14981", "b311c913942e8d1300c545a77a262e0d34e952b6", "482971637aa393e90b11589f5ff7ddd4d82c6284", "27848e58b581e66db0b21c170ef61d62165f3d93", "0adf7d0d104f59189ee082442d595c4eaa094904", "cab58a0263d454604896dce6b8fbf4df1dd99ff0", "f8d357d38bbcdd93889fe71762eb57842b2ab063", "1bed8c7541381b1f79027c240b64c9276573fc3c", "41a66997ce0a366bba3becf7c3f37c9aebb13fbd", "2dce617a4a1908a4047ae970f6e003ec6b8dfc53", "582d172769d5774774c1dfd7546b829248d5adc1", "f1a9e0830bc36c048fa4659beaa62609869895b5", "57e7af0b69325fafb371ef5d502e39ef9c90ef7e", "ce806f8d32f6fb1eaa821248a1bc4fa2cd949fbb", "f0c31511134abdd23f990310e8a2f2eb3a629b62", "a1f8082505c7e90b0a033e1b9da0a97d67aad66c", "d8e9f8c8a37cb4cd26b92ad0d942d641cd512644", "87c5b281fa43e6f27191b20a8dd694eda1126336", "9695824d7a01fad57ba9c01d7d76a519d78d65e7", "a38e0f993e4805ba8a9beae4c275c91ffcec01df", "acbdbf49f9bc3f151b93d9ca9a06009f4f6eb269", "57d1e7ac339e783898f2c3b1af55737cbeee9fc5", "633e2fbfc0b21e959a244100937c5853afca4853", "5c126ae3421f05768d8edd97ecd44b1364e2c99a", "90abbc2cf38462b954ae1b772fac9532e2ccd8b0", "2dcef55a07f8607a819c21fe84131ea269cc2e3c", "0fc5a4f52a53f7d7809b7782a2aeb96da5ec6fd1", "df2b0e26d0599ce3e70df8a9da02e51594e0e992"], "hybrid_score": 0.5917, "source": "semantic_scholar"}, {"title": "A Pipelined Collaborative Speculative Decoding Framework for Efficient Edge-Cloud LLM Inference", "abstract": "Recent advancements and widespread adoption of Large Language Models (LLMs) in both industry and academia have catalyzed significant demand for LLM serving. However, traditional cloud services incur high costs, while on-device inference alone faces challenges due to limited resources. Edge-cloud collaboration emerges as a key research direction to combine the strengths of both paradigms, yet efficiently utilizing limited network bandwidth while fully leveraging and balancing the computational capabilities of edge devices and the cloud remains an open problem. To address these challenges, we propose Pipelined Collaborative Speculative Decoding Framework (PicoSpec), a novel, general-purpose, and training-free speculative decoding framework for LLM edge-cloud collaborative inference. We design an asynchronous pipeline that resolves the mutual waiting problem inherent in vanilla speculative decoding within edge collaboration scenarios, which concurrently executes a Small Language Model (SLM) on the edge device and a LLM in the cloud. Meanwhile, to mitigate the significant communication latency caused by transmitting vocabulary distributions, we introduce separate rejection sampling with sparse compression, which completes the rejection sampling with only a one-time cost of transmitting the compressed vocabulary. Experimental results demonstrate that our solution outperforms baseline and existing methods, achieving up to 2.9 speedup.", "year": 2026, "citation_count": 0, "paper_id": "da0c32f1e9928b077746a36440b2e164b979488f", "authors": ["Yida Zhang", "Zhiyong Gao", "Shuaibing Yue", "Jie Li", "Rui Wang"], "references": ["e6a2a2fb59b38efc2056e68d7168b0ba714c22fa", "95c8599867bc517b4e7e0a4c8f07dd2352292e18", "0646bf8db2b75dd1d45e0f04104547710d02aa1b", "55a28c28a637ffc9367c138f79e71a33f9e0e6cb", "c4564a5af0ea4175f6cc3bf8d1d8ef065e3804d6", "e505bfe85148928e6486c25b8acbbea80ff3f8b7", "1d403457e3d2b1991cdc5804dd41b926cb046714", "913aebc562b499b74ee8248bd6b1806ebc40c3fe", "007e7d6cb64a7dfcdcad9fc6f4f0ba69fdc88203", "4672b31e40ca2e5f792b20f23a4d89ea498b384e", "621ba0b763e89a4e82fac4b4e264ca8fb6c04fb6", "c4a44781977a529fb909850bc588906b3d312975", "10c0da8955c4b1067fd09723ebadf665fc602d1b", "0a61e5f8c26b2463e1b6b1d51efee8e953334c68", "b9203f3844e850ed9096bd935de336405acb905a", "ace3d2bcecc658a97632c9be4ce461450271eb5b", "c80a3847fd13b06e1b0ff4e59c4bb3df06bd088d", "97b6f4357d1e3ab40a7ee60acb5260a948e3641d", "1b5db3170c195508ff24fee8eda0d4987e806f0b", "57e7af0b69325fafb371ef5d502e39ef9c90ef7e", "a1f8082505c7e90b0a033e1b9da0a97d67aad66c"], "hybrid_score": 0.532, "source": "semantic_scholar"}, {"title": "GoodSpeed: Optimizing Fair Goodput with Adaptive Speculative Decoding in Distributed Edge Inference", "abstract": "Large language models (LLMs) have revolutionized natural language processing, yet their high computational demands pose significant challenges for real-time inference, especially in multi-user server speculative decoding and resource-constrained environments. Speculative decoding has emerged as a promising technique to accelerate LLM inference by using lightweight draft models to generate candidate tokens, which are subsequently verified by a larger, more accurate model. However, ensuring both high goodput (the effective rate of accepted tokens) and fairness across multiple draft servers cooperating with a central verification server remains an open challenge. This paper introduces GOODSPEED, a novel distributed inference framework that optimizes goodput through adaptive speculative decoding. GOODSPEED employs a central verification server that coordinates a set of heterogeneous draft servers, each running a small language model to generate speculative tokens. To manage resource allocation effectively, GOODSPEED incorporates a gradient scheduling algorithm that dynamically assigns token verification tasks, maximizing a logarithmic utility function to ensure proportional fairness across servers. By processing speculative outputs from all draft servers in parallel, the framework enables efficient collaboration between the verification server and distributed draft generators, streamlining both latency and throughput. Through rigorous fluid sample path analysis, we show that GOODSPEED converges to the optimal goodput allocation in steady-state conditions and maintains near-optimal performance with provably bounded error under dynamic workloads. These results demonstrate that GOODSPEED provides a scalable, fair and efficient solution for multi-server speculative decoding in distributed LLM inference systems.", "year": 2025, "citation_count": 0, "paper_id": "3af15d52f3b08aa30b9e387aaf69eee4cf252546", "authors": ["Phuong-Nam Tran", "T. Liu", "Long Tan Le", "Tung-Anh Nguyen", "Van Quan La", "Eason Yu", "Han Shu", "Choong Seon Hong", "Nguyen H. Tran"], "references": ["6e7d41e93e262b076b4f44201330980d48dbec1e", "0646bf8db2b75dd1d45e0f04104547710d02aa1b", "c4564a5af0ea4175f6cc3bf8d1d8ef065e3804d6", "b8bd9b77b75be01c28cd57419cdf53212cdaa8ef", "bacdf9671fb872287201b53d768df89b4d6630a3", "6a111d13b9c272459fb3d7628a7b0307c8ebd0f7", "84c6a4798375d689a2d37df0569f6f40516aeb2b", "d2421cffac277e230cb97fc2355b32e03dd8bb1f", "20f090e35ad598fba2404e550c2462dc9da03a10", "f1a9e0830bc36c048fa4659beaa62609869895b5", "1b5db3170c195508ff24fee8eda0d4987e806f0b", "717bc487c987470e063ae92771e910da29ad77c2", "21e53e51ff77a5f34f43cb8ca029909c3ad9f71e", "57e7af0b69325fafb371ef5d502e39ef9c90ef7e", "72f77a393079431e4207b3afe678ee80b420e6f8", "0cee098244c9978032702862a43a09f468f691a4", "3eec0c1a7dc0d364d23e2e4544bf8772f5f8ffa3", "ad9146d98ae95bbeeef460abe083ecc2c4798672", "bc5c73c101da795cfa44e4ac7751cdedca9b6d93", "02ad9f3fefe33cb9ca546591bec65dbdf7766c80", "83b90f4a0ae4cc214eb3cc140ccfef9cd99fac05", "f0c31511134abdd23f990310e8a2f2eb3a629b62", "204e3073870fae3d05bcbc2f6a8e263d9b72e776", "363668677c459ebc0ff494655f993a93a0251009", "9d7a75601e0e50dd68d40cfb8ef0e891dad797a6", "cd8231a86589d205c64167ad462671c357f56d8d"], "hybrid_score": 0.4924, "source": "semantic_scholar"}, {"title": "Collaborative Inference and Learning between Edge SLMs and Cloud LLMs: A Survey of Algorithms, Execution, and Open Challenges", "abstract": "As large language models (LLMs) evolve, deploying them solely in the cloud or compressing them for edge devices has become inadequate due to concerns about latency, privacy, cost, and personalization. This survey explores a collaborative paradigm in which cloud-based LLMs and edge-deployed small language models (SLMs) cooperate across both inference and training. We present a unified taxonomy of edge-cloud collaboration strategies. For inference, we categorize approaches into task assignment, task division, and mixture-based collaboration at both task and token granularity, encompassing adaptive scheduling, resource-aware offloading, speculative decoding, and modular routing. For training, we review distributed adaptation techniques, including parameter alignment, pruning, bidirectional distillation, and small-model-guided optimization. We further summarize datasets, benchmarks, and deployment cases, and highlight privacy-preserving methods and vertical applications. This survey provides the first systematic foundation for LLM-SLM collaboration, bridging system and algorithm co-design to enable efficient, scalable, and trustworthy edge-cloud intelligence.", "year": 2025, "citation_count": 7, "paper_id": "bf96193f2ee46de6a9efbd859c7ce7cb6338b31a", "authors": ["Senyao Li", "Haozhao Wang", "Wenchao Xu", "Rui Zhang", "Song Guo", "Jingling Yuan", "Xian Zhong", "Tianwei Zhang", "Ruixuan Li"], "references": ["fdbaa2e1fa498b6cf2fd1c527171d74cfd044152", "b8cc53967dadb781089f86a7269bf1bbf3e1f88d", "19d9ac9c27caeefe08d9f1456ad91bcf6ada491d", "d65c0c187a22e608959f068544cdb0f407fbafd9", "9cdf086dc8ffe763795784d04c32a89c54923cea", "dd255ad3e569c9a8c28f8fb510f9f91323cb3b66", "1d403457e3d2b1991cdc5804dd41b926cb046714", "07c7ca5067516f20f975edf0b47866be70371bc4", "44c733e11c1de985f9cc31102a7e3d1a284421a2", "7b08bc3b8616aca3d50521785c893ffbcea629df", "eb5b950cce9451420992c760b7a5d3c72df045c9", "15e2e8f364791127e9ebf9888145852d6691b7a4", "913aebc562b499b74ee8248bd6b1806ebc40c3fe", "8deae9abf52e4edc36f287bcf4f2be17dce05435", "09c0c73525e05e5679db20c20f5faf87ed02d600", "c5c6efb7e1d3939d15a8a32b4a657a701d47b2b1", "007e7d6cb64a7dfcdcad9fc6f4f0ba69fdc88203", "69f69f43e31af4b4a90b2cc7ed07a07e36a0e6db", "cac086d84f2f6d9d69f9eeaa03a6ed8a92c98be3", "d31afa9013658209ad3b2b6b1f360efa27aee4c6", "355b7d2f22fe75fb4c8fd648488cda3747f99ecc", "a8891193008761315371767616ca1abb037fc9a6", "f4ecdaebd24583e0fede6b1c5648008c659d1486", "4d68752988f900e6ad3b0feefe08d02b1da2ee67", "113af95d76c2fc38a5e0fbf66ee207d55e33f594", "1c2b518a345102344e9b7fd1a530338cd23390b1", "1a3ccb6b39a27831c043d8d553c7b0726765383c", "477408d5a9ddb1c1dbe70a5aeb9235347498ac68", "8132f76f1090659317e6a3067f13222439571ff3", "ce204ce63bfceff54706a857d67658361739f5b0", "3a24f3f07a493b5e61c04dc0bf0c1ed249ff93c6", "6970cf505da8e287dc1f1c7d1aea00e86f537457", "b8bd9b77b75be01c28cd57419cdf53212cdaa8ef", "a02b039843dc4fe5d2d2fb29def6d414519541ed", "1ed4b7d14f6366fda775b4368b8dac3aaa4aaacd", "4e8cf839d45eaf804dad9186ebcab0a81ddcea7f", "c71fb42f7772536810863391e33980b1a89aabff", "c110e21884b7167b0ac07cdee9bd77e42ec68bdb", "76bc4ccc7506c51e28225593f94d437d7f88ceb3", "54ade47ed89a74f042004da95d671f22fde2917f", "62fefe643ca668e76ff8f3bfdf736b4fa587b5f0", "1db441962bd29f5c2d32e2770f9195d8200d1f24", "15cb988186158504d962a38b2ae033a45e9bb923", "ba54814cbddc40b7eebcb9fe47ad654ed15bbd85", "a9af0c5a86357fdcb4ab20e4223ea4c49a353814", "544b3155834e1cd7499ad2c4e37bb8058f181a18", "00c09022ea9aff6d4b99df0a32245a2a1a43b8a4", "8b400a2c6b203504b4162aa3ba78ffcf928e5be4", "12811af37a00d1fbe212096f044a0d671cf5b639", "2617cc5165b058bee846687057fe5b523b244d3f", "cc1adc975d62a3559fa363fed4f8338bb0d05397", "921c0d3676a76e7913a489d08f5685a02755a381", "10540a03e0182aa37c1bb4202eac1c2392c45ac4", "7099d7a8ee2c95ecbe0e32f6b40ffa65857fcdfe", "7ccec1f8e932ba24db94b772508e26b8182b84a3", "ab629e13734a7989b243a09d18e4d00b37cc8288", "17064868cd09e9c04d04b616a340a1b223cd45ea", "6d8647203161ae4b700138f4ef1f6d4e39648c4c", "b311c913942e8d1300c545a77a262e0d34e952b6", "133269ffc2391a2e2bdbab72a2ace89d7e78a107", "85488af8db54f454f132315aad9f743301478d49", "4f62ac743e2ed33e682a9c6f5f7875723d9a7025", "c9eebab6e90df58f77471cc48e4966f16ea05b0f", "110b77ec65f5458456dcd28fbb1fb4ecbdcaf6a6", "88aa6b1f37d1fd8e0a40499ce9bb87873f03aaa8", "1567d7739436fe14ad422d89bc7567d22b748744", "62c35559859ad9bd0a14891a2901f4f47d50ccaf", "a89978394b3b03ef512d1445d18ef09f741277f4", "75ed6e57e9d3261a19aaac8b101e4d2ea8307cbd", "10648ea5081116197b372103d53f4391cfb9b73c", "4b82d9fbaf5427ff3799de777ffbe3efbf98b81d", "a5634d6ea895179fd444a1a54f82ca1e66fe6463", "5de46554639b42cb89126270f4723497ea6f7605", "e7ebbf0f68620ff7382f1d85c3c32b9e779137b5", "fd3230251854b8b1a06fd7fa201317daf77f0e80", "0fec47ed6fcdcda90d9bd242e084a16cd10ddadd", "52a0ceaf916deff733410f07e05215826bb1c86d", "b76ba4bb25a195ad3593f548f0d7db0f603b078a", "9ec2bb4eaeac522b4131d7fff2dd16967d48be25", "7943ec4a67151a559b25cd34369e661c9a7924c8", "ccfa7642a0e6eac2b7c57b670785c0c12122eee4", "4385314ade0ee492d99bc03c1e8ea98ec577c44c", "05e3853b121aa8ca0dfac15abade8dd1537d886a", "bacdf9671fb872287201b53d768df89b4d6630a3", "f9ff028e460d471687f6012b983b429c2348a483", "79172e64a8b7abb8523b0fcf44802b9e1142aede", "aa1127172e8b01d94140357e06f31955dba47dab", "3bd3bbc9a7136a46ceb54a1112e336c7c6e725ed", "95031a49f9a8843558b882a58b7d9cb9c3baa6c6", "05b122fdd00274a0d66afdbfbd00353ddbcf5ed2", "267aee33599af850046ffa4c5cf0157bb43a3976", "b3a8d14b390826dc0b9a5531b73dabf0a2dcfa9b", "ace3d2bcecc658a97632c9be4ce461450271eb5b", "5cb208b2f96320192d7c17e2970b3d93282a3371", "f7dfe3153ce6f71b08dd3bde4668a336c8f71ab6", "9af8bccf3e42996cbb198a6ceccafa2a084689f6", "8fe2f0f8c135628a9d976eeeae524cc3d3cf5752", "160924af0791331ec8fa5a3d526ea125355f3b8b", "f0a55c74ed88795addf73995ea6c9e9f477b963e", "92e3b209fe9cad9078810e225f6b6b8b391f304b", "e1f097602db09042f7d38c156dadaa2c274a5a33", "6774b88e59609970cd43b98df73310a425090474", "ba4c9e0e808c63e7d40a60c84abb534a41e1f1e9", "540209de237977a2fb16d3a2c4021618e8ea8b47", "9b3239cff17327960804098e33e1ca903e7b9e85", "1b5811eafe28b50c08640189483e9b3ab46a10de", "9e015a8524261d4c040c10fa773ddbb02a49ee38", "29c67918f2a4dfd167e4d4b878defad093dd3824", "c7f9706898bdfa3241601e075b1305649b174ff1", "58636fabd0e7f9a4921732276f240f53c4818052", "a09d1f58d39550601c0e40ece1fbccf7c1c474d0", "c77f49a9db193060a29889d720bb243d680e5455", "4d8bfd68c9d1eb514cb55583a1ad1f6dce969083", "7f57648795282dddc04a6ccd8b6eb1def114f727", "7e206ea4c8b42f67d66c5205891b06d2af612667", "3bce09c68236ec61f1bd3cbfacabe45bba8c304f", "05bde17d7cfe69fff3ab574d2521b9d806fc901e", "6f33cb9ccf10f3862b392823e76d356b1e3b494b", "c80a3847fd13b06e1b0ff4e59c4bb3df06bd088d", "07d46677c31a450fecbecd3e96b096fe250031b0", "197482f191ef8bbb9c73e5e18289e736af59f709", "120780c2bf33793f871d1b706288203d618f2d1f", "5f4364f7138f9525df291b16799d1128dd3b8fad", "30394de373b65aeda84e2bd100e8efe38f4d1a8c", "b9a941c9b2c9becb07510f2cf1274678ef0cfcab", "3c465f37b0d7b4cc51c22f34c85ff3fe0f7035ca", "575d626c47d1b6b74246d7ce168a64f6bcafbe4c", "bdd842c7eb21da2f3eadbbab722adca5a92cbbaf", "4047a1a6bca0d13ad185b4e425de011b99300511", "444aa31192c87f996bb01fa856cb765a19cd5323", "980bb79b0292b033b36511354b02e8d1fb8ae16c", "1ae01a0bb36ca10890e055a1f484dc8686b9cbd6", "0e0918bd63e0b1e985fed5bab05ba6935e52e0cd", "eee108b3e51e89d160d1116935e062c4d169b475", "c17d6af40861a705a8e5e60aaf54762f43b787d8", "abdceff7d7983cdede9a5aabe6a476d4c72e41a3", "97b6f4357d1e3ab40a7ee60acb5260a948e3641d", "db229da82d4821446fb14d084ec58da7d964b7ce", "5c02e6394b8f271b027e4e216b01ed86baa0679b", "840ea9e7883ccd276ba26f97b76072686ce6be30", "0d81434f6a3fca88774ab058ebabd2206be70016", "5b46ebad43c6bbd45249fd1490b9bd7a86d1acda", "69239ee34b04c5503774ffc0235bc1b239cca4fa", "1778f1b36d07a6a0166de4c91f2f2e7a2a225f84", "f513a29c5f48d7dc246514e0bc64f03b67dcaf1b", "a915b3b2d0fdfefbf038a8746ceaf001859f3431", "9177fe8ad9ff7754c2fd27e805fe6ed882fe2954", "3c0deeefd5de5aa8ae103faf57a482281b096bc4", "ab16bf2223fcb8c749a52ee3cc0496dc5bb0b4ec", "60f066b3d7391dc5da3e3638970fd00f1aadebdf", "df5374163885b33e55bc18040b96d563d96e74eb", "4623901ec1f8311b32e27bd627b2bce56161c6bb", "602f041b7ba9f1bc0bd30d9f26c8227026e812be", "75264034416e5c2f552efb5783624d240e0ea4f6", "3df6c6cd13e49e637ced0ade0daa87738907a377", "8287fbf2c520f7604aae19a590ea9494d356aae7", "6a4cc1a573e1db5cbc8b1c3feeee193587f708fc", "d244b2ba0fd5f573d19df0c7b362692c018748a1", "13b8934468665ecb586f491d7f9f6c460cb095e5", "b28ad67a90bea98eafefe6259b888c1d75b2ccbb", "5d1e6d6e85212aefd2815c66e72aa76b5b8509a9", "382d3b37f53fd6118ab979cb56f7f3d13eb0951d", "5bbc2b5aa6c63c6a2cfccf095d6020b063ad47ac", "0cee098244c9978032702862a43a09f468f691a4", "4d7da69c61db94386424675d8fa01fb0a279f3b5", "e6f480a7d45e2447fa5e9988064735b19433c301", "efc5e94635a850ede9c1f8dbce65d5dc536f3bfb", "d6af8cab4ced2d3030e7c07d30c5273eed54014d", "6a04b7d0206552d7c657a1e673ad7274b7cd9935", "c3bacc93d8c0c6ed31763d0c0fc7951f0ab82ed8", "5acbf917da5be89e4eebd7e98c81d87069450a5d", "339b1e148d48a77d4a79042a4347d6e5863cb931", "dcfff69ae4aa905818081bb274c0ddd6b45ed208", "6b97aa78bcdb88548c44e7e1671c0ed37ed37976", "95d65e13b36006113ef3ba93200783baef07d2c3", "89d3d19ad717cd534bdd1866d28e2da253147705", "e8d16ad7da923c8b7c09b291a1b997ec62a9a3bd", "b10d8f2913a18f90561e616881a42ae1f36d74c9", "c37b0bed188bc723a18f0eaffde9a4e278dc78c6", "215de09ac6e5de81187c85065b5ace8bc01f2862", "ea1f648988c632a6dbab6d8b88432456aa021cfb", "f0ed191bdcbd56610ab4d95c5a80d5ac31ffc074", "f1019fe7918ee51cf16c91186e769a71a9d9fb42", "ddbd8fe782ac98e9c64dd98710687a962195dd9b", "92678d487e0f3a85c024d656b67eff49af87a9fe", "56767c18bb5aaa2b6377624168bed1b6dcc4b94d", "abdb0f9d1486dbb024c4bc9f8f9dc40464c58715", "c803b2236136fe7c59ea3619bc7520d331568d32", "564855d475ed9197dd7516594557ff886ff623e5", "8f72127459ade06831ec2990f7da4914cc1a6e22", "00cccb9065f0a59e845d5b4d360ce31cf25036be", "00e889fcfaf4396a20f37f681cf8b14f3e878879", "cb23a59fdf3ade707600f076df4ff27a03941fba", "8343fa664b0baad57c194a743ad952374bf81ac1", "a0ab8c94158906ce6cd4dbe99e2df292d70a03f5", "ad97671a924a9b3a060fee857e561f140ec79dd7", "338d8f3b199abcebc85f34016b0162ab3a9d5310", "ec58a564fdda29e6a9a0a7bab5eeb4c290f716d7", "5794330b785403277707d937e56423e9b5b0c31b", "43e624ddeed82df944a6cae0dedec3372438e243", "a53c8ba374d430d6c3786d13c04edb200d547750", "4c55dd37ab4c41565226a3c8194166d7412b0a02", "378406966c3edb7d31a9637093fdee07d81cddf9", "8c4f1c51bff76d736c1962ed5ae860209cedf958", "4fa53635c9dbd4b16eac9e4cdd88dfc846ff8121", "5ca3811e3df16a9f17d7563c46c2f0b61513cb92", "6bd3ee1ca608bc66a490f63f2fb107d79b44f3e2", "6f821d75968bc8de070af3ce5aa7f57bc031fafb", "c17b71f31fa708eb01310ff65ab660f2676a12a1", "f0c31511134abdd23f990310e8a2f2eb3a629b62", "dd7f28d93dc2ec3cbc1cc66c3443c1c17105f1b3", "585f8b9725f5f5e5495c3508d39f70d1c053e190", "aad167be3c902388ea625da4117fcae4325b8b7d", "b6c082fff014e9d8f960b3783934c3634ced6c42", "f8e77bd3d573d0daee0744443c65c40e3b5dc10f", "0ff42fe5e4eaf2488c62ebf88c2de3e84c312b7a", "163b4d6a79a5b19af88b8585456363340d9efd04", "b7d12aec8a0152ec4921dfa43ab525a63b334385", "67433255d3737e2718eb6e8e4a8079a73a546df1", "53d128ea815bcc0526856eb5a9c42cc977cb36a7", "e4c3dc16e971c82da57c9bba8a687f53c0d986d3", "022229b056cbff623fd947f32da33c92f6bd954b", "d8e9f8c8a37cb4cd26b92ad0d942d641cd512644", "7d645a3fd276918374fd9483fd675c28e46506d1", "5da5c2c86ebd7cc561c69bcd4318c0a9b3b1a8d3", "af11f2f2fc5b9a8d3b8d03aedd2007af7731882c", "52eab50953077c78991a9c045483664086fe37c7", "3010f6eef370d7aef6b545ad7e02682cb1e6a945", "4d2b27732aa3e12aa63cc92ee22d6e61c7d60f07", "1e4eecf536f26e78568cdc10743ac8c1588fa6a7", "c077e8f9ab701270dec08ad8934901474a46ecf1", "218c5c69f3cf0c158e9b6af239a2cc62a688c6de", "157bddb85786b980c962e649bed33d8ebbf7b4a1", "444b7ab50d4b4dd7d5e290af0d416fd598ece9b9", "f356b19619e2d9ebeb96839f0e87a07071248179", "a55c399bbb0382650459da59fc545f2dd275012b", "50796b0f3edf9cb5ff1e447c298b33755378aa4f", "a199a03e11b68c4132be880b5fcabc57251bc477", "c5c4142a01981787a71bf6ebcb791520c458ab5d", "e740a2b706fcae34850fd0e56619a2df7ee4dce7", "79cf9462a583e1889781868cbf8c31e43b36dd2f", "a3ce3004a0eade48a3ae652dbf5c04a60c2416aa", "8dcbcaaf337d7bd22e580f1bb7a795ed4bb604fd", "3e804ac49b99c6509a65ba81d11ba911bac81596", "6c7046195f64cccac1ed3275d88d77655534b5a4", "dba197afbe1cf422323eeab4f871756a7625d913", "ea313326f4dbb4d80aad9f92acc969129bf3d98b", "ba0f0183255afb920a84720b268b4f4bf78259c0", "c8e27027af1a739be70f1fa43839849e9efb101e", "da6f13c10f1675fce96398da0c83c39f798aef66", "f7f7586d901242624a6f33f129e2074fd853eaa8", "5b6fdb2aea424d02c141da81d62b04d739d62b96", "3790f7f19eeccbc4cccc02fa8b0e02bca841f907", "f21286d8b66ac011793472bea466c4e3060b6d97", "5b9c17ff9266dd72a03fbe9a2bd8df07b5eae10e", "321876e6fa45cb5b3bccf0d3f2271a81b1c7daec", "c55ae1dbc32d6908c9219191003ccce66a2d4eba", "a39d7b3a3cb29b23fa98bcced73a031a9e389c5f", "9118ad81a145313cef0b31d3e3a3c8213e75e491", "fb3dc58cb17c54b997e6301cbde7773f77427833", "c7822cdc35ad788ec87e14b3a9d45010f1f86c38"], "hybrid_score": 0.462, "source": "semantic_scholar"}]

data/cache/d86c06d1a426d4018caa471ce933586c.json

@@ -0,0 +1 @@
+[{"url": "https://arxiv.org/abs/2402.01528", "snippet": "SpeculativeDecodingis a widely used technique to speed up inference for Large Language Models (LLMs) without sacrificing quality. When performing inference,speculativedecodinguses a smaller draft model to generatespeculativetokens and then uses the target LLM to verify those draft tokens. The speedup provided byspeculativedecodingheavily depends on the choice of the draft model. In ...", "title": "[2402.01528] Decoding Speculative Decoding - arXiv.org", "inferred_year": null, "hybrid_score": 0.0, "source": "duckduckgo"}, {"url": "https://proceedings.neurips.cc/paper_files/paper/2024/file/e7349e785900b93d8b4971a3f2c1cefe-Paper-Conference.pdf", "snippet": "Thispapertackles this gap by conceptualizing thedecodingproblem via markov chain abstraction and studying the key properties, output quality and inference acceleration, from a theoretical perspective. Our analysis covers the theoretical limits ofspeculativedecoding, batch algorithms, and output quality-inference acceleration tradeoffs.", "title": "PDFA Theoretical Perspective for Speculative Decoding Algorithm", "inferred_year": null, "hybrid_score": 0.0, "source": "duckduckgo"}, {"url": "https://aclanthology.org/2025.naacl-long.328.pdf", "snippet": "Thespeedup provided byspeculativedecodingheavily de- pends on the choice of the draft model. In this work, we perform a detailed study comprising over 350 experiments with LLAMA-65B and OPT-66B usingspeculativedecodingand de- lineate the factors that affect the performance gain provided byspeculativedecoding.", "title": "PDFDecoding Speculative Decoding - ACL Anthology", "inferred_year": null, "hybrid_score": 0.0, "source": "duckduckgo"}]

src/graph.py

@@ -0,0 +1,224 @@
+import logging
+import time
+from dotenv import load_dotenv
+from langgraph.graph import StateGraph, END
+
+from src.state import ResearchState, Verdict
+from src.memory import init_db, load_session
+from src.agents.planner import planner_node
+from src.agents.retriever import retriever_node
+from src.agents.critic import critic_node
+from src.agents.synthesizer import synthesizer_node
+
+load_dotenv()
+logger = logging.getLogger(__name__)
+
+
+# ---------------------------------------------------------------------------
+# Routing function — decides what happens after the critic
+# ---------------------------------------------------------------------------
+
+def route_after_critic(state: ResearchState) -> str:
+    """
+    Returns the name of the next node based on critic verdict.
+    PASS / FORCED_PASS → synthesizer
+    STALE / CONTRADICTED / INSUFFICIENT → retriever (retry loop)
+    """
+    verdict = state.get("critic_verdict", "")
+
+    if verdict in (Verdict.PASS, Verdict.FORCED_PASS):
+        logger.info(f"Routing: {verdict} → synthesizer")
+        return "synthesizer"
+
+    retry_count = state.get("retry_count", 0)
+    if retry_count >= 2:
+        logger.info("Routing: max retries → synthesizer (forced)")
+        return "synthesizer"
+
+    logger.info(f"Routing: {verdict} → retriever (retry {retry_count})")
+    return "retriever"
+
+
+def route_after_retriever_retry(state: ResearchState) -> str:
+    """After a retry retrieval, always go to critic."""
+    return "critic"
+
+
+# ---------------------------------------------------------------------------
+# Session loader — prepended to graph as first node
+# ---------------------------------------------------------------------------
+
+def session_loader_node(state: ResearchState) -> ResearchState:
+    """
+    Loads session context from SQLite before the planner runs.
+    This gives the planner access to prior queries and positions.
+    """
+    session_id = state.get("session_id", "")
+    if session_id:
+        try:
+            ctx = load_session(session_id)
+            logger.info(
+                f"Session loaded: {len(ctx.prior_positions)} prior positions"
+            )
+            return {**state, "session_context": ctx}
+        except Exception as e:
+            logger.warning(f"Session load failed: {e}")
+
+    return state
+
+
+# ---------------------------------------------------------------------------
+# Retry retriever — uses rewritten questions from critic
+# ---------------------------------------------------------------------------
+
+def retry_retriever_node(state: ResearchState) -> ResearchState:
+    """
+    Like the retriever but uses rewritten_questions from the critic
+    instead of sub_questions from the planner.
+    Merges new results with existing ones.
+    """
+    rewritten = state.get("rewritten_questions") or []
+    if not rewritten:
+        logger.warning("Retry retriever: no rewritten questions, skipping")
+        return state
+
+    # Swap sub_questions for rewritten ones, run retriever
+    retry_state = {**state, "sub_questions": rewritten}
+    result = retriever_node(retry_state)
+
+    # Merge new papers with existing (deduplicate by paper_id)
+    existing_papers = state.get("retrieved_papers") or []
+    new_papers = result.get("retrieved_papers") or []
+
+    seen_ids = {p.paper_id for p in existing_papers if p.paper_id}
+    merged = list(existing_papers)
+    for p in new_papers:
+        if p.paper_id not in seen_ids:
+            merged.append(p)
+            if p.paper_id:
+                seen_ids.add(p.paper_id)
+
+    # Sort merged by hybrid score
+    merged.sort(key=lambda p: p.hybrid_score, reverse=True)
+
+    # Merge web results too
+    existing_web = state.get("web_results") or []
+    new_web = result.get("web_results") or []
+    seen_urls = {r.url for r in existing_web}
+    merged_web = list(existing_web)
+    for r in new_web:
+        if r.url not in seen_urls:
+            merged_web.append(r)
+            seen_urls.add(r.url)
+
+    logger.info(
+        f"Retry retriever: merged to {len(merged)} papers, "
+        f"{len(merged_web)} web results"
+    )
+
+    return {
+        **result,
+        "retrieved_papers": merged,
+        "web_results": merged_web,
+        "sub_questions": state.get("sub_questions") or [],
+    }
+
+
+# ---------------------------------------------------------------------------
+# Graph builder
+# ---------------------------------------------------------------------------
+
+def build_graph() -> StateGraph:
+    """
+    Build and compile the RECON LangGraph state machine.
+
+    Flow:
+    session_loader → planner → retriever → critic
+                                               ↓ PASS/FORCED_PASS
+                                           synthesizer → END
+                                               ↓ STALE/CONTRADICTED/INSUFFICIENT
+                                       retry_retriever → critic (loop, max 2x)
+    """
+    graph = StateGraph(ResearchState)
+
+    # Add nodes
+    graph.add_node("session_loader", session_loader_node)
+    graph.add_node("planner", planner_node)
+    graph.add_node("retriever", retriever_node)
+    graph.add_node("critic", critic_node)
+    graph.add_node("retry_retriever", retry_retriever_node)
+    graph.add_node("synthesizer", synthesizer_node)
+
+    # Linear flow: session_loader → planner → retriever → critic
+    graph.set_entry_point("session_loader")
+    graph.add_edge("session_loader", "planner")
+    graph.add_edge("planner", "retriever")
+    graph.add_edge("retriever", "critic")
+
+    # Conditional routing after critic
+    graph.add_conditional_edges(
+        "critic",
+        route_after_critic,
+        {
+            "synthesizer": "synthesizer",
+            "retriever": "retry_retriever",
+        }
+    )
+
+    # Retry loop: retry_retriever → critic
+    graph.add_edge("retry_retriever", "critic")
+
+    # Synthesizer is terminal
+    graph.add_edge("synthesizer", END)
+
+    return graph.compile()
+
+
+# ---------------------------------------------------------------------------
+# Public run function
+# ---------------------------------------------------------------------------
+
+def run_recon(
+    query: str,
+    session_id: str,
+    decay_config: str = "linear",
+) -> ResearchState:
+    """
+    Run the full RECON pipeline for a query.
+    Returns the final state.
+    """
+    init_db()
+
+    graph = build_graph()
+
+    initial_state: ResearchState = {
+        "original_query": query,
+        "session_id": session_id,
+        "session_context": None,
+        "sub_questions": [],
+        "retrieved_papers": [],
+        "citation_graph": {},
+        "web_results": [],
+        "critic_verdict": "",
+        "critic_notes": "",
+        "rewritten_questions": [],
+        "retry_count": 0,
+        "synthesized_position": "",
+        "claim_confidences": [],
+        "session_update": None,
+        "export_md": "",
+        "decay_config": decay_config,
+        "calibration_bin": "",
+        "latency_ms": 0.0,
+    }
+
+    start = time.time()
+    logger.info(f"RECON run started: '{query[:60]}'")
+
+    final_state = graph.invoke(initial_state)
+
+    elapsed_ms = (time.time() - start) * 1000
+    final_state["latency_ms"] = elapsed_ms
+    logger.info(f"RECON run complete in {elapsed_ms:.0f}ms")
+
+    return final_state

test_phase8.py

@@ -0,0 +1,47 @@
+import sys, logging
+sys.path.insert(0, ".")
+logging.basicConfig(level=logging.WARNING)
+
+import uuid
+from src.graph import run_recon
+
+print("=== Phase 8: Full LangGraph Pipeline ===\n")
+
+session_id = str(uuid.uuid4())
+print(f"Session ID: {session_id}")
+print("Running full pipeline (takes ~60s)...\n")
+
+result = run_recon(
+    query="What is the current state of speculative decoding in LLMs?",
+    session_id=session_id,
+    decay_config="linear",
+)
+
+print(f"=== Pipeline Results ===")
+print(f"  Verdict:        {result['critic_verdict']}")
+print(f"  Calibration:    {result['calibration_bin']}")
+print(f"  Papers used:    {len(result['retrieved_papers'])}")
+print(f"  Claims:         {len(result['claim_confidences'])}")
+print(f"  Latency:        {result['latency_ms']:.0f}ms")
+print(f"  Retry count:    {result['retry_count']}")
+
+print(f"\n=== Position preview ===")
+print(result['synthesized_position'][:600])
+
+print(f"\n=== Claims ===")
+for c in result['claim_confidences']:
+    flag = " ⚠️" if c.flagged else ""
+    print(f"  [{c.confidence.upper()}] {c.text[:65]}{flag}")
+
+print(f"\n=== Multi-turn test ===")
+print("Running second query in same session...")
+result2 = run_recon(
+    query="What are the limitations of speculative decoding?",
+    session_id=session_id,
+    decay_config="linear",
+)
+print(f"  Verdict: {result2['critic_verdict']}")
+print(f"  Claims:  {len(result2['claim_confidences'])}")
+print(f"  Session context used: {len(result2.get('session_context').prior_positions if result2.get('session_context') else [])} prior positions")
+
+print("\n✅ Phase 8 complete")