Neural Lexical Search with Learned Sparse Retrieval

Want to jump in?

Check out the LSR encoder demo, text LSR Colab notebook, and multimodal LSR Colab notebook.

Abstract

Learned Sparse Retrieval (LSR) techniques use neural machinery to represent queries and documents as learned bags of words. In contrast with other neural retrieval techniques, such as generative retrieval and dense retrieval, LSR has been shown to be a remarkably robust, transferable, and efficient family of methods for retrieving high-quality search results. This half-day tutorial aims to provide an extensive overview of LSR, ranging from its fundamentals to the latest emerging techniques. By the end of the tutorial, attendees will be familiar with the important design decisions of an LSR system, know how to apply them to text and other modalities, and understand the latest techniques for retrieving with them efficiently. Website: https://lsr-tutorial.github.io

CIKM 2026 Tutorial Program

Saturday, 7 November 2026, 14:00–18:00
Sapienza University of Rome, Faculty of Engineering, San Pietro in Vincoli (Via Eudossiana 18, Rome)

Part 1 – Fundamentals (14:00–16:00)

Section	Duration
Introduction to LSR	20 min
Datasets and Evaluation	10 min
LSR Framework	25 min
Text LSR (Colab)	25 min
Initialization of LSR Methods	10 min

Tea Break (16:00–16:30)

Part 2 – Emerging Topics (16:30–18:00)

Section	Duration
Multilingual LSR	20 min
Multimodal LSR (Colab)	20 min
Indexing & Efficient LSR	30 min
Hybrid Dense-Sparse Retrieval	20 min
Conclusion

References

Part 1 – Fundamentals

1. Introduction to LSR

Fabio Crestani, Mounia Lalmas, Cornelis J. Van Rijsbergen, and Iain Campbell. 1998. “Is this document relevant?… probably”: a survey of probabilistic models in information retrieval. ACM Comput. Surv. 30, 4 (1998), 528–552.
W. Bruce Croft. 1981. Document representation in probabilistic models of information retrieval. Journal of the American Society for Information Science 32, 6 (1981), 451–457.
W. Bruce Croft and David J. Harper. 1979. Probabilistic models of document retrieval with relevance information. Journal of Documentation 35, 4 (1979), 285–295.
Miles Efron, Peter Organisciak, and Katrina Fenlon. 2012. Improving retrieval of short texts through document expansion. SIGIR ’12 (2012), 911–920.
Thibault Formal, Benjamin Piwowarski, and Stéphane Clinchant. 2021. SPLADE: Sparse Lexical and Expansion Model for First Stage Ranking. In Proceedings of the 44th International ACM SIGIR Conference (2021).
G. W. Furnas, T. K. Landauer, L. M. Gomez, and S. T. Dumais. 1987. The vocabulary problem in human-system communication. Commun. ACM 30, 11 (1987), 964–971.
Sean MacAvaney, Franco Maria Nardini, Raffaele Perego, Nicola Tonellotto, Nazli Goharian, and Ophir Frieder. 2020. Expansion via Prediction of Importance with Contextualization (EPIC). SIGIR 2020 (2020).
Rodrigo Nogueira, Wei Yang, Jimmy Lin, and Kyunghyun Cho. 2019. Document Expansion by Query Prediction. arXiv preprint arXiv:1904.08375 (2019).
Stephen E. Robertson, Steve Walker, Susan Jones, et al. 1995. Okapi at TREC-3. Nist Special Publication Sp 109 (1995).
Gerard Salton and Chris Buckley. 1988. Term-Weighting Approaches in Automatic Text Retrieval. Inf. Process. Manag. 24 (1988), 513–523.
G. Salton, A. Wong, and C. S. Yang. 1975. A vector space model for automatic indexing. Commun. ACM 18, 11 (1975), 613–620.
Tefko Saracevic. 2016. The Notion of Relevance in Information Science. Morgan & Claypool Publishers (2016).
Claudio Carpineto and Giovanni Romano. 2012. A Survey of Automatic Query Expansion in Information Retrieval. ACM Comput. Surv. (2012).
Liang Wang, Nan Yang, and Furu Wei. 2023. Query2doc: Query Expansion with Large Language Models. EMNLP 2023 (2023).
Jimmy Lin, Rodrigo Nogueira, and Andrew Yates. 2020. Pretrained Transformers for Text Ranking: BERT and Beyond. arXiv:2010.06467 (2020).
Vladimir Karpukhin, Barlas Oguz, Sewon Min, et al. 2020. Dense Passage Retrieval for Open-Domain Question Answering. EMNLP 2020 (2020).
Weiwei Sun, Lingyong Yan, Xinyu Ma, et al. 2023. Is ChatGPT Good at Search? Investigating Large Language Models as Re-Ranking Agents. EMNLP 2023 (2023).
Yi Tay, Vinh Tran, Mostafa Dehghani, et al. 2022. Transformer Memory as a Differentiable Search Index. NeurIPS 2022 (2022).
Ronak Pradeep, Kai Hui, Jai Gupta, et al. 2023. How Does Generative Retrieval Scale to Millions of Passages? EMNLP 2023 (2023).
Yubao Tang, Ruqing Zhang, Jiafeng Guo, and Maarten de Rijke. 2023. Recent Advances in Generative Information Retrieval. SIGIR-AP '23 (2023).
Xiaoxi Li, Jiajie Jin, Yujia Zhou, et al. 2024. From Matching to Generation: A Survey on Generative Information Retrieval. arXiv:2404.14851 (2024).
Patrick Lewis, Ethan Perez, Aleksandra Piktus, et al. 2020. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. NeurIPS 2020 (2020).
Ori Ram, Yoav Levine, Itay Dalmedigos, et al. 2023. In-Context Retrieval-Augmented Language Models. TACL 11 (2023).
Ori Yoran, Tomer Wolfson, Ori Ram, and Jonathan Berant. 2024. Making Retrieval-Augmented Language Models Robust to Irrelevant Context. ICLR 2024 (2024).
David Rau, Hervé Déjean, Nadezhda Chirkova, et al. 2024. BERGEN: A Benchmarking Library for Retrieval-Augmented Generation. Findings of EMNLP 2024 (2024).
Seraphina Goldfarb-Tarrant, Pedro Rodriguez, Jane Dwivedi-Yu, and Patrick Lewis. 2024. MultiContrievers: Analysis of Dense Retrieval Representations. BlackboxNLP 2024 (2024).
John Morris, Volodymyr Kuleshov, Vitaly Shmatikov, and Alexander Rush. 2023. Text Embeddings Reveal (Almost) As Much As Text. EMNLP 2023 (2023).
Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai-Wei Chang. 2017. Men Also Like Shopping: Reducing Gender Bias Amplification using Corpus-level Constraints. EMNLP 2017 (2017).
Amin Abolghasemi, Leif Azzopardi, Arian Askari, Maarten de Rijke, and Suzan Verberne. 2024. Measuring Bias in a Ranked List Using Term-Based Representations. ECIR '24 (2024).

2. Datasets and Evaluation

Payal Bajaj, Daniel Campos, Nick Craswell, et al. 2018. MS MARCO: A Human Generated MAchine Reading COmprehension Dataset. arXiv:1611.09268 (2018).
Nick Craswell, Bhaskar Mitra, Emine Yilmaz, et al. 2020-2023. Overviews of the TREC Deep Learning Tracks (2019, 2020, 2021, 2022, 2023). TREC / arXiv.
Nandan Thakur, Nils Reimers, Andreas Rücklé, Abhishek Srivastava, and Iryna Gurevych. 2021. BEIR: A heterogeneous benchmark for zero-shot evaluation. arXiv:2104.08663 (2021).
Ellen M. Voorhees and Dawn M. Tice. 1999. The TREC-8 Question Answering Track Report. TREC-8 (1999).
Kalervo Järvelin and Jaana Kekäläinen. 2002. Cumulated gain-based evaluation of IR techniques. ACM Trans. Inf. Syst. (2002).
Navid Rekabsaz, Oleg Lesota, Markus Schedl, Jon Brassey, and Carsten Eickhoff. 2021. TripClick: The Log Files of a Large Health Web Search Engine. SIGIR '21 (2021).

3. LSR Framework

Thong Nguyen, Sean MacAvaney, and Andrew Yates. 2023. A Unified Framework for Learned Sparse Retrieval. 45th European Conference on Information Retrieval (ECIR '23).
Zhichao Geng, Yiwen Wang, Dongyu Ru, and Yang Yang. 2024. Towards competitive search relevance for inference-free learned sparse retrievers. arXiv:2411.04403 (2024).
Antonio Mallia, Omar Khattab, Torsten Suel, and Nicola Tonellotto. 2021. Learning Passage Impacts for Inverted Indexes. SIGIR '21 (2021).
Zhuyun Dai and Jamie Callan. 2019. Context-Aware Sentence/Passage Term Importance Estimation for First Stage Retrieval. arXiv:1910.10687 (2019).
Zhuyun Dai and Jamie Callan. 2020. Context-Aware Term Weighting for First Stage Passage Retrieval. SIGIR '20 (2020).
Zhuyun Dai and Jamie Callan. 2020. Context-Aware Document Term Weighting for Ad-Hoc Search. WWW '20 (2020).
Yang Bai, Xiaoguang Li, Gang Wang, et al. 2020. SparTerm: Learning Term-based Sparse Representation for Fast Text Retrieval. arXiv:2010.00768 (2020).
Tiancheng Zhao, Xiaopeng Lu, and Kyusong Lee. 2021. SPARTA: Efficient Open-Domain Question Answering via Sparse Transformer Matching Retrieval. NAACL 2021 (2021).
Shengyao Zhuang and Guido Zuccon. 2021. Fast Passage Re-ranking with Contextualized Exact Term Matching and Efficient Passage Expansion (TILDEv2). arXiv:2108.08513 (2021).
Zhichao Geng, Xinyuan Lu, Dagney Braun, Charlie Yang, and Fanit Kolchina. 2023. Improving Document Retrieval with Sparse Semantic Encoders. OpenSearch Blog (2023).
Franco Maria Nardini, Thong Nguyen, Cosimo Rulli, Rossano Venturini, and Andrew Yates. 2025. Effective Inference-Free Retrieval for Learned Sparse Representations. SIGIR '25 (2025).

4. Text LSR

Hamed Zamani, Mostafa Dehghani, W. Bruce Croft, et al. 2018. From Neural Re-Ranking to Neural Ranking: Learning a Sparse Representation for Inverted Indexing (SNRM). CIKM '18 (2018).
Jimmy Lin and Xueguang Ma. 2021. A Few Brief Notes on DeepImpact, COIL, and a Conceptual Framework. arXiv:2106.14807 (2021).
Thibault Formal, Carlos Lassance, Benjamin Piwowarski, and Stéphane Clinchant. 2021. SPLADE v2: Sparse Lexical and Expansion Model for Information Retrieval. arXiv:2109.10086 (2021).
Shengyao Zhuang and Guido Zuccon. 2021. TILDE: Term Independent Likelihood moDEl for Passage Re-ranking. SIGIR '21 (2021).
V Sanh. 2019. DistilBERT, A Distilled Version of BERT. arXiv:1910.01108 (2019).
Jacob Devlin. 2018. BERT: Pre-training of deep bidirectional transformers. arXiv:1810.04805 (2018).
Thibault Formal, Carlos Lassance, Benjamin Piwowarski, and Stéphane Clinchant. 2022. From Distillation to Hard Negative Sampling: Making Sparse Neural IR Models More Effective. SIGIR '22 (2022).
Thibault Formal, Carlos Lassance, Benjamin Piwowarski, and Stéphane Clinchant. 2024. Towards Effective and Efficient Sparse Neural Information Retrieval. ACM Trans. Inf. Syst. 42, 5 (2024).
Carlos Lassance and Stéphane Clinchant. 2022. An Efficiency Study for SPLADE Models. SIGIR '22 (2022).

5. Initialization of LSR Methods

Thibault Formal, Maxime Louis, Hervé Dejean, and Stéphane Clinchant. 2026. Learning Retrieval Models with Sparse Autoencoders. ICLR 2026 (2026).
Zhichao Xu, Shengyao Zhuang, Crystina Zhang, et al. 2026. LACONIC: Dense-Level Effectiveness for Scalable Sparse Retrieval. arXiv:2601.01684 (2026).
Jingfen Qiao, Thong Nguyen, Evangelos Kanoulas, and Andrew Yates. 2025. Leveraging decoder architectures for learned sparse retrieval. International Workshop on Knowledge-Enhanced IR (2025).
Zhichao Xu, Aosong Feng, Yijun Tian, et al. 2025. CSPLADE: Learned Sparse Retrieval with Causal Language Models. arXiv:2504.10816 (2025).
Hansi Zeng, Julian Killingback, and Hamed Zamani. 2025. Scaling sparse and dense retrieval in decoder-only LLMs. SIGIR '25 (2025).
Thong Nguyen, Shubham Chatterjee, Sean MacAvaney, et al. 2024. DyVo: Dynamic Vocabularies for Learned Sparse Retrieval with Entities. EMNLP '24 (2024).
Hervé Déjean, Stéphane Clinchant, Carlos Lassance, Simon Lupart, and Thibault Formal. 2023. Benchmarking Middle-Trained Language Models for Neural Search. SIGIR '23 (2023).
Carlos Lassance, Hervé Dejean, and Stéphane Clinchant. 2023. An Experimental Study on Pretraining Transformers from Scratch for IR. ECIR '23 (2023).
Tao Shen, Xiubo Geng, Chongyang Tao, et al. 2023. LexMAE: Lexicon-Bottlenecked Pretraining for Large-Scale Retrieval. ICLR 2023 (2023).
Yibin Lei, Tao Shen, Yu Cao, and Andrew Yates. 2025. Enhancing Lexicon-Based Text Embeddings with Large Language Models. ACL 2025 (2025).
Jeffrey M. Dudek, Weize Kong, Cheng Li, Mingyang Zhang, and Michael Bendersky. 2023. Learning Sparse Lexical Representations Over Specified Vocabularies for Retrieval. CIKM '23 (2023).
Benjamin Clavié, Sean Lee, Aamir Shakir, and Makoto P. Kato. 2026. Latent Terms: Dense Retrievers Contain Trivially Extractable BM25-ready Zipfian Vocabularies. arXiv:2605.29384 (2026).

Part 2 – Emerging Topics

1. Multilingual LSR

Suraj Nair, Eugene Yang, Dawn Lawrie, James Mayfield, and Douglas W. Oard. 2023. BLADE: Combining Vocabulary Pruning and Intermediate Pretraining for Scaleable Neural CLIR. SIGIR '23.
Thong Nguyen, Yibin Lei, Jia-Huei Ju, Eugene Yang, and Andrew Yates. 2025. Milco: Learned Sparse Retrieval Across Languages via a Multilingual Connector. arXiv:2510.00671 (2025).
Suraj Nair, Eugene Yang, Dawn J. Lawrie, James Mayfield, and Douglas W. Oard. 2022. Learning a Sparse Representation Model for Neural CLIR. DESIRES 2022 (2022).
Jianlyu Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu. 2024. M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillation. Findings of ACL 2024 (2024).

2. Multimodal LSR

Thong Nguyen, Mariya Hendriksen, Andrew Yates, and Maarten de Rijke. 2024. Multimodal Learned Sparse Retrieval with Probabilistic Expansion Control. ECIR '24 (2024).
Chen Chen, Bowen Zhang, Liangliang Cao, et al. 2023. STAIR: Learning Sparse Text and Image Representation in Grounded Tokens. EMNLP 2023.
Ziyang Luo, Pu Zhao, Can Xu, et al. 2023. LexLIP: Lexicon-Bottlenecked Language-Image Pre-Training for Image-Text Sparse Retrieval. ICCV 2023.
Jiawei Zhou, Xiaoguang Li, Lifeng Shang, et al. 2024. Retrieval-based Disentangled Representation Learning with Natural Language Supervision. ICLR 2024.
Donghoon Han, Eunhwan Park, and Seunghyeon Seo. 2026. Visual Words Meet BM25: Sparse Auto-Encoder Visual Word Scoring for Image Retrieval. arXiv:2603.05781 (2026).

3. Indexing & Efficient LSR

Sebastian Bruch, Franco Maria Nardini, Amir Ingber, and Edo Liberty. 2023. An approximate algorithm for maximum inner product search over streaming sparse vectors (Seismic). ACM TOIS 42, 2 (2023).
Sebastian Bruch, Franco Maria Nardini, Cosimo Rulli, and Rossano Venturini. 2024. Efficient Inverted Indexes for Approximate Retrieval (Seismic). SIGIR '24 (2024).
Sebastian Bruch, Franco Maria Nardini, Cosimo Rulli, Rossano Venturini, and Leonardo Venuta. 2025. Investigating the Scalability of Approximate Sparse Retrieval Algorithms (Seismic). ECIR '25 (2025).
Parker Carlson, Wentai Xie, Shanxiu He, and Tao Yang. 2025. Dynamic superblock pruning for fast learned sparse retrieval. SIGIR '25 (2025).
Antonio Mallia, Torsten Suel, and Nicola Tonellotto. 2024. Faster learned sparse retrieval with block-max pruning. SIGIR '24 (2024).
Joel Mackenzie, Andrew Trotman, and Jimmy Lin. 2023. Efficient document-at-a-time and score-at-a-time query evaluation. ACM TOIS 41, 4 (2023).
Yifan Qiao, Parker Carlson, Shanxiu He, Yingrui Yang, and Tao Yang. 2024. Threshold-driven Pruning with Segmented Maximum Term Weights. EMNLP '24.
Andrei Z. Broder, David Carmel, Michael Herscovici, Aya Soffer, and Jason Zien. 2003. Efficient Query Evaluation Using a Two-Level Retrieval Process. CIKM '03 (2003), 426–434.
Shuai Ding and Torsten Suel. 2011. Faster Top-k Document Retrieval Using Block-Max Indexes. SIGIR '11 (2011), 993–1002.
Constantinos Dimopoulos, Sergey Nepomnyachiy, and Torsten Suel. 2013. Optimizing Top-k Document Retrieval Strategies for Block-Max Indexes. WSDM '13 (2013), 113–122.
Antonio Mallia, Giuseppe Ottaviano, Elia Porciani, Nicola Tonellotto, and Rossano Venturini. 2017. Faster BlockMax WAND with Variable-sized Blocks. SIGIR '17 (2017).
Antonio Mallia and Elia Porciani. 2019. Faster BlockMax WAND with Longer Skipping. ECIR '19 (2019).
Jimmy Lin and Andrew Trotman. 2015. Anytime Ranking for Impact-Ordered Indexes. ICTIR '15 (2015), 301–304.
Joel Mackenzie, Matthias Petri, and Luke Gallagher. 2022. IOQP: A Simple Impact-Ordered Query Processor Written in Rust. DESIRES 2022 (2022).
Carlos Lassance, Simon Lupart, Hervé Déjean, Stéphane Clinchant, and Nicola Tonellotto. 2023. A Static Pruning Study on Sparse Neural Retrievers. SIGIR '23 (2023).
Carlos Lassance, Hervé Déjean, Stéphane Clinchant, and Nicola Tonellotto. 2024. Two-Step SPLADE: Simple, Efficient and Effective Approximation of SPLADE. ECIR '24 (2024).
Sebastian Bruch, Franco Maria Nardini, Amir Ingber, and Edo Liberty. 2024. Bridging Dense and Sparse Maximum Inner Product Search. ACM TOIS (2024).
Sebastian Bruch, Franco Maria Nardini, Cosimo Rulli, and Rossano Venturini. 2024. Pairing Clustered Inverted Indexes with κ-NN Graphs for Fast Approximate Retrieval over Learned Sparse Representations. CIKM '24 (2024), 3642–3646.
Leonardo Delfino, Domenico Erriquez, Silvio Martinico, Franco Maria Nardini, Cosimo Rulli, and Rossano Venturini. 2025. kANNolo: Sweet and Smooth Approximate k-Nearest Neighbors Search. ECIR '25 (2025), 400–406.
Parker Carlson, Wentai Xie, Rohil Shah, and Tao Yang. 2026. Efficiency Optimizations for Superblock-based Sparse Retrieval. arXiv:2602.02883 (2026).
Ruoxuan Li, Xiaoyao Zhong, Jiabao Jin, et al. 2025. SINDI: An Efficient Index for Sparse Vector Approximate Maximum Inner Product Search. arXiv:2509.08395 (2025).
Yury Malkov, Alexander Ponomarenko, Andrey Logvinov, and Vladimir Krylov. 2014. Approximate Nearest Neighbor Algorithm Based on Navigable Small World Graphs. Inf. Syst. 45 (2014).
Ikuya Yamada, Akari Asai, and Hannaneh Hajishirzi. 2021. Efficient Passage Retrieval with Hashing for Open-domain Question Answering. ACL 2021 (2021).

4. Hybrid Dense-Sparse Retrieval

Sheng-Chieh Lin and Jimmy Lin. 2023. A Dense Representation Framework for Lexical and Semantic Matching. ACM Trans. Inf. Syst. (2023).
Shengyao Zhuang, Xueguang Ma, Bevan Koopman, Jimmy Lin, and Guido Zuccon. 2024. PromptReps: Prompting LLMs to Generate Dense and Sparse Representations. arXiv:2404.18424 (2024).
Yingrui Yang, Parker Carlson, Shanxiu He, Yifan Qiao, and Tao Yang. 2024. Cluster-based Partial Dense Retrieval Fused with Sparse Text Retrieval. SIGIR '24 (2024).
Xilun Chen, Kushal Lakhotia, Barlas Oguz, et al. 2022. Salient Phrase Aware Dense Retrieval: Can a Dense Retriever Imitate a Sparse One? EMNLP 2022.
Tao Shen, Xiubo Geng, Chongyang Tao, et al. 2023. UnifieR: A Unified Retriever for Large-Scale Retrieval. KDD '23.
Jiawei Zhou, Li Dong, Furu Wei, and Lei Chen. 2024. Semi-Parametric Retrieval via Binary Token Index. arXiv:2405.01924 (2024).
Thibault Formal, Stéphane Clinchant, Hervé Déjean, and Carlos Lassance. 2024. SPLATE: Sparse Late Interaction Retrieval. SIGIR '24 (2024).
Kai Zhang, Chongyang Tao, Tao Shen, et al. 2023. LED: Lexicon-Enlightened Dense Retriever for Large-Scale Retrieval. WWW '23 (2023).