Paper Reading【BitFunnel: Revisiting Signatures for Search 】
SIGIR 2017 Best Paper
a. Since the mid-90s there has been a widely-held belief that signature files are inferior to inverted files for text indexing.
Justin Zobel, Alistair Moffat, and Kotagiri Ramamohanarao. 1998. Inverted files versus signature files for text indexing
b. Signature-based approach(BitFunnel) faces some challenges：
- Excessive Memory Reads
- Wide Range of Term Frequencies
- Wide Range of Document Lengths
- False Positives
- Hard to Configure
a. Inverted Indexes
A mapping from each term in the lexicon to the the set of documents containing the term
b. Bit-String Signatures
The signature is essentially the sequence of bits that make up a Bloom filter representing the set of terms in the document.
c. Bit-Sliced Signatures
This approach transposes signature vectors from rows to columns to allow multiple documents to be searched simultaneously
d. Bit-Sliced Blocked Signatures
Create shorter rows by assigning multiple documents to each column in the signature matrix.
A. Higher Row Rank
Each term simultaneously hashes to multiple bit-sliced signatures with different
A row of rank r ≥ 0 has a blocking factor of 2r
B. Frequency Conscious Bloom Filter
C. Sharding Across Cluster
Sharding by Document Length
D. Handing False Positive
Reject those documents that score low while never rejecting documents that score high.
Performance Model And Optimization
In order to address speed and space problems associated with bit-string and bit-sliced signatures. The article introduce higher rank rows to reduce query execution time, employ frequency-conscious signatures to reduce the memory footprint, and shard the corpus to reduce the variability in document size. Compared to inverted lists，BitFunnel improved server query capacity by a factor of 10.