Lucene 源码分析之倒排索引（三）

上文找到了 collect(…) 方法，其形参就是匹配的文档 Id，根据代码上下文，其中 doc 是由 iterator.nextDoc() 获得的，那 DefaultBulkScorer.iterator 是何时赋值的？代码如下。

public abstract class Weight implements SegmentCacheable {

    protected static class DefaultBulkScorer extends BulkScorer {

        // ...

        public DefaultBulkScorer(Scorer scorer) {

            // ...

            this.scorer = scorer;

            this.iterator = scorer.iterator();

            this.twoPhase = scorer.twoPhaseIterator();

        }

        // ...

    }

}

构造函数中 scorer.iterator() 即为匹配的文档 Id，那么 scorer 又是从何而来呢？回顾 Weight.bulkScorer(…) 方法，代码如下。根据上文可知 scorer(context) 的实现类是 TermWeight。

public abstract class Weight implements SegmentCacheable {

    public BulkScorer bulkScorer(LeafReaderContext context) throws IOException {

        Scorer scorer = scorer(context);

        // ...

        return new DefaultBulkScorer(scorer);

    }

}

public class TermQuery extends Query {

    final class TermWeight extends Weight {

        @Override

        public Scorer scorer(LeafReaderContext context) throws IOException {

            final TermsEnum termsEnum = getTermsEnum(context);

            if (termsEnum == null) {

                return null;

            }

            PostingsEnum docs = termsEnum.postings(null, needsScores ? PostingsEnum.FREQS : PostingsEnum.NONE);

            assert docs != null;

            return new TermScorer(this, docs, similarity.simScorer(stats, context));

        }

    }

}

final class TermScorer extends Scorer {

    private final PostingsEnum postingsEnum;

    TermScorer(Weight weight, PostingsEnum td, Similarity.SimScorer docScorer) {

        super(weight);

        this.docScorer = docScorer;

        this.postingsEnum = td;

    }

    @Override

    public DocIdSetIterator iterator() {

        return postingsEnum;

    }

}

至此可以确定 scorer.iterator() 来源于 termsEnum.postings(...) 。倒排索引是不是若隐若现了呢。

下面聚焦于 termsEnum 的实际类型和其 postings(...) 方法。

根据上文可知，termsEnum 来源于 TermQuery.getTermsEnum(...)，代码如下。

public class TermQuery extends Query {

    private TermsEnum getTermsEnum(LeafReaderContext context) throws IOException {

        final TermState state = termStates.get(context.ord);

        final TermsEnum termsEnum = context.reader().terms(term.field()).iterator();

        termsEnum.seekExact(term.bytes(), state);

        return termsEnum;

    }

}

public final class LeafReaderContext extends IndexReaderContext {

    private final LeafReader reader;

}

LeafReader 本身是没有 terms(...) 方法的，也就是说 context.reader() 并不是 LeaferReader，而是其子类。根据上文已知 LeafReaderContext 是 IndexSearcher.leafContexts 其中的一个元素，那么找到 IndexSearcher.leafContexts 的赋值代码也就能知道 context.reader() 的实际类型。

public class IndexSearcher {

    public IndexSearcher(IndexReader r) {

        this(r, null);

    }

    public IndexSearcher(IndexReader r, ExecutorService executor) {

        this(r.getContext(), executor);

    }

    public IndexSearcher(IndexReaderContext context, ExecutorService executor) {

        // ...

        leafContexts = context.leaves();

        // ...

    }

}

根据这部分代码可知，IndexSearcher.leafContexts 来源于 IndexReader.getContext().leaves()。一般来说，这个 IndexReader 是 DirectoryReader.open(...) 返回的一个 StandardDirectoryReader 类。代码如下。

public abstract class DirectoryReader extends BaseCompositeReader<LeafReader> {

    public static DirectoryReader open(final Directory directory) throws IOException {

        return StandardDirectoryReader.open(directory, null);

    }

}

那么 IndexSearcher.leafContexts 实际来源于 StandardDirectoryReader.getContext().leaves()

public final class StandardDirectoryReader extends DirectoryReader {

    // ...

}

public abstract class DirectoryReader extends BaseCompositeReader<LeafReader> {

    // ...

}

public abstract class BaseCompositeReader<R extends IndexReader> extends CompositeReader {

    // ...

}

public abstract class CompositeReader extends IndexReader {

    @Override

    public final CompositeReaderContext getContext() {

        // ...

        readerContext = CompositeReaderContext.create(this);

        return readerContext;

    }

    @Override

    public List<LeafReaderContext> leaves() throws UnsupportedOperationException {

        return leaves;

    }

    private final List<LeafReaderContext> leaves;

}

CompositeReaderContext.create(…) 是怎么创建的呢？

public final class CompositeReaderContext extends IndexReaderContext {

    static CompositeReaderContext create(CompositeReader reader) {

        return new Builder(reader).build();

    }

    private static final class Builder {

        public Builder(CompositeReader reader) {

            this.reader = reader;

        }

        public CompositeReaderContext build() {

            return (CompositeReaderContext) build(null, reader, 0, 0);

        }

        private IndexReaderContext build(CompositeReaderContext parent, IndexReader reader, int ord, int docBase) {

            if (reader instanceof LeafReader) {

                final LeafReader ar = (LeafReader) reader;

                final LeafReaderContext atomic = new LeafReaderContext(parent, ar, ord, docBase, leaves.size(), leafDocBase);

                leaves.add(atomic);

                leafDocBase += reader.maxDoc();

                return atomic;

            } else {

                final CompositeReader cr = (CompositeReader) reader;

                final List<? extends IndexReader> sequentialSubReaders = cr.getSequentialSubReaders();

                final List<IndexReaderContext> children = Arrays.asList(new IndexReaderContext[sequentialSubReaders.size()]);

                final CompositeReaderContext newParent;

                if (parent == null) {

                    newParent = new CompositeReaderContext(cr, children, leaves);

                } else {

                    newParent = new CompositeReaderContext(parent, cr, ord, docBase, children);

                }

                int newDocBase = 0;

                for (int i = 0, c = sequentialSubReaders.size(); i < c; i++) {

                    final IndexReader r = sequentialSubReaders.get(i);

                    children.set(i, build(newParent, r, i, newDocBase));

                    newDocBase += r.maxDoc();

                }

                assert newDocBase == cr.maxDoc();

                return newParent;

            }

        }

    }

    private CompositeReaderContext(CompositeReaderContext parent, CompositeReader reader, int ordInParent, int docbaseInParent, List<IndexReaderContext> children, List<LeafReaderContext> leaves) {

        this.leaves = leaves == null ? null : Collections.unmodifiableList(leaves);

        // ...

    }

}

build(...) 时，传入的 reader 类型是 StandardDirectoryReader，将执行 getSequentialSubReaders() 得到其所有子 reader，并以 reader 作为成员变量创建 LeafReaderContext，然后将 LeafReaderContext 加入到 leaves 中。

所以 IndexSearcher.leafContexts 的每个元素 LeafReaderContext 的 reader 即为 StandardDirectoryReader 的 getSequentialSubReaders()

public final class StandardDirectoryReader extends DirectoryReader {

    static DirectoryReader open(final Directory directory, final IndexCommit commit) throws IOException {

        return new SegmentInfos.FindSegmentsFile<DirectoryReader>(directory) {

            @Override

            protected DirectoryReader doBody(String segmentFileName) throws IOException {

                SegmentInfos sis = SegmentInfos.readCommit(directory, segmentFileName);

                final SegmentReader[] readers = new SegmentReader[sis.size()];

                boolean success = false;

                try {

                    for (int i = sis.size()-1; i >= 0; i--) {

                        readers[i] = new SegmentReader(sis.info(i), sis.getIndexCreatedVersionMajor(), IOContext.READ);

                    }

                    DirectoryReader reader = new StandardDirectoryReader(directory, readers, null, sis, false, false);

                    success = true;

                    return reader;

                }

                // ...

            }

        }.run(commit);

    }

    StandardDirectoryReader(Directory directory, LeafReader[] readers, IndexWriter writer, SegmentInfos sis, boolean applyAllDeletes, boolean writeAllDeletes) throws IOException {

        super(directory, readers);

        this.writer = writer;

        this.segmentInfos = sis;

        this.applyAllDeletes = applyAllDeletes;

        this.writeAllDeletes = writeAllDeletes;

    }

}

public abstract class DirectoryReader extends BaseCompositeReader<LeafReader> {

    protected DirectoryReader(Directory directory, LeafReader[] segmentReaders) throws IOException {

        super(segmentReaders);

        this.directory = directory;

    }

}

public abstract class BaseCompositeReader<R extends IndexReader> extends CompositeReader {

    protected BaseCompositeReader(R[] subReaders) throws IOException {

        this.subReaders = subReaders;

        // ...

    }

}

可以分析出，reader 的类型是 SegmentReader，而该类（其实是其父类）确实是有 terms(…) 方法的。代码如下。

public final class SegmentReader extends CodecReader {

    // ...

    final SegmentCoreReaders core;

    @Override

    public FieldsProducer getPostingsReader() {

        return core.fields;

    }

}

public abstract class CodecReader extends LeafReader implements Accountable {

    @Override

    public final Terms terms(String field) throws IOException {

        return getPostingsReader().terms(field);

    }

}

final class SegmentCoreReaders {

    final FieldsProducer fields;

    SegmentCoreReaders(Directory dir, SegmentCommitInfo si, IOContext context) throws IOException {

        // ...

        final Codec codec = si.info.getCodec();

        final PostingsFormat format = codec.postingsFormat();

        fields = format.fieldsProducer(segmentReadState);

        // ...

    }

}

在 lucene-7.3.0 中默认的 codec 是 Lucene70Codec，默认 postingsFomat 是 Lucene50PostingsFormat，这个分析过程请见 Lucene 源码分析之 segment（后续补上）。

所以 SegmentReader.terms(…) 实际调用的是 Lucene50PostingsFormat.fieldsProducer(…).terms(…)。

public final class Lucene50PostingsFormat extends PostingsFormat {

    @Override

    public FieldsProducer fieldsProducer(SegmentReadState state) throws IOException {

        PostingsReaderBase postingsReader = new Lucene50PostingsReader(state);

        FieldsProducer ret = new BlockTreeTermsReader(postingsReader, state);

        return ret;

    }

}

最终 SegmentReader.terms(…) 实际调用的是 BlockTreeTermsReader.terms(…)。

public final class BlockTreeTermsReader extends FieldsProducer {

    @Override

    public Terms terms(String field) throws IOException {

        return fields.get(field);

    }

    private final TreeMap<String,FieldReader> fields = new TreeMap<>();

    public BlockTreeTermsReader(PostingsReaderBase postingsReader, SegmentReadState state) throws IOException {

    	this.postingsReader = postingsReader;

        fields.put(fieldInfo.name, new FieldReader(...));

    }

}

则 BlockTreeTermsReader.terms(…) 实际返回的是 FieldReader。

再次回顾上文中的核心代码。

public class TermQuery extends Query {

    final class TermWeight extends Weight {

        @Override

        public Scorer scorer(LeafReaderContext context) throws IOException {

            final TermsEnum termsEnum = getTermsEnum(context);

            if (termsEnum == null) {

                return null;

            }

            PostingsEnum docs = termsEnum.postings(null, needsScores ? PostingsEnum.FREQS : PostingsEnum.NONE);

            assert docs != null;

            return new TermScorer(this, docs, similarity.simScorer(stats, context));

        }

    }

    private TermsEnum getTermsEnum(LeafReaderContext context) throws IOException {

        final TermState state = termStates.get(context.ord);

        final TermsEnum termsEnum = context.reader().terms(term.field()).iterator();

        termsEnum.seekExact(term.bytes(), state);

        return termsEnum;

    }

}

则 termsEnum 为 FieldReader.iterator()，是一个 SegmentTermsEnum。

public final class FieldReader extends Terms implements Accountable {

    @Override

    public TermsEnum iterator() throws IOException {

        return new SegmentTermsEnum(this);

    }

}

则 termsEnum.postings(…) 为 SegmentTermsEnum.postings(…)。

final class SegmentTermsEnum extends TermsEnum {

    @Override

    public PostingsEnum postings(PostingsEnum reuse, int flags) throws IOException {

        currentFrame.decodeMetaData();

        return fr.parent.postingsReader.postings(fr.fieldInfo, currentFrame.state, reuse, flags);

    }

    final FieldReader fr;

}

public final class FieldReader extends Terms implements Accountable {

    final BlockTreeTermsReader parent;

}

public final class BlockTreeTermsReader extends FieldsProducer {

    final PostingsReaderBase postingsReader;

}

fr 是在 SegmntTermsEnum 的构造函数里出现的。

final class SegmentTermsEnum extends TermsEnum {

    public SegmentTermsEnum(FieldReader fr) throws IOException {

        this.fr = fr;

    }

}

而这个 FieldReader 是在 BlockTreeTermsReader 的构造函数里构造的。

public final class BlockTreeTermsReader extends FieldsProducer {

    public BlockTreeTermsReader(PostingsReaderBase postingsReader, SegmentReadState state) throws IOException {

        // ...

        fields.put(fieldInfo.name, new FieldReader(this,...));

    }

}

public final class FieldReader extends Terms implements Accountable {

    FieldReader(BlockTreeTermsReader parent,...) throws IOException {

        this.parent = parent;

    }

}

则 fr.parent 是 BlockTreeTermsReader，则 fr.parent.postingsReader 是 Lucene50PostingsReader，这就是倒排索引的核心类。

巴特西

Lucene 源码分析之倒排索引（三）

最新文章

热门文章