Lucene分组收集器不支持Sorted Numeric字段，如何实现兼容？

当前IndexSearcher的collectNumericFilterOptions函数可基于搜索结果收集筛选选项，但仅支持NUMERIC字段，不支持SORTED_NUMERIC字段：

fun IndexSearcher.collectNumericFilterOptions(
    query: Query,
    sort: Sort = Sort(),
    field: String,
    topNGroups: Int = 128,
    mapper: Function<Int?, Int?> = Function { it },
): Set<Int?> {
    val firstGroupSelector = LongRangeGroupSelector(LongValuesSource.fromIntField(field), LongRangeFactory(1, 1, topNGroups.toLong()))
    val firstPassGroupingCollector = FirstPassGroupingCollector(firstGroupSelector, sort, topNGroups)
    search(query, firstPassGroupingCollector)
    val topGroups = firstPassGroupingCollector.getTopGroups(0) ?: return emptySet()
    val groupSelector = firstPassGroupingCollector.groupSelector
    val distinctValuesCollector = DistinctValuesCollector(groupSelector, topGroups, groupSelector)
    search(query, distinctValuesCollector)
    return distinctValuesCollector.groups.map { mapper.apply(it.groupValue.min.toInt()) }.toSet()
}

当字段按以下方式索引时可正常运行：

document.apply {
    add(IntPoint(ITERATION.name, iteration ?: -1))
    add(StoredField(ITERATION.name, iteration ?: -1))
    add(NumericDocValuesField(ITERATION.name, iteration?.toLong() ?: -1L)) 
}

但按以下方式索引时则无法正常工作：

document.apply {
    add(IntPoint(ITERATION.name, iteration ?: -1))
    add(StoredField(ITERATION.name, iteration ?: -1))
    add(SortedNumericDocValuesField(ITERATION.name, iteration?.toLong() ?: -1L))
}

此时会抛出如下异常：

java.lang.IllegalStateException: unexpected docvalues type SORTED_NUMERIC for field 'ITERATION' (expected=NUMERIC). Re-index with correct docvalues type.

    at org.apache.lucene.index.DocValues.checkField(DocValues.java:218)
    at org.apache.lucene.index.DocValues.getNumeric(DocValues.java:237)
    at org.apache.lucene.search.LongValuesSource$FieldValuesSource.getValues(LongValuesSource.java:254)
    at org.apache.lucene.search.grouping.LongRangeGroupSelector.setScorer(LongRangeGroupSelector.java:63)
    at org.apache.lucene.search.grouping.FirstPassGroupingCollector.setScorer(FirstPassGroupingCollector.java:157)

解决方案 #

要让该函数支持SORTED_NUMERIC字段，核心是替换原代码中仅支持NUMERIC类型的LongValuesSource.fromIntField，改用能处理SortedNumericDocValues的逻辑。以下是修改后的函数实现：

fun IndexSearcher.collectNumericFilterOptions(
    query: Query,
    sort: Sort = Sort(),
    field: String,
    topNGroups: Int = 128,
    mapper: Function<Int?, Int?> = Function { it },
): Set<Int?> {
    // 自定义LongValuesSource，支持处理SORTED_NUMERIC类型字段
    val valuesSource = object : LongValuesSource() {
        override fun getValues(context: LeafReaderContext, scorers: Bits): LongValues {
            val docValues = context.reader.getSortedNumericDocValues(field)
            return object : LongValues() {
                override fun longValue(): Long {
                    // 取当前文档的第一个值，可根据业务需求调整为取最小/最大值等
                    return if (docValues.docValueCount() > 0) docValues.nextValue() else -1L
                }

                override fun advanceExact(doc: Int): Boolean {
                    return docValues.advanceExact(doc)
                }
            }
        }

        override fun needsScores(): Boolean = false

        override fun copy(): LongValuesSource = this

        override fun toString(): String = "SortedNumericLongValuesSource(field=$field)"

        override fun equals(other: Any?): Boolean {
            if (this === other) return true
            if (javaClass != other?.javaClass) return false
            other as LongValuesSource
            return toString() == other.toString()
        }

        override fun hashCode(): Int = toString().hashCode()
    }

    val firstGroupSelector = LongRangeGroupSelector(valuesSource, LongRangeFactory(1, 1, topNGroups.toLong()))
    val firstPassGroupingCollector = FirstPassGroupingCollector(firstGroupSelector, sort, topNGroups)
    search(query, firstPassGroupingCollector)
    val topGroups = firstPassGroupingCollector.getTopGroups(0) ?: return emptySet()
    val groupSelector = firstPassGroupingCollector.groupSelector
    val distinctValuesCollector = DistinctValuesCollector(groupSelector, topGroups, groupSelector)
    search(query, distinctValuesCollector)
    return distinctValuesCollector.groups.map { mapper.apply(it.groupValue.min.toInt()) }.toSet()
}

说明 #

• 自定义的LongValuesSource通过getSortedNumericDocValues获取字段值，适配SORTED_NUMERIC类型；
• 示例中取每个文档的第一个值，若业务需要处理多值场景（比如取最小值、最大值），可修改longValue()中的逻辑，例如遍历所有值取最小：

  override fun longValue(): Long {
      var minVal = Long.MAX_VALUE
      val count = docValues.docValueCount()
      for (i in 0 until count) {
          val value = docValues.nextValue()
          if (value < minVal) minVal = value
      }
      return if (count > 0) minVal else -1L
  }
  

• 该修改无需重新索引数据，直接兼容原有的SORTED_NUMERIC字段索引方式。

内容的提问来源于stack exchange，提问作者Andras Hatvani