Azure Cognitive Search：避免EdgeNGram分词器拆分连字符词汇

问题根源在于当前使用的StandardTokenizer会将连字符（-）视为分隔符，把"7-ETREW"拆分为"7"和"etrew"两个独立token，后续的EdgeNGram过滤器仅对"etrew"处理，导致无法保留带连字符的前缀。

具体修改步骤： #

• 替换分词器：将StandardTokenizer替换为KeywordTokenizer，该分词器会将整个输入字符串作为单个token，不会拆分连字符、数字等字符。
• 保留必要的TokenFilter：继续使用Lowercase过滤器确保大小写不敏感，再应用EdgeNGram过滤器生成所需前缀。

修改后的完整代码 #

public class SearchRequest
{
    [SimpleField(IsKey = true, IsFilterable = true)]
    public string id { get; set; }

    [SearchableField(SearchAnalyzerName = LexicalAnalyzerName.Values.StandardLucene, IndexAnalyzerName = "prefixEdgeAnalyzer")]
    public string EntityID { get; set; }

    public MetaData? MetaData { get; set; }
}

public class MetaData
{
    [SearchableField(AnalyzerName = LexicalAnalyzerName.Values.EnMicrosoft)]
    public string? CustomerName { get; set; }

    [SearchableField(SearchAnalyzerName = LexicalAnalyzerName.Values.StandardLucene, IndexAnalyzerName = "prefixEdgeAnalyzer")]
    public List<string>? OpportunityIDs { get; set; }
}

public async Task<Response<SearchIndex>> CreateIndex(string indexName)
{
    try
    {
        var nedgeTokenfilter = new EdgeNGramTokenFilter("edgeNgramTokenFilterV2");
        nedgeTokenfilter.MinGram = 3;
        nedgeTokenfilter.MaxGram = 20;
        nedgeTokenfilter.Side = EdgeNGramTokenFilterSide.Front;

        // 替换分词器为KeywordTokenizer，避免拆分连字符
        var prefixEdgeAnalyzer = new CustomAnalyzer("prefixEdgeAnalyzer", LexicalTokenizerName.Keyword);
        prefixEdgeAnalyzer.TokenFilters.Add(TokenFilterName.Lowercase);
        prefixEdgeAnalyzer.TokenFilters.Add("edgeNgramTokenFilterV2");

        var suggester = new SearchSuggester("spellCheckSuggester", $"MetaData/{nameof(SearchRequest.MetaData.CustomerName)}");

        FieldBuilder fieldBuilder = new FieldBuilder();
        var searchFields = fieldBuilder.Build(typeof(SearchRequest));

        var definition = new SearchIndex(indexName, searchFields);

        definition.TokenFilters.Add(nedgeTokenfilter);
        definition.Analyzers.Add(prefixEdgeAnalyzer);
        definition.Suggesters.Add(suggester);

        var response = await _adminClient.CreateOrUpdateIndexAsync(definition).ConfigureAwait(false);

        return response;
    }
    catch (Exception ex)
    {
        _logger.LogError(ex, ex.Message);
        throw;
    }
}

验证效果 #

使用Analyze API测试"7-ETREW"时，会生成以下符合需求的token：
7-e, 7-et, 7-etr, 7-etre, 7-etrew

内容的提问来源于stack exchange，提问作者PRACHI AGARWAL