如何获取ML.NET中Latent Dirichlet Allocation生成的主题？

ML.NET官方LDA示例仅展示文档与主题的相关性，若要获取主题的具体内容，需通过模型内部API提取主题-词权重映射，结合词汇表还原关键词。以下是完整可运行示例：

完整代码实现 #

using Microsoft.ML;
using Microsoft.ML.Data;
using System;
using System.Linq;

public class DocumentData
{
    [LoadColumn(0)]
    public string Text { get; set; }
}

public class TopicPrediction
{
    [ColumnName("TopicDistribution")]
    public float[] TopicProbabilities { get; set; }
}

class Program
{
    static void Main(string[] args)
    {
        // 初始化MLContext
        var mlContext = new MLContext(seed: 1);

        // 样本文档数据
        var documents = new[]
        {
            new DocumentData { Text = "The quick brown fox jumps over the lazy dog. Foxes are clever animals." },
            new DocumentData { Text = "Cats are independent pets. They like to sleep and play with yarn." },
            new DocumentData { Text = "Dogs are loyal companions. They love to fetch and play with their owners." },
            new DocumentData { Text = "Birds sing beautiful songs. Many birds migrate south for the winter." },
            new DocumentData { Text = "Fish live in water. They swim using their fins and gills to breathe." }
        };

        // 加载数据
        var dataView = mlContext.Data.LoadFromEnumerable(documents);

        // 构建文本处理与LDA训练管道
        var pipeline = mlContext.Transforms.Text.TokenizeIntoWords("Tokens", "Text")
            .Append(mlContext.Transforms.Text.RemoveDefaultStopWords("Tokens"))
            .Append(mlContext.Transforms.Conversion.MapValueToKey("Tokens"))
            .Append(mlContext.Transforms.Text.ProduceWordBags("WordBag", "Tokens"))
            .Append(mlContext.Transforms.Text.LatentDirichletAllocation("TopicDistribution", "WordBag", numberOfTopics: 3));

        // 训练模型
        var model = pipeline.Fit(dataView);

        // 获取LDA转换器与词汇映射表
        var ldaTransformer = model.LastTransformer as Microsoft.ML.Transforms.Text.LatentDirichletAllocationTransformer;
        var vocabMap = mlContext.Data.GetMapValueToKeyModelInfo(model.First<Microsoft.ML.Transforms.Conversion.MapValueToKeyTransformer>()).KeyToValueMap;

        // 提取并展示每个主题的Top关键词
        var topicWordWeights = ldaTransformer.GetTopicWordWeights();
        int topN = 5; // 每个主题取Top5关键词

        for (int topicId = 0; topicId < topicWordWeights.GetLength(0); topicId++)
        {
            Console.WriteLine($"主题 {topicId + 1}:");
            // 按权重降序排序，取TopN词
            var topWords = topicWordWeights[topicId]
                .Select((weight, idx) => new { Word = vocabMap[idx], Weight = weight })
                .OrderByDescending(x => x.Weight)
                .Take(topN);

            foreach (var word in topWords)
            {
                Console.WriteLine($"  {word.Word}: {word.Weight:F4}");
            }
            Console.WriteLine();
        }

        // 可选：测试单文档的主题分布
        var predEngine = mlContext.Model.CreatePredictionEngine<DocumentData, TopicPrediction>(model);
        var testDoc = new DocumentData { Text = "My dog loves to play fetch in the park." };
        var prediction = predEngine.Predict(testDoc);

        Console.WriteLine("测试文档的主题分布：");
        for (int i = 0; i < prediction.TopicProbabilities.Length; i++)
        {
            Console.WriteLine($"  主题 {i + 1}: {prediction.TopicProbabilities[i]:F4}");
        }
    }
}

关键逻辑说明 #

1. 获取LDA转换器：训练后的模型中，最后一个组件是LatentDirichletAllocationTransformer，通过类型转换获取实例。
2. 还原词汇表：预处理时用MapValueToKey将词汇转为索引，需从MapValueToKeyTransformer中获取KeyToValueMap，实现索引到原始词汇的映射。
3. 提取主题-词权重：调用ldaTransformer.GetTopicWordWeights()得到二维数组，其中topicWordWeights[topicId][wordIdx]表示对应主题下该词汇的权重值。
4. 生成主题内容：对每个主题的词汇按权重降序排序，取Top N个词汇，即可得到该主题的核心关键词。

优化提示 #

• 文本预处理步骤（去停用词、自定义分词规则）直接影响主题质量，需根据数据集调整。
• numberOfTopics参数需结合文档数量、内容复杂度调整，避免主题过多或过少。
• 权重值越高，词汇在对应主题中的代表性越强，可根据需求调整Top N的数量。

内容的提问来源于stack exchange，提问作者user3786916