在Ollama上使用LLAVA分析文档失效问题求助

问题背景 #

我正在测试LLaVA用于文档理解任务，在部分科研论文和网站中看到了不错的效果。已在Windows系统的Ollama上部署该模型，并使用以下C#代码调用，但实际结果表现很差，大多为幻觉内容。有时LLM会提示需要更清晰的文档视图才能回答问题，尝试过缩小图片但仍未解决，请问是否可以分块处理图片？

调用代码 #

using System.Text;
using System.Text.Json;

public class Program
{
    private static readonly HttpClient client = new HttpClient();
    private static string? imageBase64;

    static async Task Main(string[] args)
    {
        Console.WriteLine("Welcome to the Document Analysis Application!");

        while (true)
        {
            Console.Write("Enter the path to the image file (or 'exit' to quit): ");
            string imagePath;
            do
            {
                imagePath = Console.ReadLine() ?? "";
            } while (String.IsNullOrEmpty(imagePath));

            if (imagePath.ToLower() == "exit")
                break;

            if (!File.Exists(imagePath))
            {
                Console.WriteLine("File not found. Please try again.");
                continue;
            }

            imageBase64 = Convert.ToBase64String(File.ReadAllBytes(imagePath));
            Console.WriteLine("Image loaded successfully.");

            while (true)
            {
                Console.Write("Enter your question about the document (or 'new' for a new image, 'exit' to quit): ");
                string question;
                do
                {
                    question = Console.ReadLine() ?? "";
                } while (String.IsNullOrEmpty(question));

                if (question.ToLower() == "new")
                    break;
                if (question.ToLower() == "exit")
                    return;

                Console.WriteLine("Response:");
                _ = await AnalyzeDocument(question);
                Console.WriteLine("\nEnd of response.");
            }
        }
    }

    static async Task<string> AnalyzeDocument(string question)
    {
        var requestBody = new
        {
            model = "llava:13b-v1.6",
            prompt = $"Analyze this invoice image carefully. Pay close attention to all numerical values, especially totals and subtotals. If the question is about a total or sum, make sure to double-check your calculation. After your analysis, provide a clear, concise answer to this specific question: {question}",
            images = new[] { imageBase64 },
            stream = true
        };

        var content = new StringContent(JsonSerializer.Serialize(requestBody), Encoding.UTF8, "application/json");

        try
        {
            HttpResponseMessage response = await client.PostAsync("http://localhost:11434/api/generate", content);
            response.EnsureSuccessStatusCode();

            using (var reader = new StreamReader(await response.Content.ReadAsStreamAsync()))
            {
                StringBuilder fullResponse = new StringBuilder();
                string? line;
                while ((line = await reader.ReadLineAsync()) != null)
                {
                    if (string.IsNullOrWhiteSpace(line)) continue;

                    try
                    {
                        using (JsonDocument doc = JsonDocument.Parse(line))
                        {
                            JsonElement root = doc.RootElement;
                            if (root.TryGetProperty("response", out JsonElement responseElement))
                            {
                                string responsePart = responseElement.GetString() ?? "";
                                fullResponse.Append(responsePart);
                                Console.Write(responsePart); // Print each part as it's received
                            }
                            if (root.TryGetProperty("done", out JsonElement doneElement) && doneElement.GetBoolean())
                            {
                                break;
                            }
                        }
                    }
                    catch (JsonException)
                    {
                        Console.WriteLine($"Failed to parse JSON: {line}");
                    }
                }
                return fullResponse.ToString();
            }
        }
        catch (HttpRequestException e)
        {
            return $"Error: {e.Message}";
        }
    }
}

解决方案：图片分块处理的可行性与实现 #

核心结论 #

可以通过图片分块处理解决当前问题，尤其适合文字密集、尺寸较大的文档图片，能有效提升模型对细节的识别精度，减少幻觉。

具体实现步骤 #

• 图片切割：将原始文档图片按固定尺寸（如512x512像素）分割为多个带重叠区域的小块，避免文字被切割断裂。可使用.NET的ImageSharp或System.Drawing.Common库实现裁剪逻辑。
• 分块调用模型：针对每个图片块，结合原始问题生成子提示（如“提取此区域内的所有金额和对应项目”），分别调用Ollama的LLaVA接口获取各块的分析结果。
• 结果整合校验：将各块的结果汇总，结合原始问题整理最终答案。若涉及数值计算（如发票总额），可添加校验逻辑，对比各块提取的数值确保一致性。

额外优化建议 #

• 调整模型参数：调用Ollama时，设置temperature=0.1降低随机性减少幻觉，同时设置top_p=0.9提升结果的确定性。
• 优化提示词：细化提示词，明确要求模型仅基于图片内容回答，例如：“仅根据提供的图片内容作答，不得编造信息。若无法识别内容，请直接说明。”
• 图片预处理：切割前对图片进行增强处理，比如调整对比度、锐化文字，提升模型视觉识别的准确率。

内容的提问来源于stack exchange，提问作者Patrick S