基于OpenAI Embedding的PHP问答系统相似性匹配异常排查

参考教程《Using OpenAI to create a Q&A in Laravel/PHP with embedding》实现PHP版OpenAI Embedding问答系统，流程如下：

• 对源文档分块生成embedding并序列化存储到本地文件；
• 对用户问题生成embedding后，调用getAnswer函数通过余弦相似度匹配最相关的文档块。

目前getAnswer函数返回结果异常，仅得到字符'a'，导致传入OpenAI的上下文错误，最终回答完全不符合预期。以下是完整实现代码，需排查问题原因（embedding生成环节或逻辑错误）：

第一部分代码（embedding生成与存储） #

<?php
    include('vendor/autoload.php');

    $client = OpenAI::client('API');

    /* Get file */
    $filename = 'sample-chatgpt-file.txt';
    $file_contents = file_get_contents($filename);

    // Get data split in 2000 characters because of OpenAI
    $split = openai_str_split($file_contents);

    // Get embedding from OpenAI
    $return = getInputs($split);

    // Save embedding of entry file so we can work with it later
    $file = 'embed.txt';
    file_put_contents($file, serialize($return));
    $contents = file_get_contents($file);
    $retrieved_data = unserialize($contents);

    // Display what we got
    print_r($retrieved_data);

    // Function for embedding creation
    function getInputs($prompts)
    {
        $client = OpenAI::client('API');

        return $client->embeddings()->create([
            'model' => 'text-embedding-ada-002',
            'input' => $prompts,
        ]);
    }

    // Function for splitting entry file
    function openai_str_split($text) {
        $max_length = 1996;

        $sentences = preg_split('/(?<=[.?!])\s+(?=[a-z])/i', $text);

        $chunks = array();
        $chunk = '';

        foreach ($sentences as $sentence) {
            $sentence_length = strlen($sentence);

            if (strlen($chunk) + $sentence_length > $max_length) {
                $chunks[] = $chunk;
                $chunk = '';
            }

            $chunk .= $sentence . ' ';
        }

        if (!empty($chunk)) {
            $chunks[] = $chunk;
        }

        return $chunks;
    }
?>

第二部分代码（问答匹配与生成） #

<?php
    include('vendor/autoload.php');

    $client = OpenAI::client('API');

    // Entry file
    $VhodniFile = 'sample-chatgpt-file.txt';
    $prompts = file_get_contents($VhodniFile);

    // Entry file embedding
    $file = 'embed.txt';
    $contents = file_get_contents($file);
    $inputs = unserialize($contents);

    // Question
    $userQuestion = 'How does closing of source items work?';

    // Get embedding for question
    $question = $client->embeddings()->create([
                'model' => 'text-embedding-ada-002',
                'input' => $userQuestion,
            ]);

    // Get answer
    $answer = getAnswer($prompts, $inputs, $question);

    //print_r($answer);

    // Display match
    print_r("The ada match: " . $prompts[$answer['index']]);

    // Prompt to send to OpenAI
    $davinci = "Rewrite the question and give the answer with an example in PHP from the context
            Context: {$prompts[$answer['index']]}
            Question: {$userQuestion}
            Answer:";
    //print_r($davinci);

    // Send prompt to GPT-3 DaVinci
    $result = $client->completions()->create([
                  'model' => 'text-davinci-003',
                  'prompt' => $davinci,
                  'temperature' => 0.5,
                  'max_tokens' => 1000,
              ]);

    print_r($result);

    // Result output
    print("Naredi berljivo: {$result['choices'][0]['text']}");

    function getAnswer($prompts, $inputs, $question)
    {
        // Loops through all the inputs and compare on a cosine similarity to the question and output the correct answer
        $results = [];
        for ($i = 0; $i < count($inputs->embeddings); $i++) {
            $similarity = cosineSimilarity($inputs->embeddings[$i]->embedding, $question->embeddings[0]->embedding);
            // Store the similarity and index in an array and sort by the similarity
            $results[] = [
                'similarity' => $similarity,
                'index' => $i,
                'input' => $prompts[$i],
            ];
        }
        usort($results, function ($a, $b) {
            return $a['similarity'] <=> $b['similarity'];
        });

        return end($results);
    }

    function cosineSimilarity($u, $v)
    {
        $dotProduct = 0;
        $uLength = 0;
        $vLength = 0;
        for ($i = 0; $i < count($u); $i++) {
            $dotProduct += $u[$i] * $v[$i];
            $uLength += $u[$i] * $u[$i];
            $vLength += $v[$i] * $v[$i];
        }
        $uLength = sqrt($uLength);
        $vLength = sqrt($vLength);
        return $dotProduct / ($uLength * $vLength);
    }
?>

问题出在第二部分代码的文档块引用逻辑错误，和embedding生成环节无关，具体错误点及修复方案如下：

1. 核心错误：文档块变量类型不匹配 #

第二部分代码中，$prompts被赋值为整个文件的字符串内容（file_get_contents($VhodniFile)），但第一部分生成embedding的是分块后的数组。当在getAnswer函数中使用$prompts[$i]时，实际是取字符串的第i个字符，而非对应索引的文档块，这就是返回字符'a'的直接原因。

2. 次要优化：排序逻辑与边界防护 #

• 原排序为升序，用end($results)取最高相似度元素逻辑可行，但改为降序后用reset()取更直观；
• 余弦相似度函数未处理向量长度不一致或模长为0的情况，易引发报错。

修复后的第二部分代码 #

<?php
    include('vendor/autoload.php');

    $client = OpenAI::client('API');

    // 读取文档并分块，确保与embedding生成时的分块完全一致
    $VhodniFile = 'sample-chatgpt-file.txt';
    $file_contents = file_get_contents($VhodniFile);
    $prompts = openai_str_split($file_contents);

    // 读取存储的embedding数据
    $file = 'embed.txt';
    $contents = file_get_contents($file);
    $inputs = unserialize($contents);

    // 用户问题
    $userQuestion = 'How does closing of source items work?';

    // 生成问题的embedding
    $question = $client->embeddings()->create([
                'model' => 'text-embedding-ada-002',
                'input' => $userQuestion,
            ]);

    // 获取最相关的文档块
    $answer = getAnswer($prompts, $inputs, $question);

    // 输出匹配的上下文
    print_r("匹配到的上下文: " . $answer['input']);

    // 构建发送给OpenAI的Prompt
    $davinci = "Rewrite the question and give the answer with an example in PHP from the context
            Context: {$answer['input']}
            Question: {$userQuestion}
            Answer:";

    // 调用GPT-3 DaVinci生成回答
    $result = $client->completions()->create([
                  'model' => 'text-davinci-003',
                  'prompt' => $davinci,
                  'temperature' => 0.5,
                  'max_tokens' => 1000,
              ]);

    print_r($result);

    // 输出格式化结果
    print("易读版回答: {$result['choices'][0]['text']}");

    function getAnswer($prompts, $inputs, $question)
    {
        $results = [];
        $embeddingCount = count($inputs->embeddings);
        for ($i = 0; $i < $embeddingCount; $i++) {
            $similarity = cosineSimilarity($inputs->embeddings[$i]->embedding, $question->embeddings[0]->embedding);
            $results[] = [
                'similarity' => $similarity,
                'index' => $i,
                'input' => $prompts[$i],
            ];
        }
        // 降序排序，相似度最高的元素排在数组开头
        usort($results, function ($a, $b) {
            return $b['similarity'] <=> $a['similarity'];
        });

        return reset($results);
    }

    function cosineSimilarity($u, $v)
    {
        $dotProduct = 0;
        $uLength = 0;
        $vLength = 0;
        // 取两个向量的最小长度，避免越界
        $maxIndex = min(count($u), count($v));
        
        for ($i = 0; $i < $maxIndex; $i++) {
            $dotProduct += $u[$i] * $v[$i];
            $uLength += $u[$i] * $u[$i];
            $vLength += $v[$i] * $v[$i];
        }

        $uLength = sqrt($uLength);
        $vLength = sqrt($vLength);
        
        // 避免除以0的情况
        if ($uLength == 0 || $vLength == 0) {
            return 0;
        }

        return $dotProduct / ($uLength * $vLength);
    }

    // 复用第一部分的文档分块函数
    function openai_str_split($text) {
        $max_length = 1996;

        $sentences = preg_split('/(?<=[.?!])\s+(?=[a-z])/i', $text);

        $chunks = array();
        $chunk = '';

        foreach ($sentences as $sentence) {
            $sentence_length = strlen($sentence);

            if (strlen($chunk) + $sentence_length > $max_length) {
                $chunks[] = $chunk;
                $chunk = '';
            }

            $chunk .= $sentence . ' ';
        }

        if (!empty($chunk)) {
            $chunks[] = $chunk;
        }

        return $chunks;
    }
?>

内容的提问来源于stack exchange，提问作者Luka S.