PHP proc_open调用Python脚本触发UnicodeEncodeError问题排查

通过PHP的proc_open函数调用Python脚本时，抛出UnicodeEncodeError异常，但直接在命令行运行该Python脚本完全正常，怀疑与PHP捕获输出的处理方式有关。

相关代码 #

PHP代码 #

<?php

function my_shell_exec($cmd, &$stdout=null, &$stderr=null) {
    $proc = proc_open($cmd,[
        1 => ['pipe','w'],
        2 => ['pipe','w'],
    ],$pipes);
    $stdout = stream_get_contents($pipes[1]);
    fclose($pipes[1]);
    $stderr = stream_get_contents($pipes[2]);
    fclose($pipes[2]);
    return proc_close($proc);
}

$output = my_shell_exec('.\.venv\Scripts\activate && .\.venv\Scripts\python.exe infer.py', $stdout, $stderr);
var_dump($output);
var_dump($stdout);
var_dump($stderr);
exit();

PHP执行输出 #

int(1)
string(183) "Init model
Model already exists in keras-model/model-transventricular-v3.keras, skipping model init.
Infer image
(1, 256, 256, 1)
" keras-model\model-transventricular-v3.keras "
"
string(2416) "2024-06-09 22:57:14.361024: I tensorflow/core/util/port.cc:113] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
...
Traceback (most recent call last):
  File "D:\path\to\infer.py", line 220, in <module>
    infer_image("input.dat")
  File "D:\path\to\infer.py", line 201, in infer_image
    prediction = model.predict(images)
  File "C:\path\to\python\lib\encodings\cp1252.py", line 19, in encode
    return codecs.charmap_encode(input,self.errors,encoding_table)[0]
UnicodeEncodeError: 'charmap' codec can't encode characters in position 19-38: character maps to <undefined>

命令行直接执行输出（正常） #

(.venv) D:\path\to\project>\.venv\Scripts\activate && .\.venv\Scripts\python.exe infer.py
2024-06-09 22:56:04.593392: I tensorflow/core/util/port.cc:113] oneDNN custom operations are on.
...
Init model
Model already exists in keras-model/model-transventricular-v3.keras, skipping model init.
Infer image
(1, 256, 256, 1)
I0000 00:00:1717948579.475089   18516 service.cc:153]   StreamExecutor device (0): Host, Default Version
2024-06-09 22:56:19.530820: I tensorflow/compiler/mlir/tensorflow/utils/dump_mlir_util.cc:268] disabling MLIR crash reproducer, set env var `MLIR_CRASH_REPRODUCER_DIRECTORY` to enable.
I0000 00:00:1717948580.392302   18516 device_compiler.h:188] Compiled cluster using XLA!  This line is logged at most once for the lifetime of the process.
1/1 ━━━━━━━━━━━━━━━━━━━━ 2s 2s/step

Python脚本（infer.py） #

import os
from keras.models import load_model  # type: ignore
# from keras.optimizers import Adam # type: ignore
from dotenv import load_dotenv
import cv2
import numpy as np
import gdown

if __name__ == '__main__':
    if not os.path.exists('input.dat'):
        print("input.dat not exists")
        exit(0)

    print("Init model")
    init_model()

    print("Infer image")
    infer_image("input.dat")

def infer_image(image_filepath):
    # Preprocess image
    images = preproces(image_filepath)
    print(images.shape)

    path = os.path.join('keras-model', 'model-transventricular-v3.keras')
    print('"', path, '"')

    # load model
    model = load_model(path)

    # Predict image
    prediction = model.predict(images)
    predictions = np.argmax(prediction, axis=1)

    # further processing

核心原因是PHP调用Python时，子进程默认编码为cp1252，无法处理TensorFlow输出的Unicode字符（如进度条的━符号），触发编码异常。以下是三种可行修复方案：

方案1：Python脚本强制设置UTF-8输出编码 #

在Python脚本开头添加代码，强制stdout和stderr使用UTF-8编码：

import sys
sys.stdout.reconfigure(encoding='utf-8')
sys.stderr.reconfigure(encoding='utf-8')

方案2：PHP调用时设置Python环境变量 #

修改PHP的proc_open调用，传入PYTHONIOENCODING=utf-8环境变量，让Python进程默认用UTF-8处理IO：

function my_shell_exec($cmd, &$stdout=null, &$stderr=null) {
    // 合并原有环境变量，添加编码设置
    $env = array_merge($_ENV, ['PYTHONIOENCODING' => 'utf-8']);
    $proc = proc_open($cmd,[
        1 => ['pipe','w'],
        2 => ['pipe','w'],
    ],$pipes, null, $env);
    $stdout = stream_get_contents($pipes[1]);
    fclose($pipes[1]);
    $stderr = stream_get_contents($pipes[2]);
    fclose($pipes[2]);
    return proc_close($proc);
}

方案3：禁用TensorFlow的Unicode输出 #

如果不需要进度条等日志输出，在调用model.predict时关闭verbose模式：

prediction = model.predict(images, verbose=0)

verbose=0会关闭进度条及相关日志，避免产生Unicode字符。

内容的提问来源于stack exchange，提问作者Muhammad Ikhwan Perwira