如何将在线图片传入Gemini模型？遇Pillow转Vertex AI格式报错

问题场景 #

使用Gemini模型生成在线图片描述时，执行Part.from_image(pil_image)代码行触发错误：AttributeError: 'JpegImageFile' object has no attribute 'data'。

错误原因 #

Part.from_image()方法要求传入的是Vertex AI官方的vertexai.generative_models.Image对象，而非Pillow库返回的JpegImageFile实例，直接传入Pillow图片对象会因属性不匹配报错。

解决方案 #

提供两种可行的修正方案：

方案1：将Pillow图片转为字节流后创建Vertex AI Image对象 #

先把Pillow图片转换为字节流，再用Image.from_bytes()生成符合要求的Image对象，最后传入Part.from_image()：

import requests
from io import BytesIO
import vertexai
from vertexai.generative_models import Part, Image, GenerativeModel
from PIL import Image as PILImage

image_path = "https://i.natgeofe.com/n/548467d8-c5f1-4551-9f58-6817a8d2c45e/NationalGeographic_2572187_square.jpg"
response = requests.get(image_path)
pil_image = PILImage.open(BytesIO(response.content))

# 将Pillow图片转为字节流
img_byte_arr = BytesIO()
pil_image.save(img_byte_arr, format='JPEG')
img_byte_arr = img_byte_arr.getvalue()

# 创建Vertex AI Image对象并生成Part
vertex_image = Image.from_bytes(img_byte_arr)
image_part = Part.from_image(vertex_image)

vertexai.init(project="project-name", location="region")
model = GenerativeModel(model_name="gemini-1.0-pro-vision-001")

query = "Please describe the image."
response = model.generate_content([image_part, query])
print(response.text)

方案2：直接使用图片URL创建Part（更简洁） #

若目标图片为公开可访问的远程链接，无需下载到本地处理，直接通过Part.from_uri()生成图片Part：

import vertexai
from vertexai.generative_models import Part, GenerativeModel

image_path = "https://i.natgeofe.com/n/548467d8-c5f1-4551-9f58-6817a8d2c45e/NationalGeographic_2572187_square.jpg"

# 直接通过URL创建图片Part
image_part = Part.from_uri(image_path, mime_type="image/jpeg")

vertexai.init(project="project-name", location="region")
model = GenerativeModel(model_name="gemini-1.0-pro-vision-001")

query = "Please describe the image."
response = model.generate_content([image_part, query])
print(response.text)

内容的提问来源于stack exchange，提问作者Koala S