描述 #

多模态向量生成处理器

核心功能 #

• 多模态向量化支持：支持图像/视频与文本的联合向量生成，实现跨模态检索能力，参考文档。
• 输入格式自适应：
  • 原生支持图像/视频的base64编码、二进制数据、URL等输入格式
  • 自动处理媒体格式转换（JPEG/PNG/MP4/AVI等）
• 模型名称：doubao-embedding-vision

输入输出规范 #

• 输入格式：
  • 图片/视频数据/文本数据：string类型，支持base64编码/url地址
• 输出格式：
  • 默认模式：float数组类型的向量表示

POST https://operator.las.cn-beijing.volces.com/api/v1/embeddings/multimodal

接口说明 #

调用该算子进行图文向量化。

请求参数 #

返回数据 #

示例 #

请求示例 #

curl --location "https://operator.las.cn-beijing.volces.com/api/v1/embeddings/multimodal" \
--header "Content-Type: application/json" \
--header "Authorization: Bearer $LAS_API_KEY" \
--data '{
    "model": "doubao-embedding-vision-250615",
    "encoding_format": "float",
    "input": [
        {
            "type": "image_url",
            "image_url": {
                "url": "https://las-ai-cn-beijing-baseline.tos-cn-beijing.volces.com/operator_cards_serving/public/baseline/doubao_embedding_vision/cat_ip_adapter.jpeg"
            }
        },
        {
            "type": "text",
            "text": "图片里有什么"
        }
    ]
}'

返回示例 #

{
    "id": "021764041607868c64dcafb5669f61b8aa56083c290255748bdbb",
    "created": "1764041608",
    "model": "doubao-embedding-vision-250615",
    "object": "list",
    "data": {
        "object": "embedding",
        "embedding": [
            0.0002918243408203125,
            -0.0189208984375,
            -0.003814697265625,
            0.058349609375,
            -0.002838134765625,
            0.032958984375,
            "..."
        ]
    },
    "usage": {
        "prompt_tokens": 1335,
        "total_tokens": 1335,
        "prompt_tokens_details": {
            "text_tokens": 23,
            "image_tokens": 1312
        }
    }
}

算子参数 #

输入 #

输出 #

返回模型处理后的向量化数组。类型为list[float]

参数 #

如参数没有默认值，则为必填参数

调用示例 #

下面的代码展示了如何使用 daft访问火山方舟图像向量化模型进行向量化计算。

from __future__ import annotations

import os

import daft
from daft import col
from daft.las.functions.ark_llm.doubao_embedding_vision import DoubaoEmbeddingVision
from daft.las.functions.udf import las_udf

if os.getenv("DAFT_RUNNER", "native") == "ray":
    import logging

    import ray

    def configure_logging():
        logging.basicConfig(
            level=logging.INFO,
            format="%(asctime)s - %(name)s - %(levelname)s - %(message)s",
            datefmt="%Y-%m-%d %H:%M:%S.%s".format(),
        )
        logging.getLogger("tracing.span").setLevel(logging.WARNING)
        logging.getLogger("daft_io.stats").setLevel(logging.WARNING)
        logging.getLogger("DaftStatisticsManager").setLevel(logging.WARNING)
        logging.getLogger("DaftFlotillaScheduler").setLevel(logging.WARNING)
        logging.getLogger("DaftFlotillaDispatcher").setLevel(logging.WARNING)

    ray.init(dashboard_host="0.0.0.0", runtime_env={"worker_process_setup_hook": configure_logging})
    daft.context.set_runner_ray()

daft.set_execution_config(min_cpu_per_task=0)

if __name__ == "__main__":
    # 需配置环境变量 LAS_API_KEY ： LAS_API_KEY 通过在 LAS 服务页面上创建获取
    tos_dir_url = os.getenv("TOS_DIR_URL", "las-cn-beijing-public-online.tos-cn-beijing.volces.com")
    samples = {
        "image_path": [
            f"https://{tos_dir_url}/public/shared_image_dataset/cat_ip_adapter.jpeg"
        ],
        "text": ["猫"],
    }

    df = daft.from_pydict(samples)
    # 计算图片和文本的向量化数据
    df = df.with_column(
        "embeeding_for_image_text",
        las_udf(
            DoubaoEmbeddingVision,
            construct_args={
                "image_format": "jpeg",
            },
        )(col("image_path"), col("text")),
    )

    # 计算图片向量化数据
    df = df.with_column(
        "embeeding_for_image",
        las_udf(
            DoubaoEmbeddingVision,
            construct_args={
                "image_format": "jpeg",
            },
        )(col("image_path")),
    )

    # 计算文本向量化数据
    df = df.with_column(
        "embeeding_for_text",
        las_udf(
            DoubaoEmbeddingVision,
            construct_args={
                "multimodal_type": "text",
            },
        )(col("text")),
    )
    df.show()

    #  输出(每次大模型推理结果可能不同)
    # ╭────────────────────────────────┬────────┬────────────────────────────────┬────────────────────────────────┬────────────────────────────────╮
    # │ image_path                     ┆ text   ┆ embeeding_for_image_text       ┆ embeeding_for_image            ┆ embeeding_for_text             │
    # │ ---                            ┆ ---    ┆ ---                            ┆ ---                            ┆ ---                            │
    # │ String                         ┆ String ┆ List[Float32]                  ┆ List[Float32]                  ┆ List[Float32]                  │
    # ╞════════════════════════════════╪════════╪════════════════════════════════╪════════════════════════════════╪════════════════════════════════╡
    # │ https://las-public-data-qa.to… ┆ 猫     ┆ [0.010437012, -0.009399414, -… ┆ [0.0076293945, -0.014404297, … ┆ [0.019165039, 0.007598877, -0… │
    # ╰────────────────────────────────┴────────┴────────────────────────────────┴────────────────────────────────┴────────────────────────────────╯