基于UUID和文件类型配置Solr从S3导入Base64文件数据

原有数据库存储的Solr索引配置 #

<?xml version="1.0" encoding="UTF-8" ?>
<dataConfig>
    <dataSource name="fieldStream" type="FieldStreamDataSource"/>
    <dataSource name="db"
                type="JdbcDataSource"
                driver="oracle.jdbc.OracleDriver"
                url=""
                user=""
                password=""/>
    <document>
        <entity name="attachments"
                query="select tdata.ID as DATA_ID, null, tdata.NAME, att.DATA AS TEXT,
                      
                from T_DATA tdata 
                left join ATTACHED_DATA att on tdata.id = att.TDATA_ID where tdata.ACTIVE = 1
                      
                pk="DATA_ID"
                dataSource="db">
            <field column="DATA_ID" name="dataId"/>
            <field column="NAME" name="dataName"/>
   
            <entity name="attachment"
                    dataSource="fieldStream"
                    processor="TikaEntityProcessor"
                    tikaConfig="tikaconfig.xml"
                    dataField="attachments.TEXT"
                    url="TEXT"
                    format="text"
                    onError="continue">
                <field column="text" name="docBody"/>
            </entity>
        </entity>
    </document>
</dataConfig>

新场景说明 #

• 文件已迁移至S3存储，数据表新增UUID和FILE_TYPE字段
• 现有服务支持通过GET /s3/data/file/{UUID}/{FILE_TYPE}接口返回Base64格式的文件数据，响应示例：

{"id": "d32eba0a-ad02-4da0-bd0e-a5f2bca8b1fc","extension": "txt","file": "MTIz"}

已尝试的配置方案 #

已制作TikaTransformer.jar并放置于opt/solr/server/lib目录，尝试了以下三种配置：

1. URLDataSource配置 #

<dataSource name="s3Data"
            type="URLDataSource"
            baseUrl="http://localhost:8080"
            encoding="UTF-8"
            connectionTimeout="5000"
            readTimeout="10000"/>

2. XPath处理器+自定义转换器配置 #

<entity name="attachment"
        processor="XPathEntityProcessor"
        url="/s3/data/file/${uuid}/${fileType}"
        forEach="/"
        transformer="script:transformer">
    
    <field column="file" name="base64File" xpath="/file"/>
</entity>

<script>
    function transformer(row) {
        if (row.base64File) {
            var decoded = java.util.Base64.getDecoder().decode(row.base64File);
            row.docBody = new java.lang.String(decoded, "UTF-8");
        }
        return row;
    }
</script>

3. TikaEntityProcessor配置 #

<field column="UUID" name="uuid"/>
<field column="FILE_TYPE" name="file_type"/>
<entity name="attachment"
        dataSource="s3Data"
        processor="TikaEntityProcessor"
        tikaConfig="tikaconfig.xml"
        url="/s3/data/file/${attachments.UUID}/${attachments.FILE_TYPE}"
        format="text"
        onError="continue">
    <field column="text" name="docBody"/>
</entity>

求助需求 #

当前所有尝试均无法正常访问服务获取文件内容，需指导排查方向、修正配置内容，必要时可调整服务返回格式。

内容的提问来源于stack exchange，提问作者Ivan Klemenko