使用 Flink Datastream 访问 LAS 元信息,比如:Paimon on LAS、Iceberg on LAS 等,需要在原有开源 Paimon、Iceberg Connector 依赖的基础上,添加 LAS 相关的依赖,并在打包时把 LAS 依赖打包到 uber jar 里。具体步骤如下:
LAS 包含四个主要的依赖,分别是:hive-exec、lf-client-3、lf-client-common-3、lf-common-3,需要先下载到本地,通过 maven install 安装后使用:
安装到本地 Maven 仓库的命令如下:
mvn install:install-file \ -Dfile=./lf-client-3-1.5.0-RELEASE.jar \ -DpomFile=lf-client-3-1.5.0-RELEASE.pom \ -DgroupId=org.apache.hive \ -DartifactId=lf-client-3 \ -Dversion=1.5.0-RELEASE \ -Dpackaging=jar mvn install:install-file \ -Dfile=./lf-client-common-3-1.5.0-RELEASE.jar \ -DpomFile=lf-client-common-3-1.5.0-RELEASE.pom \ -DgroupId=org.apache.hive \ -DartifactId=lf-client-common-3 \ -Dversion=1.5.0-RELEASE \ -Dpackaging=jar mvn install:install-file \ -Dfile=./lf-common-3-1.5.0-RELEASE.jar \ -DpomFile=lf-common-3-1.5.0-RELEASE.pom \ -DgroupId=org.apache.hive \ -DartifactId=lf-common-3 \ -Dversion=1.5.0-RELEASE \ -Dpackaging=jar mvn install:install-file \ -Dfile=./common-unify-1.0.8.jar \ -DpomFile=common-unify-1.0.8.pom \ -DgroupId=com.bytedance.las \ -DartifactId=common \ -Dversion=unify-1.0.8 \ -Dpackaging=jar mvn install:install-file \ -Dfile=./hive-1.5.0-RELEASE.pom \ -DpomFile=hive-1.5.0-RELEASE.pom \ -DgroupId=org.apache.hive \ -DartifactId=hive \ -Dversion=1.5.0-RELEASE \ -Dpackaging=pom
包含 LAS 依赖的基础 pom.xml 如下,需要在原有 Flink 作业的依赖上,增加如下 pom 配置:
注意
<?xml version="1.0" encoding="UTF-8"?> <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"> <modelVersion>4.0.0</modelVersion> <dependencies> <!-- LF client 依赖,需要在 hive-metastore 之前,防止依赖冲突 --> <dependency> <groupId>org.apache.hive</groupId> <artifactId>lf-client-3</artifactId> <version>1.5.0-RELEASE</version> </dependency> <!-- 使用与你的环境匹配的 Hive 版本 --> <dependency> <groupId>org.apache.hive</groupId> <artifactId>hive-metastore</artifactId> <version>3.1.2</version> <scope>provided</scope> </dependency> ... 其他依赖,例如 hive、iceberg 等 ... </dependencies> <build> <plugins> <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-shade-plugin</artifactId> <executions> <execution> <id>shade-flink</id> <phase>package</phase> <goals> <goal>shade</goal> </goals> <configuration> <artifactSet> <includes> <!-- 把 LAS 依赖打包到最终的作业 Jar 里: --> <include>org.apache.hive:lf-client-3</include> <include>org.apache.hive:lf-client-common-3</include> <include>org.apache.hive:lf-common-3</include> <include>org.apache.thrift:libfb303</include> </includes> </artifactSet> </configuration> </execution> </executions> </plugin> </plugins> </build> </project>
打包命令如下:
mvn clean package -DskipTests
得到作业 Jar 包后,上传到 Flink 使用即可。