# 1.可视化建模 Open API 概述可视化建模（也称 Prep）提供丰富多样的数据清洗、筛选、聚合、机器学习等算子，支持用户创建任务，进行数据的抽取、转换能力，输出至数据集以供后续的报表制作、可视化查询、数据大屏使用。本平台将Prep强大的数据处理能力通过 Open API 的方式提供出来，为客户提供更灵活的业务处理方式。您通过使用 Open API，实现和可视化建模同样的效果。可视化建模 Open API 涵盖了可视化建模的所有基础能力。 # 2.基本概念 ## 2.1 鉴权相关 1） Client Client是用户服务和Prep服务通信的凭证，包含Client Id、Client Secret两部分。其中Client Id是该通信凭证在Prep系统中的唯一Id，Client Secret是该通信凭证的秘钥。在请求Open API前，用户服务需要使用Client信息来获取需要请求的用户的Token。 2）Token Token代表某一用户的身份信息，可使用Client信息请求Prep系统来获取。有了Token后，就可以在请求中带上Token来请求Prep系统了。 ## 2.2 可视化建模相关概念 1）可视化建模是指通过简单的可视化Web界面，完成专业的数据清洗、筛选、聚合、机器学习等复杂的数据ETL操作。 2）任务、子任务任务是一个完整的数据处理逻辑的集合，包含数据节点的属性、数据任务的调度、数据异常报警等。 3）数据集数据集是可视化分析的基础，是一组有特殊意义的数据集合。用户需将需要分析的数据导入到数据集，方可进行后续的报表制作工作。 # 3.准备阶段在调用系统接口前，你需要先申请连接系统的Client，Client包括Client Id和Client Secret。接着通过Client Id和Client Secret来获取指定用户的Token。然后就可以使用Token通过开放接口来请求系统服务啦。 ## 3.1 申请Client 如果你是智能数据洞察平台的系统管理员，你申请的得到的Client和秘钥是系统级别的。如果你不是智能数据洞察平台的系统管理员，你申请得到的Client和秘钥是用户级别的。（如果你所在的环境是位于火山引擎，并且需要使用到系统级Client来调用Prep Open API，请联系智能数据洞察平台的运维人员进行评估然后添加。） | | | | \ | |申请人 |获取Token的范围 | |---|---|---| | | | | \ |用户级Client |普通用户 |自己 | | | | | \ |系统级Client |系统管理员 |所有用户Token | 申请Client需要开发者获取自己在系统中的sessionid，然后通过使用session_id来调用auth/client接口进行Client的获取。下图举例说明了如何找到自己的sessionid。 ![图片](https://portal.volccdn.com/obj/volcfe/cloud-universal-doc/upload_80560acbb248d4e0770ced069b3a84b6.png =2752x) ## 3.2 获取用户Token 当你拥有一个Client之后，请将你的Client信息保存下来，这是获取用户Token的凭证。通过调用auth/token，附带上Client信息和需要获取Token的用户名，即可得到访问Open API的Token。在请求具体业务相关Open API的时候，将token放入Header中即可。格式为 Authorization: Bearer+空格+Token。例如： | | | \ |key |value | |---|---| | | | \ |Authorization |Bearer eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJ0b2tlbl90eXBlIjoib3BlbmFwaSIsInVzZXJuYW1lIjoiZ3VvaGFpc2h1YWkiLCJ1c2VyX3BheWxvYWQiOnt9LCJleHAiOjE2NTQ4NDgwOTMsImNsaWVudF9pZCI6ImNjYmQ4Mjg4ODZkZTRmOTRkODc2ZDQ4MTRhMGE1ODE3In0.WOfrqVWU40eZx-TgfhxMJ1TyuOiAhh98QLYtcNUE-f5q06qNN4bUv7ZD0tX6F8K9OqlIzBzXro-5pfRrcC4u4cBPnvuzl9etb8C3Nh0eAFlmqQrSZR5auNl9EaF8UaVuCSFlfal9aocUT2Re8JQN2UT_ww4y9GIZFhhBkjYZV--sBNuCh2B0UdjfwLEvevwOL-pW04pC81xLM5_aFX9FHa6TYK96o8dVLrHNjhQU2huh4D1il9UG9HAA_z6nhNyvuYcgs6ZZcapanuMc4YnTRPrmGdnoXMFFNImq6N3z943ACUrdinnM19TCj4fcKER3ZQ3XFT6UUnjHtn9sJ7rU4yvW9AcHBazVmjwscHwx0ooKe2pCMTiBa9QHi-R3nHIsuevx_AweiIZQ7vJCQ1PAve7mZVUi25QOlYJJCGiZmxd778sNX8M-BGFsGCLNugLMy1gaptTSma7YhQ-SwRt_Qxeeo3O6FgkyJMLnDHldwtYpcI6X2QUztAX4a4fzTwMMxTFIq0E_Az2xaz3O4-tX959o9KG93rD1EMWJKXn2IvrZjAA-7x-2d_HNVvl5NyDTvhTFRK8uYL1ZXPSSsBZXEhMuzJRpSawJohEUzgoYwNUGx75J15ZOmXRkeQm2J7OE3HEwHCrJ6zShZ7M8bFdjB1TZVoD2Mph4YroNq2jZbfE | # 4.常见返回码 1）正常返回 ```Plain Text { "code": "prep/ok", "extra_msg": null, "msg": "成功" } ``` 2） Token过期 ```Plain Text { "code": "prep/openAPI/tokenExpired", "message": "token已过期" } ``` 3） Token不正确 ```Plain Text { "code": "prep/openAPI/tokenError", "message": "token不正确" } ``` 4）触发限流 ```Plain Text { "code": "prep/openAPI/rateLimit", "message": "请求被限流" } ``` # 5.接口文档因为部署环境不同，Prep User Open API的实际域名会有不同，后续的的API地址中采用{domain}来代替域名，实际使用时换成具体域名即可。 ## 5.1 Client和Token相关可用版本：2.46+ (CDP**1.15+**) ### 5.1.1 申请Client **接口说明** 本API提供申请Client的方式，申请的Client用于后续与Prep系统进行交互。 **请求地址** ```Plain Text POST https://{domain}/aeolus/prep/userOpenAPI/v1/auth/client ``` **请求参数** | | | | | | \ |**参数** |**类型** |**默认值** |**必填** |**说明** | |---|---|---|---|---| | | | | | | \ |sessionid |string | |是 |cookie中取的用户sessionId | | | | | | | \ |bindingType |int | |是 |0 表示系统级，1 表示用户级 | **样例输入** **返回示例** ```Plain Text { "code": "prep/ok", "message": "成功", "extraMsg": null, "advice": null, "data": { "clientId": "{ClientId}", "clientSecret": "{clientSecret}", "bindingType": "system", "ownerEmailPrefix": "{ownerEmailPrefix}" } } ``` ### 5.1.2 获取用户Token **接口说明** 本API可以获取指定用户的访问Token。 **请求地址** ```Plain Text POST https://{domain}/aeolus/prep/userOpenAPI/v1/auth/token ``` **请求参数** | | | | | | \ |**参数** |**类型** |**默认值** |**必填** |**说明** | |---|---|---|---|---| | | | | | | \ |clientId |string | |是 |auth/client接口获取的clientId | | | | | | | \ |clientSecret |string | |是 |auth/client接口获取的clientSecret | | | | | | | \ |proxyUser |string | |是 |token的用户名称 | | | | | | | \ |expire |int |360 |是 |过期时间，单位为秒，最大的token过期时间为3天 | **返回示例** ```Plain Text { "code": "prep/ok", "message": "成功", "extraMsg": null, "advice": null, "data": { "token": "eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJ0b2tlbl90eXBlIjoib3BlbmFwaSIsInVzZXJuYW1lIjoiZ3VvaGFpc2h1YWkiLCJ1c2VyX3BheWxvYWQiOnt9LCJleHAiOjE2NTQ5MzUyNzcsImNsaWVudF9pZCI6IjY2NjFlNmJiNDgyZjRiMWQ4MzQwZDIyNjhmZDExODY3In0.j-h7tGPOCsOBGQnwf4BER7Aewp2_cOs0ZsUDXEllAk837SBtMAGl9xp4BnS3j9iwceRWc6bEYmVxBzdDCQrt9p12Neeyu-bKscCQ863Er7YPo6-_DaLdjkyf46Wqo52BdNcd4ECjTZj5b1WaDv1dTQ6ar_I7c30yVCT0vKtfFzO4bEzP5LuC26llpDHyLpuTmMQDkqLBulkteLwiym3tdvGidyE1_bhcTooNeZtxtSHKJ7vGj5NxBNBhuX40iXWW7LM4PLvBq7rg1yFuxTlAIJpX9d2tTiyU3_PqRvgaGdj_qLZc6V1IbxgINgrslqurNyD7z_if6-YDKhDFaSQJfth-apCL2FOCoaJFB5s3_kXK3HTh_jD8XTwVhlRtRiPgIz-OC-s_HiHz6iOkhY9MWvvY2IqPW24FZv6OX16-6JHPaKH1Bh1LqPBXVvupKvQRdHT-uvFquvb2avYJeeeR9G02esjuF9ctUrWIBos-63rQH0AtRHw4Swd9Sp2T-sKUI1uVJmx4ylX7pGMQotB5XGQFkb1mOiftBtFQVW6MRP3UKM2jy0QrG55LJAQXcPanLb0kb4BiycbmYrLbj-KsMgG7O2Pp5zw1Jzh-GYiViiTwbAWhPAUxB3JJMFy1jb7tl-DmK47boGyCyJQB79djtd_4UKN3JUfOddsS-xOV7vM" } } ``` ## 5.2 任务接口 ### 5.2.1 创建任务 **接口说明** 通过本接口，可以创建一个Prep任务。 **请求地址** ```Plain Text POST https://{domain}/aeolus/prep/userOpenAPI/v1/task ``` **请求参数** | | | | | | \ |**参数** |**类型** |**默认值** |**必填** |**说明** | |---|---|---|---|---| | | | | | | \ |appId |int | |是 | | | | | | | | \ |name |string | |是 | | | | | | | | \ |taskType |int |0 |否 |0 离线；1 实时；2 标签导入任务 | | | | | | | \ |region |string |"" |否 | | | | | | | | \ |parentId |int | |否 | | | | | | | | \ |alarmConf |object | |否 | | | | | | | | \ |dagConf |object | |否 | | | | | | | | \ |scheduleConf |object | |否 | | | | | | | | \ |subTaskConf |object | |否 | | **请求示例** ```Plain Text { "appId": {appId}, "name": "task_post_*********", "region": "cn", "parentId": 0, "dagConf": { "dag": { "nodes": [ { "key": "Read_aa1_165********", // 节点key "name": "aa1", "type": "Read", // 节点类型，Read输入，Load输出，Filter过滤行，Join连接等 "dataType": "hive", // hive_dataset ch_dataset hive clickhouse等 "nodeStatus": "DEFAULT", "connection": { "clusterName": "cn", "path": "", "dbName": "aaa", "tableName": "aa1", "displayTableName": "aa1", "partitions": [ "e" ] }, // 输入输出节点的数据连接配置 "isDataSetColumn": null, "outputColumns": [ { "name": "w", "type": "string", "originPrepType": "string", "originType": "string", "is_support": 1, "isSelect": true }, { "name": "e", "type": "string", "originPrepType": "string", "originType": "string", "is_support": 1, "isSelect": true } ], // 节点透出的列 "operators": { "filters": { "logic": true, "children": [ { "groupId": 7, "type": "string", "operator": "=", "delimiter": "comma", "operand": [ { "name": "e" }, { "type": "string", "value": "${date}" } ] } ] } }, "params": { "extractType": "{\"key\":\"full\"}", "readNum": "{\"key\":\"sample\",\"value\":\"200\"}" }, "frontendInfo": { "position": { "x": 100, "y": 100 } } } ], "flows": [ { "from": "", // 有向边的头节点key "to": "", // 有向边的尾节点key "fromPort": "", "toPort": "" } ], "frontendInfo": { "layout": "custom", "position": null } } }, "scheduleConf": { "scheduleType": 0, // 调度类型，0手动，1自动调度(dorado调度) "scheduleTimeType": 0, // 运行频率 0: 精确时间; 1: 间隔时间 // 以scheduleTimeSettings= ["13,15,20", "6"]为例 // hourly：13:06、15:06、20:06这三个点进行调度 // daily：每天的13:00、15:00、20:00这三个点进行调度，只取数组的第一个值 // monthly：每月的13,15,20这三天的06:00时间点进行调度 "scheduleFrequency": "hourly", // 调度频率 hourly,daily,weekly, monthly "scheduleTimeSettings": [ "13,15,20", "6" ], // 执行时间：例如分钟级: ["10", "5"]; 小时级: ["13,15,20", "6"]; 周级: ["1,3,4", "02:36"]; 月级: ["16,18,25", "04:27"] "dependencyNodes": [] // 依赖节点 }, // 任务调度配置 "subTaskConf": { "confList": [ { "nodeKey": "", // 输出节点Key "scheduleConf": {}, "advancedParamConf": {}, // 高级参数 "monitorConf": { "alarmRules": [ { "normalNoticeConf": [ { "noticeChannel": "lark", "users": [], // 通知用户 "dutySchedules": [] // 值班计划uk (仅支持域内) 例如:aeolus_dataset } ], // 通知渠道: lark, phone, email, webex "larkGroups": [], // 发送消息到lark群(lark群ID) "emailGroups": [], // 直接发送邮件(邮件地址) "webHookUrl": "", // 推送报警消息的web hook "failedAlarmItems": [ { "item": "fixed_time_not_success", // 运行超时running_timeout, 重试失败retry_failed, 同步任务未开始运行latest_not_start, 同步任务未结束运行latest_not_end, 截止时间同步任务未成功fixed_time_not_success, 结果异常监控项-结果行数result_row_num, 同步超限sync_over_limit "params": { "unit": "", // 单位: hour/minute "interval": "", // 时间间隔 "fixTime": "", // 指定时间 HH:mm "rowNum": "", // 行数 "operator": "" // 操作符: >, =, < } } ], // 失败报警项 "timeoutAlarmItems": [ {} ], // 超时报警项 "resultAlarmItems": [ {} ], // 结果报警项 } ] } } ] }, // 子任务配置 "alarmConf": {} // 监控报警配置 } ``` **返回示例** ```Plain Text { "code": "prep/ok", "message": "成功", "extraMsg": null, "advice": null, "data": { "taskId": 6*** } } ``` ### 5.2.2 获取任务详情 **接口说明** 通过本接口，可以获取指定的任务的详细信息。 **请求地址** ```Plain Text Get https://{domain}/aeolus/prep/userOpenAPI/v1/task ``` **请求参数** | | | | | | | \ |**参数** |**类型** |**默认值** |**必填** |**说明** |**示例** | |---|---|---|---|---|---| | | | | | | | \ |taskId |long | |是 | |6425 | **返回示例** ```Plain Text { "code": "prep/ok", "message": "成功", "extraMsg": null, "advice": null, "data": { "task": { "id": 9***, "appId": {appId}, "name": "ghs_test_task_post", "ownerEmailPrefix": "guohaishuai", "version": 0, "scheduleType": 1, "taskType": 0, "status": 0, "region": "", "parentId": 0, "taskSource": 0, "previewId": null, "executeStatus": 8, "lastSyncTime": null, "deleteTime": null, "mtime": "2022-06-07 22:35:34", "ctime": "2022-06-07 22:35:33", "node": [], "subTask": [], "baseConf": null, "dagConf": { "dag": { "nodes": [], "flows": [], "frontendInfo": { "layout": "custom", "position": null } }, "outputNode": null, "taskId": 9***, "taskType": 0, "ownerEmailPrefix": "guohaishuai", "taskTime": null, "frequency": null, "paramConf": null, "appId": {appId} }, "alarmConf": null, "scheduleConf": { "scheduleType": 0, "scheduleTimeType": 0, "scheduleFrequency": null, "scheduleTimeSettings": [], "interval": null, "intervalUnit": null, "dependencyNodes": [], "clusterId": null, "queue": null, "priority": null }, "advancedParamConf": null, "subTaskConf": { "confList": [] } }, "latestInstance": null, "actions": [ "admin" ] } } ``` ### 5.2.3 更新任务 **接口说明** 通过本接口，可以对已有的任务进行更新。 **请求地址** ```Plain Text PUT https://{domain}/aeolus/prep/userOpenAPI/v1/task ``` **请求参数** | | | | | | \ |**参数** |**类型** |**默认值** |**必填** |**说明** | |---|---|---|---|---| | | | | | | \ |appId |int | |是 | | | | | | | | \ |id |int | |是 | | | | | | | | \ |name |string | |是 | | | | | | | | \ |taskType |int | |否 | | | | | | | | \ |parentId |int | |否 | | | | | | | | \ |taskSource |int | |否 | | | | | | | | \ |alarmConf |object | |否 | | | | | | | | \ |dagConf |object | |是 | | | | | | | | \ |subTaskConf |object | |否 | | | | | | | | \ |scheduleConf |object | |否 | | **请求示例** ```Plain Text { "appId": {appId}, "id": 6087, "previewId": null, "parentId": 0, "taskType": 0, "name": "task_post_0615", "region": "cn", "dagConf": { "dag": { "nodes": [ { "key": "Read_524get_test_**********", "name": "524get_test", "type": "Read", "dataType": "restapi", "nodeStatus": "DEFAULT", "connection": { "clusterName": "cn", "dataSourceId": 185, "path": "", "dbName": "524get_test", "tableName": "524get_test", "displayTableName": "524get_test", "partitions": [] }, "outputColumns": [ { "name": "city", "type": "string", "originPrepType": "string", "originType": "string", "is_support": 1, "isSelect": true }, { "name": "weather", "type": "string", "originPrepType": "string", "originType": "string", "is_support": 1, "isSelect": true } ], "operators": { "filters": {} }, "params": { "extractType": "{\"key\":\"full\"}", "readNum": "{\"key\":\"sample\",\"value\":\"200\"}" }, "frontendInfo": { "position": { "x": 100, "y": 100 } } } ], "flows": [], "frontendInfo": { "layout": "custom", "position": null } } }, "scheduleConf": { "scheduleType": 0, "dependencyNodes": [] }, "subTaskConf": { "confList": [] }, "alarmConf": {} } ``` **返回示例** ```Plain Text { "code": "prep/ok", "message": "成功", "extraMsg": null, "advice": null, "data": { "taskId": 9*** } } ``` ### 5.2.4 更新任务基本信息 **接口说明** 通过本接口，可以更新任务名称。 **请求地址** ```Plain Text PATCH https://{domain}/aeolus/prep/userOpenAPI/v1/task ``` **请求参数** | | | | | | \ |**参数** |**类型** |**默认值** |**必填** |**说明** | |---|---|---|---|---| | | | | | | \ |appId |int | |是 | | | | | | | | \ |id |int | |是 | | | | | | | | \ |name |string | |是 | | **请求示例** ```Plain Text { "appId": {appId}, "id": 6087, "name": "task_post_061*****" } ``` **返回示例** ```Plain Text { "code": "prep/ok", "message": "成功", "extraMsg": null, "advice": null, "data": { "taskId": 6087 } } ``` ### 5.2.5 将任务放入回收站 **接口说明** 通过本接口，可以将指定的任务放入回收站中。 **请求地址** ```Plain Text POST https://{domain}/aeolus/prep/userOpenAPI/v1/task/recycleTask ``` **请求参数** | | | | | | \ |**参数** |**类型** |**默认值** |**必填** |**说明** | |---|---|---|---|---| | | | | | | \ |appId |int | |是 | | | | | | | | \ |taskIdList |array | |是 | | **请求示例** ```Plain Text { "appId": 5****7, "taskIdList": [ 6087 ] } ``` **返回示例** ```Plain Text { "code": "prep/ok", "message": "成功", "extraMsg": null, "advice": null, "data": null } ``` ### 5.2.6 将任务从回收站恢复 **接口说明** 通过本接口，可以将任务从回收站恢复。 **请求地址** ```Plain Text POST https://{domain}/aeolus/prep/userOpenAPI/v1/task/restoreTask ``` **请求参数** | | | | | | \ |**参数** |**类型** |**默认值** |**必填** |**说明** | |---|---|---|---|---| | | | | | | \ |appId |int | |是 | | | | | | | | \ |taskIdList |array | |是 | | **请求示例** ```Plain Text { "appId": {appId}, "taskIdList": [ 6*** ] } ``` **返回示例** ```Plain Text { "code": "prep/ok", "message": "成功", "extraMsg": null, "advice": null, "data": null } ``` ### 5.2.7 将任务彻底删除 **接口说明** 通过本接口，可以将指定任务彻底删除。 **请求地址** ```Plain Text DELETE https://{domain}/aeolus/prep/userOpenAPI/v1/task ``` **请求参数** | | | | | | \ |**参数** |**类型** |**默认值** |**必填** |**说明** | |---|---|---|---|---| | | | | | | \ |appId |int | |是 | | | | | | | | \ |taskIdList |array | |是 | | **请求示例** ```Plain Text { "appId": {appId}, "taskIdList": [ 6089 ] } ``` **返回示例** ```Plain Text { "code": "prep/ok", "message": "成功", "extraMsg": null, "advice": null, "data": null } ``` ### 5.2.8 更改任务的Owner **接口说明** 接口说明通过本接口，可以获取指定用户的访问Token **请求地址** ```Plain Text POST https://{domain}/aeolus/prep/userOpenAPI/v1/task/updateOwner ``` **请求参数** | | | | | | \ |**参数** |**类型** |**默认值** |**必填** |**说明** | |---|---|---|---|---| | | | | | | \ |taskId |int | |是 | | | | | | | | \ |owner |string | |是 |需要更改的owner | **请求示例** ```Plain Text { "owner": "guohaishuai", "taskId": 6091 } ``` **返回示例** ```Plain Text { "code": "prep/ok", "message": "成功", "extraMsg": null, "advice": null, "data": null } ``` ### 5.2.9 获取子任务实例列表 **接口说明** 本接口通过taskId以及各种过滤条件来查询taskId对应任务包含的子任务实例列表。 **请求地址** ```Plain Text GET https://{domain}/aeolus/prep/userOpenAPI/v1/task/subTaskInfoList ``` **请求参数** | | | | | | | \ |**参数** |**类型** |**默认值** |**必填** |**说明** |**示例值** | |---|---|---|---|---|---| | | | | | | | \ |taskId |long | |是 | |6218 | | | | | | | | \ |pageNo |long | |否 | |1 | | | | | | | | \ |pageSize |long | |否 | |30 | | | | | | | | \ |startTaskTime |string | |否 | |2022-06-14 | | | | | | | | \ |endTaskTime |string | |否 | |2022-06-21 | | | | | | | | \ |insStatus |string | |否 | |default | | | | | | | | \ |sortColumns |string | |否 | |taskTime | | | | | | | | \ |sortOrders |string | |否 | |desc | | | | | | | | \ |outputKW |string | |否 | | | | | | | | | | \ |refresh |boolean | |否 | |0 | **返回示例** ```Plain Text { "code": "prep/ok", "message": "成功", "extraMsg": null, "advice": null, "data": { "total": 1, "list": [ { "statusDescCode": 30, "runType": 10, "parentRelationNodes": null, "runTypeAlias": "rerun", "businessId": null, "isBinlogDump": null, "type": "shell", "adminUserOrDepartmentNames": null, "frequency": "daily", "scheduleIp": "2", "loopCount": null, "lookbackNotes": null, "id": "{id}", "flow": false, "consumeTime": "3m21s", "slaTime": null, "taskTimeFormat": "2022-06-07 00:00", "earliestBackfillTime": null, "level": null, "triggerId": 8429044, "priority": "normal", "workName": null, "version": 1, "doradoReturnConsumeTime": "3m 31s ", "avgConsumeTime": null, "lookbackNoteType": "allow", "tags": [], "node": { "nodeName": "输出_1", "partitions": [ "p_date" ], "dataSetStatus": 0, "dbName": "aeolus_data_db_cnch_alpha_lq_202205_dev", "clusterName": "cnch_alpha_lq", "nodeKey": "{nodeKey}", "nodeType": "table", "dataSourceType": "click_house", "tableName": "{tableName}", "status": 0 }, "submitTime": "2022-06-08 10:18:40", "dependencyType": "depend_upstream_all", "sourceType": null, "name": "su_test_任务数量限制_load_16********_***", "frequencyName": "天级", "diagPageUrl": "https://data.bytedance.net/dorado/instance-detail/diagnosis?project=cn_756&id=105917769&schedule=2022-06-07%2000%3A00%3A00&pid=756&status=4&instanceId=988525416&groupName=cn", "region": "cn", "projectName": null, "projectId": 756, "syncStatus": 4, "status": 4, "scheduleRunTime": "2022-06-08 00:00:00", "instanceKey": "880ab9db0a2e992933ef0c3ddd4f2c9d", "virtual": false, "cautionConfirmType": "default", "roles": null, "statusAlias": "succeed", "slaOffset": null, "ownerDisplayName": "sushiyan.test", "clusterName": null, "startTime": "2022-06-08 10:18:50", "runCount": 1, "flowId": -1, "owner": "sushiyan.test", "hasAuth": null, "flowTaskId": -1, "statusDesc": "执行成功", "retryCount": 0, "inputParams": [ { "hasAuth": null, "flowTaskId": null, "debugVal": null, "description": null, "type": "task_custom", "name": "back_track_queue", "typeString": null, "isFlow": null, "taskName": null, "projectName": null, "projectId": null, "value": "back_track_queue", "taskId": null, "order": null, "paramValue": "#" } ], "logPageUrl": "https://data.bytedance.net/dorado/instance-detail/log?project=cn_756&id=105917769&schedule=2022-06-07%2000%3A00%3A00&pid=756&instanceId=988525416&groupName=cn", "relationId": -1276131694, "workId": "104", "cronExpression": "0 0 * * *", "schedule": "2022-06-07 00:00:00", "distanceToBaseIns": null, "createTime": "2022-06-08 10:18:39.000", "location": "cn", "endTime": "2022-06-08 10:22:11", "syncStatusDescCode": 30, "taskId": {taskId}, "queue": null, "dc": null } ] } } ``` ### 5.2.10 获取任务和文件夹 **接口说明：** 接口说明通过本接口，可以获取任务和文件夹 **请求地址** ```Plain Text POST https://{domain}/aeolus/prep/userOpenAPI/v1/task/allTaskOrDirectory ``` **请求参数** | | | | | | | \ |**参数** |**类型** |**默认值** |**必填** |**说明** |**示例** | |---|---|---|---|---|---| | | | | | | | \ |appId |long | |是 | |555327 | | | | | | | | \ |owner |string | |否 | |guohaishuai | | | | | | | | \ |kw |string | |否 | |a | | | | | | | | \ |resType |int | |否 |0 task; 1 文件夹 |0 | | | | | | | | \ |resIds |Array | |否 |数组元素是long | | | | | | | | | \ |kwMatchId |boolean |true |否 | | | | | | | | | | \ |taskSource |int |0 |否 |resId为空时staskSource才生效 | | **返回示例** ```Plain Text { "code": "prep/ok", "message": "成功", "extraMsg": null, "advice": null, "data": [ { "appId": {appId}, "resType": 0, "id": ****, "parentId": 0, "name": "test_w_003", "ownerEmailPrefix": "wanghanqi.john", "taskType": 0, "actions": [], "taskSource": 0 } ] } ``` ### 5.2.11 获取项目下全部任务 **接口说明：** 接口说明通过本接口，可以获取全部文件夹列表。 **请求地址** ```Plain Text POST https://{domain}/aeolus/prep/userOpenAPI/v1/task/allTaskDirectory ``` **请求参数** | | | | | | | \ |**参数** |**类型** |**默认值** |**必填** |**说明** |**示例值** | |---|---|---|---|---|---| | | | | | | | \ |appId |long | |是 | |555327 | | | | | | | | \ |status |int |0 |否 | |0 | | | | | | | | \ |struct |string | |否 | |tree | | | | | | | | \ |kw |string | |否 | | | | | | | | | | \ |emailPrefix |string | |否 | | | | | | | | | | \ |isMine |boolean |false |否 | | | | | | | | | | \ |needActions |boolean |false |否 | | | **返回示例** ```Plain Text { "code": "prep/ok", "message": "成功", "extraMsg": null, "advice": null, "data": [ { "appId": {appId}, "resType": 1, "id": 164, "parentId": 0, "name": "yuqi测试文件夹", "ownerEmailPrefix": "wang****.*****2", "children": [ { "appId": {appId}, "resType": 0, "id": 5653, "parentId": 164, "name": "{name}", "ownerEmailPrefix": "wang****.******", "taskType": 0, "actions": [], "taskSource": 0 } ], "actions": [], "taskSource": 0 }, { "appId": 555327, "resType": 0, "id": 1109, "parentId": 0, "name": "test_w_003", "ownerEmailPrefix": "wanghanqi.john", "taskType": 0, "actions": [], "taskSource": 0 } ] } ``` ### 5.2.12 获取任务依赖列表 **接口说明：** 接口说明通过本接口，可以获取任务的依赖列表。 **请求地址** ```Plain Text POST https://{domain}/aeolus/prep/userOpenAPI/v1/task/dependencyList ``` **请求参数** | | | | | | \ |**参数** |**类型** |**默认值** |**必填** |**说明** | |---|---|---|---|---| | | | | | | \ |appId |int | |是 | | | | | | | | \ |id |int | |是 | | | | | | | | \ |dagConf |object | |是 | | | | | | | | \ |scheduleConf |object | |否 | | **请求示例** ```Plain Text { "appId": {appId}, "id": ****, "dagConf": { "dag": { "nodes": [ { "key": "{key}", "name": "aa1", "type": "Read", "dataType": "hive", "nodeStatus": "DEFAULT", "connection": { "dataSourceId": null, "prepDataSetId": null, "datasetName": "", "fileId": null, "contentType": null, "path": null, "host": null, "port": null, "user": null, "password": null, "catalogName": null, "dbName": "aaa", "driverName": null, "tableName": "aa1", "sql": null, "oriSql": null, "dataSetIdList": null, "partitions": [ "e" ], "encryptMethod": 0, "additionalParamsStr": null, "jdbcConnectionString": null, "shardNum": 0, "chPartitionType": null, "clusterName": "cn", "shardColumn": null, "bootstrapServers": null, "topicName": null, "groupId": null, "checkpointLocation": null, "displayTableName": "aa1" }, "sampleJson": null, "isDataSetColumn": null, "outputColumns": [ { "name": "w", "type": "string", "originType": "string", "is_support": 1, "isSelect": true }, { "name": "e", "type": "string", "originType": "string", "is_support": 1, "isSelect": true } ], "operators": { "filters": { "logic": true, "children": [ { "type": "string", "operator": "=", "operand": [ { "name": "e" }, { "type": "string", "value": "${date}" } ], "children": [] } ] } }, "params": { "extractType": "{\"key\":\"full\"}", "readNum": "{\"key\":\"sample\",\"value\":\"200\"}" }, "advanceSettings": [], "frontendInfo": { "layout": null, "position": { "x": 100, "y": 100 } } }, { "key": "Load_165*********", "name": "输出_1", "type": "Load", "dataType": "hive_dataset", "nodeStatus": "DEFAULT", "frontendInfo": { "position": { "x": 320, "y": 100 } } } ], "flows": [ { "from": "Read_aa1_165*********", "to": "Load_165**********", "fromPort": "output", "toPort": "input" } ], "frontendInfo": { "layout": "custom", "position": null } } }, "scheduleConf": { "dependencyNodes": [ { "nodeKey": "Load_165*********", "dependencyMethod": 0 } ] } } ``` **返回示例** ```Plain Text { "code": "prep/ok", "message": "成功", "extraMsg": null, "advice": null, "data": [ { "nodeKey": "Load_165*********", "nodeName": "输出_1", "dependencies": [ { "dbName": "dm_lab_speech", "tableName": "ailab_speech_mdd_session_all", "dataType": "hive", "subTasks": [ { "id": {id}, "nodeKey": null, "nodeName": null, "taskId": null, "taskName": "HPSensor_dm_lab_speech.ailab_speech_mdd_session_all_#1", "source": "raw_data", "frequency": null, "scheduleConf": { "scheduleType": 1, "scheduleTimeType": 0, "scheduleFrequency": "daily", "scheduleTimeSettings": [], "interval": null, "intervalUnit": 0 }, "offsetInfo": { "offsets": [ 0 ], "offsetType": "set", "offsetFrequency": "daily" }, "mtime": null, "dependencyStatus": 1, "dependencyType": 1, "recommendTaskInfo": { "projectId": 5, "databaseName": "dm_lab_speech", "tableName": "ailab_speech_mdd_session_all", "frequency": "daily", "path": "date=${date}/", "type": "hive_partition-sensor", "taskPartitions": [ { "value": "${date}", "key": "date" } ] } } ], "inputNodes": [ { "nodeKey": "Read_ailab_speech_mdd_session_all_165*********", "nodeName": "ailab_speech_mdd_session_all" } ], "readPartitionInfo": { "logic": true, "type": null, "operator": null, "operand": null, "children": [ { "logic": null, "type": "string", "operator": "=", "operand": [ { "name": "date", "distinct": false }, { "type": "string", "value": "${date}", "distinct": false } ], "children": [] } ] } } ], "nodeDepStatus": 1, "dependencyMethod": 0, "filterUnCompleteSensor": null, "isSelfDepend": false, "earliestBackTime": null } ] } ``` ### 5.2.13 获取预览数据 **接口说明：** 接口说明通过本接口，可以获取任务的预览数据。 **请求地址** ```Plain Text POST https://{domain}/aeolus/prep/userOpenAPI/v1/task/previewSample ``` **请求参数** | | | | | | \ |**参数** |**类型** |**默认值** |**必填** |**说明** | |---|---|---|---|---| | | | | | | \ |appId |int | |是 | | | | | | | | \ |dagConf |object | |是 | | | | | | | | \ |ownerEmailPrefix |string | |是 | | **请求示例** ```Plain Text { "ownerEmailPrefix": "********", "dagConf": { "dag": { "nodes": [ { "outputColumns": [ { "name": "city", "originType": "string", "namespace": "cn_***_***", "hasColPriviledge": null, "value": null, "alias": null, "isSelect": true, "type": "string" }, { "name": "weather", "originType": "string", "namespace": "cn_***_***", "hasColPriviledge": null, "value": null, "alias": null, "isSelect": true, "type": "string" } ], "name": "cn_***_***", "nodeStatus": null, "operators": {}, "connection": { "shardNum": 0, "prepDataSetId": null, "contentType": 0, "clusterName": "cn", "dataSourceType": "restapi", "chPartitionType": null, "tableName": "185", "dbName": "185", "dataSourceId": 185, "user": null, "sql": "", "path": null, "catalogName": null, "prepDataSourceType": null, "shardColumn": null, "partitions": [] }, "subTaskId": null, "dataType": "restapi", "advanceSettings": [], "taskTime": null, "key": "cn_185_185", "params": { "extractType": "{\"key\":\"full\"}" }, "type": "Read" }, { "name": "{name}", "dataType": "Column", "operators": { "fields": [ { "name": "city", "type": "string", "namespace": "cn_***_***", "hasColPriviledge": null, "value": null, "alias": "city", "isSelect": true, "originType": "string" }, { "name": "weather", "type": "string", "namespace": "cn_***_***", "hasColPriviledge": null, "value": null, "alias": "weather", "isSelect": true, "originType": "string" } ], "preNodes": [], "filters": {}, "joinNodes": [] }, "nodeStatus": "DEFAULT", "key": "{key}", "type": "Reset" } ], "flows": [ { "toPort": "input", "to": "Reset_688*********", "from": "cn_***_***", "fromPort": "output" } ] }, "outputNode": { "key": "{key}" } }, "appId": {appId} } ``` **返回示例** ```Plain Text { "code": "prep/ok", "message": "成功", "extraMsg": null, "advice": null, "data": [ { "type": "data", "name": "output", "type": "data", "data": { "values": [ [ "东城区", "阴" ] ], "fields": [ { "name": "city", "type": "string", "comment": null, "metadata": {} }, { "name": "weather", "type": "string", "comment": null, "metadata": {} } ] }, "total": 0, "size": 0, "exceed": null } ] } ``` ### 5.2.14 更新标签任务 **接口说明** 接口说明通过本接口，可以更新标签任务。 **请求地址** ```Plain Text PUT https://{domain}/aeolus/prep/userOpenAPI/v1/task/labelTask ``` **请求参数** | | | | | | \ |**参数** |**类型** |**默认值** |**必填** |**说明** | |---|---|---|---|---| | | | | | | \ |labelId |long | |是 | | | | | | | | \ |prepId |long | |是 | | | | | | | | \ |nodeKey |string | |是 | | | | | | | | \ |labelType |string | |是 | | | | | | | | \ |labelName |string | |是 | | | | | | | | \ |devId |long | |否 | | | | | | | | \ |task |object | |是 | | | | | | | | \ |labelEntityType |string | |否 | | **请求示例** ```Plain Text { "labelId": 378, "prepId": 6450, "nodeKey": "load_label_164******", "labelType": "string", "labelName": "age", "devId": null, "task": { "id": 4388, "appId": {appId}, "name": "测试新需求", "ownerEmailPrefix": "tangying.ty", "version": 22, "scheduleType": 1, "taskType": 0, "status": 0, "region": "cn", "parentId": 0, "taskSource": 0, "previewId": "{previewId}", "executeStatus": 5, "lastSyncTime": "2022-07-12 00:22:01", "deleteTime": null, "mtime": "2022-07-12 00:22:08", "ctime": "2022-01-05 18:29:43", "baseConf": null, "dagConf": { "dag": { "nodes": [ { "name": "user_cdp", "type": "Read", "nodeStatus": "SUCCEEDED", "subTaskId": null, "dataType": "hive", "taskTime": "2022-07-11T00:00:00.000+0800", "key": "{key}", "connection": { "dataSourceId": null, "prepDataSetId": null, "datasetName": "", "dataSourceType": "hive", "fileId": null, "contentType": null, "path": null, "host": null, "port": null, "user": null, "password": null, "catalogName": null, "dbName": "cdp", "driverName": null, "tableName": "user_cdp", "sql": null, "oriSql": null, "dataSetIdList": null, "partitions": [ "p_date=undefined" ], "encryptMethod": 0, "additionalParams": null, "additionalParamsStr": null, "jdbcConnectionString": null, "shardNum": 0, "chPartitionType": null, "clusterName": "cn", "shardColumn": null, "bootstrapServers": null, "topicName": null, "groupId": null, "checkpointLocation": null, "displayTableName": "user_cdp" }, "outputColumns": [ { "name": "user_id", "type": "long", "originType": "bigint", "isSelect": true, "isAutoDataType": false }, { "name": "gender", "type": "string", "originType": "string", "isSelect": true, "isAutoDataType": false }, { "name": "age", "type": "string", "originType": "string", "isSelect": true, "isAutoDataType": false }, { "name": "edu", "type": "string", "originType": "string", "isSelect": true, "isAutoDataType": false }, { "name": "duozhi", "type": "array", "originType": "array", "isSelect": true, "isAutoDataType": false }, { "name": "birthday", "type": "string", "originType": "string", "isSelect": true, "isAutoDataType": false }, { "name": "birthdaytime", "type": "string", "originType": "string", "isSelect": true, "isAutoDataType": false }, { "name": "xiaoshu", "type": "string", "originType": "string", "isSelect": true, "isAutoDataType": false }, { "name": "job", "type": "string", "originType": "string", "isSelect": true, "isAutoDataType": false }, { "name": "marry", "type": "string", "originType": "string", "isSelect": true, "isAutoDataType": false }, { "name": "phone", "type": "string", "originType": "string", "isSelect": true, "isAutoDataType": false }, { "name": "phoneid", "type": "string", "originType": "string", "isSelect": true, "isAutoDataType": false }, { "name": "price", "type": "string", "originType": "string", "isSelect": true, "isAutoDataType": false }, { "name": "operater", "type": "string", "originType": "string", "isSelect": true, "isAutoDataType": false }, { "name": "recent_city", "type": "string", "originType": "string", "isSelect": true, "isAutoDataType": false }, { "name": "city", "type": "string", "originType": "string", "isSelect": true, "isAutoDataType": false }, { "name": "city_level", "type": "string", "originType": "string", "isSelect": true, "isAutoDataType": false }, { "name": "base_id", "type": "long", "originType": "bigint", "isSelect": true, "isAutoDataType": false }, { "name": "p_date", "type": "string", "originType": "string", "isSelect": true, "isAutoDataType": false } ], "inputColumns": null, "sampleJson": null, "engineType": null, "startStatistic": false, "operators": { "filters": { "logic": true, "type": "string", "operator": null, "operand": [], "children": [ { "logic": null, "type": "string", "operator": "=", "operand": [ { "name": "p_date", "isAutoDataType": false, "distinct": false }, { "type": "string", "value": "2021-09-01", "isAutoDataType": false, "distinct": false } ], "children": [] } ] }, "fields": null, "group": null, "joinNodes": null, "preNodes": null, "preNodeInfos": null, "replaceColumn": null, "flows": null, "joinConfs": null, "unpivotConf": null, "mapColumnInfoList": null, "pivotConf": null }, "advanceSettings": [], "frontendInfo": { "layout": null, "position": { "x": 100, "y": 100 } }, "isDataSetColumn": false, "params": { "extractType": "{\"key\":\"full\"}", "readNum": "{\"key\":\"sample\",\"value\":\"200\"}" } }, { "name": "计算列_1", "type": "Add", "nodeStatus": "SUCCEEDED", "subTaskId": null, "dataType": "Expr", "taskTime": null, "key": "{key}", "connection": null, "outputColumns": null, "inputColumns": null, "sampleJson": null, "engineType": null, "startStatistic": false, "operators": { "filters": null, "fields": [ { "name": "pdate", "type": "date", "value": "base64;e1RBU0tfREFURX0=", "isAutoDataType": false, "distinct": false } ], "group": null, "joinNodes": null, "preNodes": null, "preNodeInfos": null, "replaceColumn": null, "flows": null, "joinConfs": null, "unpivotConf": null, "mapColumnInfoList": null, "pivotConf": null }, "advanceSettings": null, "frontendInfo": { "layout": null, "position": { "x": 320, "y": 100 } }, "isDataSetColumn": false, "params": {} }, { "name": "字段设置_1", "type": "Reset", "nodeStatus": "SUCCEEDED", "subTaskId": null, "dataType": "Column", "taskTime": null, "key": "{key}", "connection": null, "outputColumns": null, "inputColumns": null, "sampleJson": null, "engineType": null, "startStatistic": false, "operators": { "filters": null, "fields": [ { "name": "user_id", "type": "long", "originType": "long", "alias": "user_id", "isSelect": true, "isAutoDataType": false, "distinct": false }, { "name": "gender", "type": "string", "originType": "string", "alias": "gender", "isSelect": true, "isAutoDataType": false, "distinct": false }, { "name": "age", "type": "string", "originType": "string", "alias": "age", "isSelect": true, "isAutoDataType": false, "distinct": false }, { "name": "edu", "type": "string", "originType": "string", "alias": "edu", "isSelect": true, "isAutoDataType": false, "distinct": false }, { "name": "duozhi", "type": "array", "originType": "array", "alias": "duozhi", "isSelect": true, "isAutoDataType": false, "distinct": false }, { "name": "birthday", "type": "date", "originType": "string", "alias": "birthday", "isSelect": true, "isAutoDataType": false, "distinct": false }, { "name": "birthdaytime", "type": "timestamp", "originType": "string", "alias": "birthdaytime", "isSelect": true, "isAutoDataType": false, "distinct": false }, { "name": "xiaoshu", "type": "string", "originType": "string", "alias": "xiaoshu", "isSelect": true, "isAutoDataType": false, "distinct": false }, { "name": "job", "type": "string", "originType": "string", "alias": "job", "isSelect": true, "isAutoDataType": false, "distinct": false }, { "name": "marry", "type": "string", "originType": "string", "alias": "marry", "isSelect": true, "isAutoDataType": false, "distinct": false }, { "name": "phone", "type": "string", "originType": "string", "alias": "phone", "isSelect": true, "isAutoDataType": false, "distinct": false }, { "name": "phoneid", "type": "string", "originType": "string", "alias": "phoneid", "isSelect": true, "isAutoDataType": false, "distinct": false }, { "name": "price", "type": "string", "originType": "string", "alias": "price", "isSelect": true, "isAutoDataType": false, "distinct": false }, { "name": "operater", "type": "string", "originType": "string", "alias": "operater", "isSelect": true, "isAutoDataType": false, "distinct": false }, { "name": "recent_city", "type": "string", "originType": "string", "alias": "recent_city", "isSelect": true, "isAutoDataType": false, "distinct": false }, { "name": "city", "type": "string", "originType": "string", "alias": "city", "isSelect": true, "isAutoDataType": false, "distinct": false }, { "name": "city_level", "type": "string", "originType": "string", "alias": "city_level", "isSelect": true, "isAutoDataType": false, "distinct": false }, { "name": "base_id", "type": "long", "originType": "long", "alias": "base_id", "isSelect": true, "isAutoDataType": false, "distinct": false }, { "name": "p_date", "type": "string", "originType": "string", "alias": "p_date", "isSelect": true, "isAutoDataType": false, "distinct": false }, { "name": "pdate", "type": "date", "originType": "date", "alias": "pdate", "isSelect": true, "isAutoDataType": false, "distinct": false } ], "group": null, "joinNodes": null, "preNodes": null, "preNodeInfos": null, "replaceColumn": null, "flows": null, "joinConfs": null, "unpivotConf": null, "mapColumnInfoList": null, "pivotConf": null }, "advanceSettings": null, "frontendInfo": { "layout": null, "position": { "x": 540, "y": 100 } }, "isDataSetColumn": false, "params": {} }, { "name": "输出标签_1", "type": "load_label", "nodeStatus": "DEFAULT", "subTaskId": null, "dataType": "", "taskTime": null, "key": "{key}", "connection": { "dataSourceId": null, "prepDataSetId": null, "datasetName": "", "dataSourceType": "click_house", "fileId": null, "contentType": null, "path": null, "host": null, "port": null, "user": null, "password": null, "catalogName": null, "dbName": "cdp_internal", "driverName": null, "tableName": "tag_data_source", "sql": null, "oriSql": null, "dataSetIdList": null, "partitions": null, "encryptMethod": 1, "additionalParams": null, "additionalParamsStr": null, "jdbcConnectionString": null, "shardNum": 0, "chPartitionType": null, "clusterName": "cdp_to_test", "shardColumn": null, "bootstrapServers": null, "topicName": null, "groupId": null, "checkpointLocation": null, "displayTableName": null }, "outputColumns": null, "inputColumns": null, "sampleJson": null, "engineType": null, "startStatistic": false, "operators": { "filters": null, "fields": null, "group": null, "joinNodes": null, "preNodes": null, "preNodeInfos": null, "replaceColumn": null, "flows": null, "joinConfs": null, "unpivotConf": null, "mapColumnInfoList": null, "pivotConf": null }, "advanceSettings": null, "frontendInfo": { "layout": null, "position": { "x": 760, "y": 100 } }, "isDataSetColumn": false, "params": { "labelEntityType": "主体1", "chloadColumnKeys": "base_id", "chloadColumns": "birthday", "labelName": "模型日期1", "labelDesc": null, "dataSecurityLevel": "L3", "isOnlineApiService": "false" } }, { "name": "输出标签_2", "type": "load_label", "nodeStatus": "DEFAULT", "subTaskId": null, "dataType": "", "taskTime": null, "key": "{key}", "connection": { "dataSourceId": null, "prepDataSetId": null, "datasetName": "", "dataSourceType": "click_house", "fileId": null, "contentType": null, "path": null, "host": null, "port": null, "user": null, "password": null, "catalogName": null, "dbName": "cdp_internal", "driverName": null, "tableName": "tag_data_source", "sql": null, "oriSql": null, "dataSetIdList": null, "partitions": null, "encryptMethod": 1, "additionalParams": null, "additionalParamsStr": null, "jdbcConnectionString": null, "shardNum": 0, "chPartitionType": null, "clusterName": "cdp_to_test", "shardColumn": null, "bootstrapServers": null, "topicName": null, "groupId": null, "checkpointLocation": null, "displayTableName": null }, "outputColumns": null, "inputColumns": null, "sampleJson": null, "engineType": null, "startStatistic": false, "operators": { "filters": null, "fields": null, "group": null, "joinNodes": null, "preNodes": null, "preNodeInfos": null, "replaceColumn": null, "flows": null, "joinConfs": null, "unpivotConf": null, "mapColumnInfoList": null, "pivotConf": null }, "advanceSettings": null, "frontendInfo": { "layout": null, "position": { "x": 760, "y": 220 } }, "isDataSetColumn": false, "params": { "labelEntityType": "主体1", "chloadColumnKeys": "base_id", "chloadColumns": "birthdaytime", "labelName": "模型日期时间1", "labelDesc": null, "dataSecurityLevel": "L4", "isOnlineApiService": "false" } }, { "name": "输出标签_3", "type": "load_label", "nodeStatus": "DEFAULT", "subTaskId": null, "dataType": "", "taskTime": null, "key": "{key}", "connection": { "dataSourceId": null, "prepDataSetId": null, "datasetName": "", "dataSourceType": "click_house", "fileId": null, "contentType": null, "path": null, "host": null, "port": null, "user": null, "password": null, "catalogName": null, "dbName": "cdp_internal", "driverName": null, "tableName": "tag_data_source", "sql": null, "oriSql": null, "dataSetIdList": null, "partitions": null, "encryptMethod": 1, "additionalParams": null, "additionalParamsStr": null, "jdbcConnectionString": null, "shardNum": 0, "chPartitionType": null, "clusterName": "cdp_to_test", "shardColumn": null, "bootstrapServers": null, "topicName": null, "groupId": null, "checkpointLocation": null, "displayTableName": null }, "outputColumns": null, "inputColumns": null, "sampleJson": null, "engineType": null, "startStatistic": false, "operators": { "filters": null, "fields": null, "group": null, "joinNodes": null, "preNodes": null, "preNodeInfos": null, "replaceColumn": null, "flows": null, "joinConfs": null, "unpivotConf": null, "mapColumnInfoList": null, "pivotConf": null }, "advanceSettings": null, "frontendInfo": { "layout": null, "position": { "x": 760, "y": -20 } }, "isDataSetColumn": false, "params": { "labelEntityType": "主体1", "chloadColumnKeys": "base_id", "chloadColumns": "marry", "labelName": "模型文本marry", "labelDesc": null, "dataSecurityLevel": "L3", "isOnlineApiService": "false" } }, { "name": "输出标签_4", "type": "load_label", "nodeStatus": "DEFAULT", "subTaskId": null, "dataType": "", "taskTime": null, "key": "{key}", "connection": { "dataSourceId": null, "prepDataSetId": null, "datasetName": "", "dataSourceType": "click_house", "fileId": null, "contentType": null, "path": null, "host": null, "port": null, "user": null, "password": null, "catalogName": null, "dbName": "cdp_internal", "driverName": null, "tableName": "tag_data_source", "sql": null, "oriSql": null, "dataSetIdList": null, "partitions": null, "encryptMethod": 1, "additionalParams": null, "additionalParamsStr": null, "jdbcConnectionString": null, "shardNum": 0, "chPartitionType": null, "clusterName": "cdp_to_test", "shardColumn": null, "bootstrapServers": null, "topicName": null, "groupId": null, "checkpointLocation": null, "displayTableName": null }, "outputColumns": null, "inputColumns": null, "sampleJson": null, "engineType": null, "startStatistic": false, "operators": { "filters": null, "fields": null, "group": null, "joinNodes": null, "preNodes": null, "preNodeInfos": null, "replaceColumn": null, "flows": null, "joinConfs": null, "unpivotConf": null, "mapColumnInfoList": null, "pivotConf": null }, "advanceSettings": null, "frontendInfo": null, "isDataSetColumn": false, "params": { "labelEntityType": "主体1", "chloadColumnKeys": "base_id", "chloadColumns": "job", "labelName": "{labelName}", "labelDesc": null, "dataSecurityLevel": "L3", "isOnlineApiService": "false" } } ], "flows": [ { "from": "Read_user_cdp_164**********", "to": "compute_col_164**********", "fromPort": "output", "toPort": "input" }, { "from": "compute_col_164**********", "to": "reset_column_164**********", "fromPort": "output", "toPort": "input" }, { "from": "reset_column_164**********", "to": "load_label_164**********", "fromPort": "output", "toPort": "input" }, { "from": "reset_column_164**********", "to": "load_label_164**********", "fromPort": "output", "toPort": "input" }, { "from": "reset_column_164**********", "to": "load_label_164**********", "fromPort": "output", "toPort": "input" }, { "from": "reset_column_164**********", "to": "load_label_164**********", "fromPort": "output", "toPort": "input" } ], "frontendInfo": null }, "outputNode": { "key": "load_label_164**********" }, "taskId": 4388, "taskType": 0, "ownerEmailPrefix": "tangying.ty", "taskTime": null, "frequency": null, "paramConf": { "engineParams": [ { "key": "spark.driver.extraJavaOptions", "value": "-Ddb2.jcc.charsetDecoderEncoder=3" }, { "key": "spark.executor.extraJavaOptions", "value": "-Ddb2.jcc.charsetDecoderEncoder=3" } ], "nonEngineParams": [] }, "appId": null }, "scheduleConf": { "scheduleType": 1, "scheduleTimeType": 0, "scheduleFrequency": "daily", "scheduleTimeSettings": [ "00:01" ], "interval": null, "intervalUnit": null, "dependencyNodes": [ { "nodeKey": "load_label_164**********", "nodeName": "输出标签_1", "dependencies": [ { "dbName": "cdp", "tableName": "user_cdp", "dataType": "hive", "subTasks": [ { "id": 105168936, "nodeKey": null, "nodeName": null, "taskId": null, "taskName": "HPSensor_cdp.user_cdp_#1", "source": "raw_data", "frequency": null, "scheduleConf": { "scheduleType": 1, "scheduleTimeType": 0, "scheduleFrequency": "daily", "scheduleTimeSettings": [], "interval": null, "intervalUnit": 0 }, "offsetInfo": { "offsets": [ 0 ], "offsetType": "set", "offsetFrequency": "daily" }, "mtime": null, "dependencyStatus": 1, "dependencyType": 1, "recommendTaskInfo": { "projectId": 5, "databaseName": "cdp", "tableName": "user_cdp", "frequency": "daily", "path": "p_date=2021-09-01/", "type": "hive_partition-sensor", "taskPartitions": [ { "value": "2021-09-01", "key": "p_date" } ] } } ], "inputNodes": [ { "nodeKey": "Read_user_cdp_164**********", "nodeName": "user_cdp" } ], "readPartitionInfo": { "logic": true, "type": null, "operator": null, "operand": null, "children": [ { "logic": null, "type": "string", "operator": "=", "operand": [ { "name": "p_date", "isAutoDataType": false, "distinct": false }, { "type": "string", "dataType": "string", "value": "2021-09-01", "isAutoDataType": false, "distinct": false } ], "children": [] } ] } } ], "nodeDepStatus": 1, "dependencyMethod": 0, "filterUnCompleteSensor": null, "isSelfDepend": false, "earliestBackTime": null }, { "nodeKey": "load_label_164**********", "nodeName": "输出标签_2", "dependencies": [ { "dbName": "cdp", "tableName": "user_cdp", "dataType": "hive", "subTasks": [ { "id": 105168936, "nodeKey": null, "nodeName": null, "taskId": null, "taskName": "HPSensor_cdp.user_cdp_#1", "source": "raw_data", "frequency": null, "scheduleConf": { "scheduleType": 1, "scheduleTimeType": 0, "scheduleFrequency": "daily", "scheduleTimeSettings": [], "interval": null, "intervalUnit": 0 }, "offsetInfo": { "offsets": [ 0 ], "offsetType": "set", "offsetFrequency": "daily" }, "mtime": null, "dependencyStatus": 1, "dependencyType": 1, "recommendTaskInfo": { "projectId": 5, "databaseName": "cdp", "tableName": "user_cdp", "frequency": "daily", "path": "p_date=2021-09-01/", "type": "hive_partition-sensor", "taskPartitions": [ { "value": "2021-09-01", "key": "p_date" } ] } } ], "inputNodes": [ { "nodeKey": "Read_user_cdp_164**********", "nodeName": "user_cdp" } ], "readPartitionInfo": { "logic": true, "type": null, "operator": null, "operand": null, "children": [ { "logic": null, "type": "string", "operator": "=", "operand": [ { "name": "p_date", "isAutoDataType": false, "distinct": false }, { "type": "string", "dataType": "string", "value": "2021-09-01", "isAutoDataType": false, "distinct": false } ], "children": [] } ] } } ], "nodeDepStatus": 1, "dependencyMethod": 0, "filterUnCompleteSensor": null, "isSelfDepend": false, "earliestBackTime": null }, { "nodeKey": "load_label_164**********", "nodeName": "输出标签_3", "dependencies": [ { "dbName": "cdp", "tableName": "user_cdp", "dataType": "hive", "subTasks": [ { "id": 105168936, "nodeKey": null, "nodeName": null, "taskId": null, "taskName": "HPSensor_cdp.user_cdp_#1", "source": "raw_data", "frequency": null, "scheduleConf": { "scheduleType": 1, "scheduleTimeType": 0, "scheduleFrequency": "daily", "scheduleTimeSettings": [], "interval": null, "intervalUnit": 0 }, "offsetInfo": { "offsets": [ 0 ], "offsetType": "set", "offsetFrequency": "daily" }, "mtime": null, "dependencyStatus": 1, "dependencyType": 1, "recommendTaskInfo": { "projectId": 5, "databaseName": "cdp", "tableName": "user_cdp", "frequency": "daily", "path": "p_date=2021-09-01/", "type": "hive_partition-sensor", "taskPartitions": [ { "value": "2021-09-01", "key": "p_date" } ] } } ], "inputNodes": [ { "nodeKey": "Read_user_cdp_164**********", "nodeName": "user_cdp" } ], "readPartitionInfo": { "logic": true, "type": null, "operator": null, "operand": null, "children": [ { "logic": null, "type": "string", "operator": "=", "operand": [ { "name": "p_date", "isAutoDataType": false, "distinct": false }, { "type": "string", "dataType": "string", "value": "2021-09-01", "isAutoDataType": false, "distinct": false } ], "children": [] } ] } } ], "nodeDepStatus": 1, "dependencyMethod": 0, "filterUnCompleteSensor": null, "isSelfDepend": false, "earliestBackTime": null }, { "nodeKey": "load_label_164**********", "nodeName": "输出标签_4", "dependencies": [ { "dbName": "cdp", "tableName": "user_cdp", "dataType": "hive", "subTasks": [ { "id": 105168936, "nodeKey": null, "nodeName": null, "taskId": null, "taskName": "HPSensor_cdp.user_cdp_#1", "source": "raw_data", "frequency": null, "scheduleConf": { "scheduleType": 1, "scheduleTimeType": 0, "scheduleFrequency": "daily", "scheduleTimeSettings": [], "interval": null, "intervalUnit": 0 }, "offsetInfo": { "offsets": [ 0 ], "offsetType": "set", "offsetFrequency": "daily" }, "mtime": null, "dependencyStatus": 1, "dependencyType": 1, "recommendTaskInfo": { "projectId": 5, "databaseName": "cdp", "tableName": "user_cdp", "frequency": "daily", "path": "p_date=2021-09-01/", "type": "hive_partition-sensor", "taskPartitions": [ { "value": "2021-09-01", "key": "p_date" } ] } } ], "inputNodes": [ { "nodeKey": "Read_user_cdp_164**********", "nodeName": "user_cdp" } ], "readPartitionInfo": { "logic": true, "type": null, "operator": null, "operand": null, "children": [ { "logic": null, "type": "string", "operator": "=", "operand": [ { "name": "p_date", "isAutoDataType": false, "distinct": false }, { "type": "string", "dataType": "string", "value": "2021-09-01", "isAutoDataType": false, "distinct": false } ], "children": [] } ] } } ], "nodeDepStatus": 1, "dependencyMethod": 0, "filterUnCompleteSensor": null, "isSelfDepend": false, "earliestBackTime": null } ], "clusterId": null, "queue": null, "priority": null }, "advancedParamConf": null, "subTaskConf": { "confList": [ { "nodeKey": "load_label_164**********", "scheduleConf": { "clusterId": null, "queue": null, "priority": null, "scheduleType": 0, "scheduleTimeType": null, "scheduleFrequency": null, "scheduleTimeSettings": [], "interval": null, "intervalUnit": null, "hasSavePoint": false, "consumerStatus": 0, "isDynamicPartition": false, "retryNum": 1, "retryInterval": 10 }, "advancedParamConf": { "params": [] }, "monitorConf": { "alarmRules": [] } }, { "nodeKey": "load_label_164**********", "scheduleConf": { "clusterId": null, "queue": null, "priority": null, "scheduleType": 0, "scheduleTimeType": null, "scheduleFrequency": null, "scheduleTimeSettings": [], "interval": null, "intervalUnit": null, "hasSavePoint": false, "consumerStatus": 0, "isDynamicPartition": false, "retryNum": 1, "retryInterval": 10 }, "advancedParamConf": { "params": [] }, "monitorConf": { "alarmRules": [] } }, { "nodeKey": "load_label_164**********", "scheduleConf": { "clusterId": null, "queue": null, "priority": null, "scheduleType": 0, "scheduleTimeType": null, "scheduleFrequency": null, "scheduleTimeSettings": [], "interval": null, "intervalUnit": null, "hasSavePoint": false, "consumerStatus": 0, "isDynamicPartition": false, "retryNum": 1, "retryInterval": 10 }, "advancedParamConf": { "params": [] }, "monitorConf": { "alarmRules": [] } }, { "nodeKey": "load_label_164**********", "scheduleConf": { "clusterId": null, "queue": null, "priority": null, "scheduleType": 0, "scheduleTimeType": null, "scheduleFrequency": null, "scheduleTimeSettings": [], "interval": null, "intervalUnit": null, "hasSavePoint": false, "consumerStatus": 0, "isDynamicPartition": false, "retryNum": 1, "retryInterval": 10 }, "advancedParamConf": { "params": [] }, "monitorConf": { "alarmRules": [] } } ] } }, "labelEntityType": null } ``` **返回示例** ```Plain Text { "code": "prep/ok", "data": null, "extra_msg": null, "msg": "成功" } ``` ### 5.2.15 获取依赖指定表的子任务 **接口说明** 接口说明通过本接口，可以获取依赖某个表的子任务列表。 **请求地址** ```Plain Text GET https://{domain}/aeolus/prep/userOpenAPI/v1/task/scheduleTasksInfoByDbTable ``` **请求参数** | | | | | | | \ |**参数** |**类型** |**默认值** |**必填** |**说明** |**示例** | |---|---|---|---|---|---| | | | | | | | \ |dataSourceType |string | |是 | |hive | | | | | | | | \ |dbName |string | |是 | |dw_aeolus | | | | | | | | \ |tableName |string | |是 | |tb_acc0ddaa638ca3b0a74bab211cb7a9f4 | **返回示例** ```Plain Text { "code": "prep/ok", "message": "成功", "extraMsg": null, "advice": null, "data": { "dependencies": [ { "taskType": 0, "taskScheduleType": 1, "scheduleTaskId": {scheduleTaskId}, "projectId": ***, "frequency": "daily", "scheduleType": 0, "taskId": ****, "subTaskId": **** } ] } } ``` ### 5.2.16 获取依赖指定表列表的子任务列表 **接口说明** 接口说明通过本接口，可以获取某些表的子任务列表 **请求地址** ```Plain Text GET https://{domain}/aeolus/prep/userOpenAPI/v1/task/scheduleTasksInfoListByDbTableList ``` **请求参数** | | | | | | \ |**参数** |**类型** |**默认值** |**必填** |**说明** | |---|---|---|---|---| | | | | | | \ |dbTableList |array | |是 |{dataSourceType, dbName, tableName} | **请求示例** ```Plain Text { "dbTableList": [{ "dataSourceType": "hive", "dbName": "aaa", "tableName": "event_log_hourly" }] } ``` **返回示例** 正确示例： ```Plain Text { "code": "prep/ok", "message": "成功", "extraMsg": null, "advice": null, "data": [ { "dbName": "aaa", "tableName": "event_log_hourly", "dataSourceType": "hive", "dependencies": [] } ] } ``` ### 5.2.17 通过模板创建任务 **接口说明** 接口说明通过本接口，可以通过模板创建任务。 **请求地址** ```Plain Text POST https://{domain}/aeolus/prep/userOpenAPI/v1/task/createByTemplate ``` **请求参数** | | | | | | \ |**参数** |**类型** |**默认值** |**必填** |**说明** | |---|---|---|---|---| | | | | | | \ |id |long | |是 | | | | | | | | \ |taskId |long | |否 | | | | | | | | \ |doradoIdList |array | |否 |数组元素是long | | | | | | | \ |appId; |long | |是 | | | | | | | | \ |name |string | |是 | | | | | | | | \ |ownerEmailPrefix |string | |是 | | | | | | | | \ |status |int | |否 | | | | | | | | \ |taskType |int |1 |否 | | | | | | | | \ |taskSource |int | |否 | | | | | | | | \ |region |string | |否 | | | | | | | | \ |params |map | |是 |map | | | | | | | \ |needRun |boolean |true |否 | | **请求示例** ```Plain Text { "id": 9, "taskId": 0, "appId": 555327, "region": "cn", "name": "{name}", "ownerEmailPrefix": "******.****", "taskType": 0, "needRun": false, "params": { "redis.psm": "vpc.tendis.cluster", "ch.clusterName": "rangers", "ch.sql": "select toInt32(1) as `id`, toFloat32('\''1.1'\''), toFloat64('\''1.1'\''), toDate('\''2022-03-03'\'')", "redis.key.prefix": "caokai_test", "redis.ttl": "0" } } ``` **返回示例** ```Plain Text { "code": "prep/ok", "data": null, "extra_msg": null, "msg": "成功" } ``` ### 5.2.18 根据Prep数据类型批量获取引擎数据类型 **接口说明** 接口说明通过本接口，可以获取引擎数据类型 **请求地址** ```Plain Text GET https://{domain}/aeolus/prep/userOpenAPI/v1/task/engineTypeFromPrepTypeBatch ``` **请求参数** | | | | | | | \ |**参数** |**类型** |**默认值** |**必填** |**说明** |**示例** | |---|---|---|---|---|---| | | | | | | | \ |dataSourceType |string | |是 | |hive | | | | | | | | \ |typeList |string | |是 | |long;short | **返回示例** ```Plain Text { "code": "prep/ok", "data": null, "extra_msg": null, "msg": "成功" } ``` ### 5.2.19 根据引擎数据类型批量获取Prep数据类型 **接口说明** 接口说明通过本接口，可以获取Prep数据类型。 **请求地址** ```Plain Text GET https://{domain}/aeolus/prep/userOpenAPI/v1/task/prepTypeFromEngineTypeBatch ``` **请求参数** | | | | | | | \ |**参数** |**类型** |**默认值** |**必填** |**说明** |**示例** | |---|---|---|---|---|---| | | | | | | | \ |dataSourceType |string | |是 | |hive | | | | | | | | \ |typeList |string | |是 | |bigint;smallint | **返回示例** ```Plain Text { "code": "prep/ok", "data": null, "extra_msg": null, "msg": "成功" } ``` ### 5.2.20 批量获取任务基本信息 **接口说明** 接口说明通过本接口，可以批量获取任务的基本信息 **请求地址** ```Plain Text POST https://{domain}/aeolus/prep/userOpenAPI/v1/task/taskBasicInfoBatch ``` **请求参数** param： | | | | | | | \ |**参数** |**类型** |**默认值** |**必填** |**说明** |**示例** | |---|---|---|---|---|---| | | | | | | | \ |skipIfNotFound |string |false |否 | |true | body： | | | | | | | \ |**参数** |**类型** |**默认值** |**必填** |**说明** |**示例** | |---|---|---|---|---|---| | | | | | | | \ |taskIds |array | |是 | |[6087, 12345678] | **返回示例** ```Plain Text { "code": "prep/ok", "data": null, "extra_msg": null, "msg": "成功" } ``` ### 5.2.21 获取Prep数据类型列表 **接口说明** 本接口可以获取Prep中不同数据源的数据类型列表。 **请求地址** ```Plain Text GET https://{domain}/aeolus/prep/userOpenAPI/v1/task/prepTypeList ``` **请求参数** | | | | | | | \ |**参数** |**类型** |**默认值** |**必填** |**说明** |**示例** | |---|---|---|---|---|---| | | | | | | | \ |dataSourceType |string | |否 | |hive | **返回示例** ```Plain Text { "code": "prep/ok", "message": "成功", "extraMsg": null, "advice": null, "data": [ "string", "int", "double", "date", "timestamp", "long", "short", "float", "decimal", "binary", "boolean", "byte" ] } ``` ### 5.2.22 获取子任务的执行DAG **接口说明** 本接口可以获取子任务的执行DAG。 **请求地址** ```Plain Text GET https://{domain}/aeolus/prep/userOpenAPI/v1/task/taskDagInfoList ``` **请求参数** | | | | | | | \ |**参数** |**类型** |**默认值** |**必填** |**说明** |**示例** | |---|---|---|---|---|---| | | | | | | | \ |taskId |int | |是 | |6447 | | | | | | | | \ |taskTime |string | |是 | | | | | | | | | | \ |relation |string | |是 | | | | | | | | | | \ |depth |int | |是 | | | | | | | | | | \ |scheduleTaskId |int | |否 | | | **返回示例** ```Plain Text { "code": "prep/ok", "message": "成功", "extraMsg": null, "advice": null, "data": null } ``` ### 5.2.23 获取流任务的列表 **接口说明** 本接口可以获取Prep中流任务列表。 **请求地址** ```Plain Text GET https://{domain}/aeolus/prep/userOpenAPI/v1/streamTask/allListByAppId ``` **请求参数** | | | | | | | \ |**参数** |**类型** |**默认值** |**必填** |**说明** |**示例** | |---|---|---|---|---|---| | | | | | | | \ |appId |int | |是 | |555327 | | | | | | | | \ |type |string | |是 | |kafka | **返回示例** ```Plain Text { "code": "prep/ok", "message": "成功", "extraMsg": null, "advice": null, "data": null } ``` ### 5.2.24 获取节点Schema信息 **接口说明** 本接口可以获取节点Schema信息。 **请求地址** ```Plain Text POST https://{domain}/aeolus/prep/userOpenAPI/v1/task/resultSchema ``` **请求参数** | | | | | | \ |**参数** |**类型** |**默认值** |**必填** |**说明** | |---|---|---|---|---| | | | | | | \ |portName |string | |否 | | **请求体** | | | | | | \ |**参数** |**类型** |**默认值** |**必填** |**说明** | |---|---|---|---|---| | | | | | | \ |dagConf |object | |是 | | | | | | | | \ |ownerEmailPrefix |string | |是 | | **请求示例** ```Plain Text { "dagConf": { "dag": { "nodes": [ { "key": "{key}", "name": "{name}", "type": "Read", "dataType": "restapi", "nodeStatus": "DEFAULT", "connection": { "dataSourceId": 185, "prepDataSetId": null, "datasetName": "", "fileId": null, "contentType": null, "path": null, "host": null, "port": null, "user": null, "password": null, "catalogName": null, "dbName": "524get_test", "driverName": null, "tableName": "524get_test", "sql": null, "oriSql": null, "dataSetIdList": null, "partitions": [], "encryptMethod": 0, "additionalParamsStr": null, "jdbcConnectionString": null, "shardNum": 0, "chPartitionType": null, "clusterName": "cn", "shardColumn": null, "bootstrapServers": null, "topicName": null, "groupId": null, "checkpointLocation": null, "displayTableName": "*****" }, "sampleJson": null, "isDataSetColumn": null, "outputColumns": [ { "name": "city", "type": "string", "originType": "string", "is_support": 1, "isSelect": true }, { "name": "weather", "type": "string", "originType": "string", "is_support": 1, "isSelect": true } ], "operators": { "filters": {} }, "params": { "extractType": "{\"key\":\"full\"}", "readNum": "{\"key\":\"sample\",\"value\":\"200\"}" }, "advanceSettings": [], "frontendInfo": { "layout": null, "position": { "x": 100, "y": 100 } } } ], "flows": [] }, "outputNode": { "key": "{key}" } }, "ownerEmailPrefix": "guohaishuai" } ``` **返回示例** ```Plain Text { "code": "prep/ok", "message": "成功", "extraMsg": null, "advice": null, "data": null } ``` ## 5.3 任务实例 ### 5.3.1 获取任务最近实例 **接口说明** 接口说明通过本接口，可以获取任务子任务的最新实例id。 **请求地址** ```Plain Text POST https://{domain}/aeolus/prep/userOpenAPI/v1/task/latestInstance ``` **请求参数** | | | | | | \ |**参数** |**类型** |**默认值** |**必填** |**说明** | |---|---|---|---|---| | | | | | | \ |taskId |int | |是 | | **请求示例** ```Plain Text { "taskId": 6359 } ``` **返回示例** ```Plain Text { "code": "prep/ok", "data": { "scheduleInstanceId": 123 // dorado任务实例ID }, "extra_msg": null, "msg": "成功" } ``` ### 5.3.2 调度任务搜索 **接口说明** 本接口可以查询调度任务。 **请求地址** ```Plain Text POST https://{domain}/aeolus/prep/userOpenAPI/v1/task/doradoTaskSearch ``` **请求参数** | | | | | | \ |**参数** |**类型** |**默认值** |**必填** |**说明** | |---|---|---|---|---| | | | | | | \ |pageNo |int |1 |否 | | | | | | | | \ |pageSize |int |20 |否 | | | | | | | | \ |category |string |mixed |否 | | | | | | | | \ |key |string | |否 | | | | | | | | \ |keyword |string | |否 | | | | | | | | \ |frequencySet |array | |否 | | | | | | | | \ |searchForDependencies |boolean | |否 | | **请求示例** ```Plain Text { "appId": {appId}, "pageNo": 1, "keyword": "a", "frequencySet": [] } ``` **返回示例** ```Plain Text { "code": "prep/ok", "message": "成功", "extraMsg": null, "advice": null, "data": null } ``` ### 5.3.3 设置任务的回调 **接口说明** 接口说明通过本接口，可以设置任务的回调信息。 **请求地址** ```Plain Text POST https://{domain}/aeolus/prep/userOpenAPI/v1/task/callback ``` **请求参数** | | | | | | \ |**参数** |**类型** |**默认值** |**必填** |**说明** | |---|---|---|---|---| | | | | | | \ |dataSourceType |string | |是 | | | | | | | | \ |dbName |string | |是 | | | | | | | | \ |tableName |string | |是 | | | | | | | | \ |dataTableId |long | |是 | | | | | | | | \ |tableSchemaUpdateCallbackPath |string | |是 | | | | | | | | \ |tableDependencyUpdateCallbackPath |string | |是 | | **请求示例** ```Plain Text { "dataSourceType": "click_house", "dbName": "{dbName}", "tableName": "{tableName}", "dataTableId": 93, "tableSchemaUpdateCallbackPath": "/profile_platform/openApi/v2/atlas/labelMeta/syncDataSourceTableMeta?sourceToken=aeolusPrep", "tableDependencyUpdateCallbackPath": "/profile_platform/openApi/v2/atlas/labelMeta/modifyDataSourceTablePrepDepends?sourceToken=aeolusPrep" } ``` **返回示例** ```Plain Text { "code": "prep/ok", "data": null, "extra_msg": null, "msg": "成功" } ``` ### 5.3.4 任务手动回溯 **接口说明** 接口说明通过本接口，执行指定时间范围的任务。 **请求地址** ```Plain Text POST https://{domain}/aeolus/prep/userOpenAPI/v1/task/execute ``` **请求参数** | | | | | | \ |**参数** |**类型** |**默认值** |**必填** |**说明** | |---|---|---|---|---| | | | | | | \ |taskId |int | |是 | | | | | | | | \ |nodeKeys |string[] | |是 |输出节点的key | | | | | | | \ |startTime |string | |是 |回溯的时间范围-起始时间 |\ | | | | |格式取决于同步频次 |\ | | | | |yyyy-MM-dd 默认格式 |\ | | | | |yyyy-MM-dd HH 小时级 hourly |\ | | | | |yyyy-MM-dd HH:mm 十分钟级 |\ | | | | |every_ten_minutes | | | | | | | \ |endTime |string | |是 |回溯的时间范围-结束时间 | | | | | | | \ |skipCheck |boolean |false |否 | | | | | | | | \ |partitionCheck |boolean |false |否 | | | | | | | | \ |isSpecifyQueue |boolean |false |否 | | | | | | | | \ |queueName |string | |否 | | **请求示例** ```Plain Text { "taskId": 6359, "nodeKeys": ["Load_xxx"], "startTime": "2023-02-01", "endTime": "2023-02-13" } ``` **返回示例** ```Plain Text { "code": "prep/ok", "data": true, "extra_msg": null, "msg": "成功" } ``` ## 5.4 数据集 ### 5.4.1 创建数据集 **接口说明** 本接口可以创建数据集，数据集可以用于任务的输出节点。 **请求地址** ```Plain Text POST https://{domain}/aeolus/prep/userOpenAPI/v1/dataset ``` **请求参数** | | | | | | | \ |**参数** |**类型** |**默认值** |**必填** |**说明** | | |---|---|---|---|---|---| | | | | | | | \ |name |string | |是 | | | | | | | | | | \ |appId |int | |是 |项目ID | | | | | | | | | \ |ownerEmailPrefix |string | |是 |用户名 | | | | | | | | | \ |dataSetSourceId |int |0 |否 |数据集来源id, 默认0 | | | | | | | | | \ |clusterName |string | |是 |集群名 | | | | | | | | | \ |dbName |string | |是 |库名 | | | | | | | | | \ |tableName |string | |是 |表名 | | | | | | | | | \ |dataSourceType |string | |是 |hive |click_house | | | | | | | | \ |dataSetSource |int | |否 |0:prep, 1: abi, 2: cdp | | | | | | | | | \ |type |string | |否 |1:实时，0:离线 | | | | | | | | | \ |dataSetConf |object | |否 | | | **请求示例** ```Plain Text { "name": "dataset_from_post_mann", "appId": 555327, "ownerEmailPrefix": "guohaishuai", "dataSetSourceId": {id}, "clusterName": "cn", "dbName": "dw_aeolus", "tableName": "{tableName}", "dataSourceType": "hive", "dataSetSource": 1, "type": 0 } ``` **返回示例** ```Plain Text { "code": "prep/ok", "message": "成功", "extraMsg": null, "advice": null, "data": { "id": 21, "name": "dataset_from_post", "descr": "", "appId": 1, "ownerEmailPrefix": "admin", "dataSetSource": 2, "dataSetSourceId": 2, "abiDataSetId": 0, "dataSourceType": "hive", "resourceGroup": 0, "region": "cn", "clusterName": "cn", "dbName": "dw_aeolus", // 数据集底表的库名 "tableName": "bank_churners", // 数据集底表的表名 "tableSize": 0, "tableLineNum": 0, "dataSetLabelType": 0, "dataSetTaskType": 0, "confidentiality": "L3", "ttl": null, "lineOfBusiness": [], "regionType": [], "productName": "", "generalTagSourceIds": [], "authKey": "prep_data_set_21", "status": 0, "extraAppIds": null, "ctime": null, "mtime": null, "dataSetConf": null, "isPersonalDataRelated": false, "dataRegion": null } } ``` ### 5.4.2 获取数据集信息 **接口说明** 本接口可以获取数据集的详细信息。 **请求地址** ```Plain Text GET https://{domain}/aeolus/prep/userOpenAPI/v1/dataset ``` **请求参数** | | | | | | | \ |**参数** |**类型** |**默认值** |**必填** |**说明** |**示例** | |---|---|---|---|---|---| | | | | | | | \ |dataSetId |int | |是 | |17830 | **返回示例** ```Plain Text { "code": "prep/ok", "message": "成功", "extraMsg": null, "advice": "", // 错误建议，可空 "data": { "id": 21, "name": "dataset_from_post", "descr": "", "appId": 1, "ownerEmailPrefix": "admin", "dataSetSource": 2, "dataSetSourceId": 2, "abiDataSetId": 0, "dataSourceType": "hive", "resourceGroup": 0, "region": "cn", "clusterName": "cn", "dbName": "dw_aeolus", // 数据集底表的库名 "tableName": "bank_churners", // 数据集底表的表名 "tableSize": 0, "tableLineNum": 0, "dataSetLabelType": 0, "dataSetTaskType": 0, "confidentiality": "L3", "ttl": 1, // 底表的ttl，单位为天，null/0 表示没有ttl "lineOfBusiness": [], "regionType": [], "productName": "", "generalTagSourceIds": [], "authKey": "prep_data_set_21", "status": 0, "extraAppIds": [], // List 镜像数据集app集合 "ctime": 1, // 创建时间，毫秒时间戳 "mtime": 1, // 修改时间，毫秒时间戳 "dataSetConf": {}, // 主要存kafka信息或cdw实时导入任务id等信息 "isPersonalDataRelated": false, // 资源是否包含用户信息 "dataRegion": [] // List 用户信息所属地区 } } ``` ### 5.4.3 获取数据集的Schema信息 **接口说明** 本接口可以获取数据集的Schema信息。 **请求地址** ```Plain Text GET https://{domain}/aeolus/prep/userOpenAPI/v1/dataset/dataSetSchema ``` **请求参数** | | | | | | | \ |**参数** |**类型** |**默认值** |**必填** |**说明** |**示例** | |---|---|---|---|---|---| | | | | | | | \ |dataSetId |int | |是 | |17830 | **返回示例** ```Plain Text { "code": "prep/ok", "message": "成功", "extraMsg": null, "advice": null, "data": null } ``` ### 5.4.4 根据库表名获取对应的数据集信息 **接口说明** 本接口可以获取Prep中不同数据源的数据类型列表。 **请求地址** ```Plain Text GET https://{domain}/aeolus/prep/userOpenAPI/v1/dataset/getDataSetListByTb ``` **请求参数** | | | | | | | \ |**参数** |**类型** |**默认值** |**必填** |**说明** |**示例** | |---|---|---|---|---|---| | | | | | | | \ |dbName |string | |是 | |dw_aeolus | | | | | | | | \ |tableName |string | |是 | |tb_282b9e7e3b5b8877489811f2fa8f6fe3 | | | | | | | | \ |dataSourceType |string | |是 | |hive | **返回示例** ```Plain Text { "code": "prep/ok", "message": "成功", "extraMsg": null, "advice": null, "data": [ "string", "int", "double", "date", "timestamp", "long", "short", "float", "decimal", "binary", "boolean", "byte" ] } ``` ### 5.4.5 获取数据集列表 **接口说明** 本接口可以获取Prep中数据集列表（未分类）。 **请求地址** ```Plain Text GET https://{domain}/aeolus/prep/userOpenAPI/v1/dataset/dataSetListWithoutClassification ``` **请求参数** | | | | | | | \ |**参数** |**类型** |**默认值** |**必填** |**说明** |**示例** | |---|---|---|---|---|---| | | | | | | | \ |appId |int | |是 |项目ID |555327 | | | | | | | | \ |ownerEmailPrefix |string | |是 |用户名 |guohaishuai | | | | | | | | \ |kw |string | |否 | |test_712_ | **返回示例** ```Plain Text { "code": "prep/ok", "message": "成功", "extraMsg": null, "advice": null, "data": [ { "id": {id}, "name": "{name}", "descr": "", "appId": {appId}, "ownerEmailPrefix": "{ownerEmailPrefix}", "dataSourceType": "click_house", "dataSetSource": 0, "region": "cn", "clusterName": "multi_disk_test", "dbName": "aeolus_data_db_multi_disk_test_202207_dev", "tableName": "{tableName}", "ctime": "2022-07-12T23:06:31.000+0800", "mtime": "2022-07-12T23:07:02.000+0800", "roleList": [ { "roleName": "管理", "action": "admin" }, { "roleName": "编辑", "action": "write_role" }, { "roleName": "查看", "action": "read_role" } ], "taskOwnerRoleList": [], "authKey": "prep_data_set_17835", "abiDataSetId": ****** } ] } ``` ## 5.5 SQL和DDL相关 ### 5.5.1 解析SQL中依赖的表和列 **接口说明** 本接口可以获取给定SQL中依赖的数据表和列。 **请求地址** ```Plain Text POST https://{domain}/aeolus/prep/userOpenAPI/v1/sql ``` **请求参数** | | | | | | \ |**参数** |**类型** |**默认值** |**必填** |**说明** | |---|---|---|---|---| | | | | | | \ |engineType |string | |是 | | | | | | | | \ |sql |string | |是 | | **请求示例** ```Plain Text { "engineType": "hive", "sql": "select a, aa, cd from aaa.aaaa where aa='abc'" } ``` **返回示例** ```Plain Text { "code": "prep/ok", "message": "成功", "extraMsg": null, "advice": null, "data": [ { "dbName": "aaa", "tableName": "aaaa", "columns": [ { "name": "cd", "keys": [] }, { "name": "adc", "keys": [] }, { "name": "aa", "keys": [] }, { "name": "aa", "keys": [] } ] } ] } ``` ### 5.5.2 获取SQL中的任务依赖 **接口说明** 本接口可以获取Prep中不同数据源的数据类型列表。 **请求地址** ```Plain Text POST https://{domain}/aeolus/prep/userOpenAPI/v1/sql/dependencies ``` **请求参数** | | | | | | \ |**参数** |**类型** |**默认值** |**必填** |**说明** | |---|---|---|---|---| | | | | | | \ |engineType |string | |是 | | | | | | | | \ |sql |string | |是 | | **请求示例** ```Plain Text { "engineType": "hive", "sql": "select app_id from dw_aeolus.tb_458f78fe05eccdb8********* where date = '${date}'" } ``` **返回示例** ```Plain Text { "code": "prep/ok", "message": "成功", "extraMsg": null, "advice": null, "data": [ { "dbName": null, "tableName": null, "dataType": "hive", "subTasks": [ { "id": 105976995, "nodeKey": null, "nodeName": null, "taskId": null, "taskName": "标签测试_load_label_165*********", "source": "raw_data", "frequency": null, "scheduleConf": { "scheduleType": 1, "scheduleTimeType": 0, "scheduleFrequency": "daily", "scheduleTimeSettings": [], "interval": null, "intervalUnit": 0 }, "offsetInfo": { "offsets": [ 0 ], "offsetType": "set", "offsetFrequency": "daily" }, "mtime": null, "dependencyStatus": 1, "dependencyType": 1, "recommendTaskInfo": { "projectId": 756, "databaseName": null, "tableName": null, "frequency": "daily", "path": null, "type": "shell", "taskPartitions": [] } } ], "inputNodes": null, "readPartitionInfo": { "logic": true, "type": null, "operator": null, "operand": null, "children": [] } } ] } ``` ### 5.5.3 创建数据表 **接口说明** 本接口可以根据传入的信息创建对应的。 **请求地址** ```Plain Text POST https://{domain}/aeolus/prep/userOpenAPI/v1/ddl/createTable ``` **请求参数** | | | | | | \ |**参数** |**类型** |**默认值** |**必填** |**说明** | |---|---|---|---|---| | | | | | | \ |sourceTable |object | |是 |示例：{ |\ | | | | |"clusterName": "rangers", |\ | | | | |"dataSourceType": "click_house", |\ | | | | |"tableName": "tb_038bbfcabd5ba614bf*********", |\ | | | | |"dbName": "aeolus_data_db_rangers_202206", |\ | | | | |"partitions": [ |\ | | | | |"p_date" |\ | | | | |] |\ | | | | |} | | | | | | | \ |sourceRegisterInfo |object | |是 |示例：{ |\ | | | | |"partitionSourceType": "fromSource", |\ | | | | |"ownerEmailPrefix": "admin", |\ | | | | |"baseIdName": "base_id", |\ | | | | |"appId": 1, |\ | | | | |"partitionName": "p_date", |\ | | | | |"tableType": "click_house", |\ | | | | |"columns": [ |\ | | | | |{ |\ | | | | |"type": "Nullable(Int64)", |\ | | | | |"name": "uid2" |\ | | | | |}, |\ | | | | |{ |\ | | | | |"type": "Nullable(Int64)", |\ | | | | |"name": "uid3" |\ | | | | |}, |\ | | | | |{ |\ | | | | |"type": "Nullable(String)", |\ | | | | |"name": "open_id" |\ | | | | |}, |\ | | | | |{ |\ | | | | |"type": "Date", |\ | | | | |"name": "p_date" |\ | | | | |}, |\ | | | | |{ |\ | | | | |"type": "Nullable(String)", |\ | | | | |"name": "app_id" |\ | | | | |}, |\ | | | | |{ |\ | | | | |"type": "Nullable(Int64)", |\ | | | | |"name": "phone" |\ | | | | |}, |\ | | | | |{ |\ | | | | |"type": "Int64", |\ | | | | |"name": "row_id" |\ | | | | |}, |\ | | | | |{ |\ | | | | |"type": "Nullable(Int64)", |\ | | | | |"name": "uid" |\ | | | | |} |\ | | | | |] |\ | | | | |} | **请求示例** ```Plain Text { "sourceRegisterInfo": { "partitionSourceType": "fromSource", "ownerEmailPrefix": "admin", "baseIdName": "base_id", "appId": 1, "partitionName": "p_date", "tableType": "click_house", "columns": [ { "type": "Nullable(Int64)", "name": "uid2" }, { "type": "Nullable(Int64)", "name": "uid3" }, { "type": "Nullable(String)", "name": "open_id" }, { "type": "Date", "name": "p_date" }, { "type": "Nullable(String)", "name": "app_id" }, { "type": "Nullable(Int64)", "name": "phone" }, { "type": "Int64", "name": "row_id" }, { "type": "Nullable(Int64)", "name": "uid" } ] }, "sourceTable": { "clusterName": "rangers", "dataSourceType": "click_house", "tableName": "tb_038bbfcabd5ba614bf9e**********", "dbName": "aeolus_data_db_rangers_202206", "partitions": [ "p_date" ] } } ``` **返回示例** ```Plain Text { "code": "prep/ok", "message": "成功", "extraMsg": null, "advice": null, "data": { "tableSchema": { "columnSchema": [ { "name": "uid2", "type": "long", "isEngineType": false, "engineType": "Int64" }, { "name": "uid3", "type": "long", "isEngineType": false, "engineType": "Int64" }, { "name": "open_id", "type": "string", "isEngineType": false, "engineType": "String" }, { "name": "p_date", "type": "date", "isEngineType": false, "engineType": "Date" }, { "name": "app_id", "type": "string", "isEngineType": false, "engineType": "String" }, { "name": "phone", "type": "long", "isEngineType": false, "engineType": "Int64" }, { "name": "row_id", "type": "long", "isEngineType": false, "engineType": "Int64" }, { "name": "uid", "type": "long", "isEngineType": false, "engineType": "Int64" }, { "name": "base_id", "type": "long", "isEngineType": false, "engineType": "Int64" } ], "primaryKeyList": [ "base_id" ], "shardKey": "base_id", "sampleKey": "base_id", "ttl": 0, "partitionKeyList": [ { "name": "p_date", "exprName": "p_date", "pattern": "${DATE}", "values": [] } ], "kafkaTopic": null, "kafkaCluster": null, "groupId": null, "dataSourceId": null, "hiveTableParams": null }, "clusterName": "cdp_to_test", "dbName": "cdp_internal", "tableName": "tb_038bbfcabd5ba614b__prep", "region": "cn", "dataSetTaskType": null, "baseIdColumn": null } } ``` ### 5.5.4 创建数据表-v2 POST https://{domain}/aeolus/prep/userOpenAPI/v1/table/ddl/prepTable **请求参数** ```Plain Text { "tableType": "clickhouse", // 表类型，hive/clickhouse "appId": 1, // 项目ID "ttl": 3, // TTL，单位天，默认0永久 "clusterName": "public_cn", // 集群名 "dbName": "aeolus_data_db_public_cn_202006", // 库名 "tableName": "jk_test_tab221121_ck", // 表名 "region": "cn", // 区域 "tableSchema": { "primaryKeyList": [ "p_date" ], "columnSchema": [ { "type": "string", "name": "open_id" }, { "type": "date", "name": "p_date" }, { "type": "long", "name": "uid" } ], // 数据列信息 "shardKey": "uid", "sampleKey": "uid", "partitionKeyList": [ { "name": "p_date" } ] // 分区字段 } } ``` ### 5.5.5 恢复表分区 POST https://{domain}/aeolus/prep/userOpenAPI/v1/table/ddl/recoverPartitions **请求参数** ```Plain Text { "tableType":"hive", "dbName":"dw_aeolus", "tableName":"asset_car_tob_pudongyongda_bhv" } ``` ### 5.5.6 添加表分区 POST https://{domain}/aeolus/prep/userOpenAPI/v1/table/ddl/tablePartition | | | | | | \ |**参数** |**类型** |**默认值** |**必填** |**说明** | |---|---|---|---|---| | | | | | | \ |tableType |string |hive |是 |默认填hive | | | | | | | \ |dbName |string | |是 |数据库名 | | | | | | | \ |tableName |string | |是 |表名 | | | | | | | \ |location | | |是 |分区存储的路径，可以是s3、hdfs、oss | | | | | | | \ |partitionSpec |json | |是 |用于分区字段和分区字段值的描述 | **请求参数** ```Plain Text { "tableType":"hive", "dbName":"dw_aeolus", "tableName":"tb_addd3ca12278f116d*********", "location":"/user/hive/warehouse/dw_aeolus.db/tb_addd3ca12278f116dafc*********/date=20201022", "partitionSpec":[{"partitionCol":"date","partitionColValue":"20201022"}] } ``` ### 5.5.7 删除表分区 DELETE https://{domain}/aeolus/prep/userOpenAPI/v1/table/ddl/tablePartition **请求参数** | | | | | | \ |**参数** |**类型** |**默认值** |**必填** |**说明** | |---|---|---|---|---| | | | | | | \ |tableType |string |hive |是 |默认填hive | | | | | | | \ |dbName |string | |是 |数据库名 | | | | | | | \ |tableName |string | |是 |表名 | | | | | | | \ |partitionSpec |json | |是 |用于分区字段和分区字段值的描述 | ```Plain Text { "tableType":"hive", "dbName":"dw_aeolus", "tableName":"tb_addd3ca12278f116**************", "partitionSpec":[{"partitionCol":"date","partitionColValue":"20201022"}] } ``` ### 5.5.8 设置表TTL POST https://{domain}/aeolus/prep/userOpenAPI/v1/table/ddl/tableTTL | | | | | | \ |**参数** |**类型** |**默认值** |**必填** |**说明** | |---|---|---|---|---| | | | | | | \ |tableType |string |hive |是 |默认填hive | | | | | | | \ |dbName |string | |是 |数据库名 | | | | | | | \ |tableName |string | |是 |表名 | | | | | | | \ |ttl |int | |是 |表ttl | | | | | | | \ |ownerEmailPrefix |string |admin |是 |用户名 | **请求参数** ```Plain Text { "tableType":"hive", "dbName":"dw_aeolus", "tableName":"test_table_ttl", "ttl":2, "ownerEmailPrefix":"admin" } ``` ## 5.6 数据连接 ### 5.6.1 创建数据连接 **接口说明** 本接口可以根据传入的信息创建数据连接 **请求地址** ```Plain Text POST https://{domain}/aeolus/prep/userOpenAPI/v1/upstream ``` **请求参数** | | | | | | \ |**参数** |**类型** |**默认值** |**必填** |**说明** | |---|---|---|---|---| | | | | | | \ |appId |int | |是 |项目ID | | | | | | | \ |type |string | |是 |数据连接类型，示例：mysql | | | | | | | \ |datasourceName |string | |是 | | | | | | | | \ |connectionParams |object | |是 |数据连接参数，示例：{ |\ | | | | |"host": "[10.225.118.181](/docs/4726/65184)", |\ | | | | |"port": "3306", |\ | | | | |"user": "aeolus_db_w", |\ | | | | |"password": "xxx" |\ | | | | |} | | | | | | | \ |uuid |string | |否 |示例：c428121e-654c-4433-8a53-87555a9c0be4 | | | | | | | \ |fileIds |int[] | |否 |文件数据连接ID列表 | **请求示例** ```Plain Text { "appId": 555327, "type": "mysql", "datasourceName": "{datasourceName}", "connectionParams": { "host": "10.225.***.***", "port": "****", "user": "aeolus_db_w", "password": "xxx" }, "uuid": "{uuid}" } ``` **返回示例** ```Plain Text { "code": "prep/ok", "message": "成功", "extraMsg": null, "advice": null, "data": 24461 } ``` ### 5.6.2 删除数据连接 **接口说明** 本接口可以根据传入的信息删除数据连接 **请求地址** ```Plain Text DELETE https://{domain}/aeolus/prep/userOpenAPI/v1/upstream ``` **请求参数** | | | | | | \ |**参数** |**类型** |**默认值** |**必填** |**说明** | |---|---|---|---|---| | | | | | | \ |upstreamDatasourceId |int | |是 |数据连接数据源ID | | | | | | | \ |appId |int | |是 |项目ID | | | | | | | \ |type |string | |是 |数据源类型，示例：mysql | | | | | | | \ |fileIds |int[] | |否 |文件数据连接ID列表 | **请求示例** ```Plain Text { "upstreamDatasourceId": *****, "appId": {appId}, "type": "mysql" } ``` **返回示例** ```Plain Text { "code": "prep/ok", "message": "成功", "extraMsg": null, "advice": null, "data": true } ``` ### 5.6.3 修改数据连接 **接口说明** 本接口可以根据传入的信息修改数据连接。 **请求地址** ```Plain Text PUT https://{domain}/aeolus/prep/userOpenAPI/v1/upstream ``` **请求参数** | | | | | | \ |**参数** |**类型** |**默认值** |**必填** |**说明** | |---|---|---|---|---| | | | | | | \ |id |int | |是 |数据连接 ID | | | | | | | \ |appId |int | |是 |项目ID | | | | | | | \ |type |string | |是 |数据连接类型，示例：mysql | | | | | | | \ |datasourceName |string | |是 | | | | | | | | \ |connectionParams |object | |是 |数据连接参数，示例：{ |\ | | | | |"host": "[10.225.118.181](/docs/4726/65184)", |\ | | | | |"port": "3306", |\ | | | | |"user": "aeolus_db_w", |\ | | | | |"password": "xxx" |\ | | | | |} | | | | | | | \ |uuid |string | |否 |示例：c428121e-654c-4433-8a53-87555a9c0be4 | | | | | | | \ |fileIds |int[] | |否 |文件数据连接ID列表 | **请求示例** ```Plain Text { "id": 24461, "appId": {appId}, "type": "mysql", "datasourceName": "数据连接_202209*********", "connectionParams": { "host": "10.225.***.***", "port": "****", "user": "aeolus_db_w", "password": "xxx" }, "uuid": "{uuid}" } ``` **返回示例** ```Plain Text { "code": "prep/ok", "message": "成功", "extraMsg": null, "advice": null, "data": true } ``` ### 5.6.4 查询数据连接 **接口说明** 本接口可以根据传入的信息查询数据连接 **请求地址** ```Plain Text GET https://{domain}/aeolus/prep/userOpenAPI/v1/upstream ``` **请求参数** | | | | | | \ |**参数** |**类型** |**默认值** |**必填** |**说明** | |---|---|---|---|---| | | | | | | \ |id |int | |是 |查询参数，数据连接 ID | **请求示例** ```Plain Text curl https://${domain}/aeolus/prep/userOpenAPI/v1/upstream?id=24461 ``` **返回示例** ```Plain Text { "code": "prep/ok", "message": "成功", "extraMsg": null, "advice": null, "data": { "id": 24461, "name": "数据连接_20220908*******", "appId": 555327, "host": "10.225.***.***", "port": "****", "type": "mysql", "user": "aeolus_db_w", "additionalParams": {}, "doradoDatasourceId": *****, "ownerEmailPrefix": "****.*******", "encryptMethod": 1, "ctime": "2022-09-08 18:17:24", "mtime": "2022-09-08 18:19:41", "roleList": [], "taskList": [] } } ``` ### 5.6.5 获取数据连接元信息 **接口说明** 本接口可以获得数据连接的元信息 **请求地址** ```Plain Text GET https://{domain}/aeolus/prep/userOpenAPI/v1/upstream/metadata ``` **请求参数** | | | | | | \ |**参数** |**类型** |**默认值** |**必填** |**说明** | |---|---|---|---|---| | | | | | | \ |type |string | |是 |数据连接类型 | **请求示例** ```Plain Text curl https://${domain}/aeolus/prep/userOpenAPI/v1/upstream/metadata?type=mysql ``` **返回结果** "data"部分为type类型数据连接的元信息，不同类型数据连接元信息存在差异，元信息对应建立数据连接时需要填写的信息。 **返回示例** ```Plain Text { "code": "prep/ok", "message": "成功", "extraMsg": null, "advice": null, "data": [ { "name": "host", "required": 1, "sensitiveInput": 0, "defaultValue": "", "conditions": null, "fieldType": "", "choices": null }, { "name": "port", "required": 1, "sensitiveInput": 0, "defaultValue": "", "conditions": null, "fieldType": "", "choices": null }, { "name": "user", "required": 1, "sensitiveInput": 0, "defaultValue": "", "conditions": null, "fieldType": "", "choices": null }, { "name": "password", "required": 1, "sensitiveInput": 1, "defaultValue": "", "conditions": null, "fieldType": "password", "choices": null } ] } ``` ### 5.6.6 获取数据源类型的信息 **接口说明** 本接口可以获得数据源类型的信息 **请求地址** ```Plain Text GET https://{domain}/aeolus/prep/userOpenAPI/v1/upstream/dataSourceTypeList ``` **请求参数** | | | | | | \ |**参数** |**类型** |**默认值** |**必填** |**说明** | |---|---|---|---|---| | | | | | | \ |dataSourceTypeList |string | |是 |数据源类型 | **请求示例** ```Plain Text curl https://${domain}/aeolus/prep/userOpenAPI/v1/upstream/dataSourceTypeList?dataSourceType=mysql ``` **返回示例** ```Plain Text { "code": "prep/ok", "message": "成功", "extraMsg": null, "advice": null, "data": [ { "alias": "MySQL", "category": [ "关系型数据库" ], "config": { "enableCustomSql": true, "enableLiveMode": true, "enableCustomSqlPreview": true, "showClusterDesc": true, "enableTestConnection": true, "loadToExpandDB": true }, "dataSourceType": "mysql" } ] } ``` ### 5.6.7 获取数据连接列表 **接口说明** 本接口可以获得数据连接列表 **请求地址** ```Plain Text GET https://{domain}/aeolus/prep/userOpenAPI/v1/upstream/list ``` **请求参数** | | | | | | \ |**参数** |**类型** |**默认值** |**必填** |**说明** | |---|---|---|---|---| | | | | | | \ |appId |long | |是 |项目ID | | | | | | | \ |kw |string |"" |否 | | | | | | | | \ |type |string | |否 |数据连接类型 | **请求示例** ```Plain Text curl https://${domain}/aeolus/prep/userOpenAPI/v1/upstream/list?appId=******&kw=&type=mysql ``` **返回示例** ```Plain Text { "code":"prep/ok", "message":"成功", "extraMsg":null, "advice":null, "data":[ { "id":25750, "name":"jk开发机_MySQL57", "appId":{appId}, "host":"10.37.1.208", "port":"3306", "type":"mysql", "user":"root", "additionalParams":{ }, "doradoDatasourceId":*****, "ownerEmailPrefix":"jinke.jk", "encryptMethod":1, "ctime":"2022-11-24 19:52:28", "mtime":"2022-11-24 19:52:28", "roleList":[ { "roleName":"管理", "action":"admin" }, { "roleName":"编辑", "action":"write_role" }, { "roleName":"查看", "action":"read_role" } ], "taskList":[ ] }, { "id":25701, "name":"域内mysql2", "appId":{appId}, "host":"{host}", "port":"*****", "type":"mysql", "user":"root", "additionalParams":{ }, "doradoDatasourceId":*****, "ownerEmailPrefix":"dingdong.kent", "encryptMethod":1, "ctime":"2022-11-21 16:40:29", "mtime":"2022-11-21 16:40:29", "roleList":[ { "roleName":"管理", "action":"admin" }, { "roleName":"编辑", "action":"write_role" }, { "roleName":"查看", "action":"read_role" } ], "taskList":[ ] } ] } ``` ### 5.6.8 获取数据源token **接口说明** 本接口可以获得数据源token **请求地址** ```Plain Text GET https://{domain}/aeolus/prep/userOpenAPI/v1/upstream/upstreamDatasourceToken ``` **请求参数** | | | | | | \ |**参数** |**类型** |**默认值** |**必填** |**说明** | |---|---|---|---|---| | | | | | | \ |ownerEmailPrefix |string | |是 |用户名 | | | | | | | \ |dataSourceType |string | |是 |数据源类型 | **请求示例** ```Plain Text curl https://${domain}/aeolus/prep/userOpenAPI/v1/upstream/upstreamDatasourceToken?ownerEmailPrefix=yangshaojie.ysj&dataSourceType=mysql ``` **返回示例** ```Plain Text { "code": "prep/dataSource/metaUpstreamTokenNotFound", "message": "未找到用户yangshaojie.ysj的${type}数据源的token信息", "extraMsg": null, "advice": null, "data": null } ``` ## 5.6 权限相关 ### 5.6.1 单资源授权 **接口说明** 本接口可以将可视化建模任务，数据集和数据连接授权给其他用户，用户组或角色 **请求地址** ```Plain Text PUT https://{domain}/aeolus/prep/userOpenAPI/v1/resourceAuth ``` **请求参数** | | | | | | \ |**参数** |**类型** |**默认值** |**必填** |**说明** | |---|---|---|---|---| | | | | | | \ |appId |int | |是 |项目ID | | | | | | | \ |resId |int | |是 |资源ID | | | | | | | \ |resType |string | |是 |资源类型 | | | | | | | \ |needNotice |boolean |false |否 | | | | | | | | \ |resList |object[] | |是 |资源列表 | | | | | | | \ |subjectList |object[] | |是 |授权对象列表，type表示授权对象类型 | | | | | | | \ |externalGroupsInfo |object | |否 | | | | | | | | \ |actions |string[] | |是 |授予的权限，包括"read_role"，"write_role"，"admin"，对应查看，编辑，管理权限 | **请求示例** ```Plain Text { "appId":{appId}, "resId":*****, "resType":"prep_task", "needNotice":false, "resList":[ { "resId":*****, "resType":"prep_task" } ], "actions":[ "admin" ], "subjectList":[ { "originId":"liusenmao", "type":0 } ], "externalGroupsInfo":{ "externalGroupAuthType":1, "chatIdList":[] } } ``` **返回示例** ```Plain Text { "code":"prep/ok", "message":"成功", "extraMsg":null, "advice":null, "data":null } ```