如何在Azure Data Factory中将Lookup活动值传入Data Flow?
使用Data Flow将Kusto数据导出至ADLS:Lookup值的传递与问题排查
问题背景
已有可正常运行的Azure Data Factory(ADF)管道,计划替换Copy活动为Data Flow活动,将Kusto数据导出至ADLS并对比性能,核心需求是将Lookup活动的输出值用于Data Flow中的Kusto查询映射。
Lookup活动的Kusto查询及输出
查询语句
let pickDate = datetime(2023-03-01); let Conversion = datatable(Decimal: int, HexCode: string, Binary: string) [ 0, '0', '0000', 1, '1', '0001', 2, '2', '0010', 3, '3', '0011', 4, '4', '0100', 5, '5', '0101', 6, '6', '0110', 7, '7', '0111', 8, '8', '1000', 9, '9', '1001', 10, 'a', '1010', 11, 'b', '1011', 12, 'c', '1100', 13, 'd', '1101', 14, 'e', '1110', 15, 'f', '1111', ]; let data = materialize(range Dates from pickDate to pickDate step 1d | extend Dates = format_datetime(Dates, 'yyyy-MM-dd') | extend JoinColumn = 1 | join kind=inner ( Conversion | extend JoinColumn = 1 ) on JoinColumn | project Dates, HexCode); let HexCodes = data | summarize make_set(HexCode); data | extend SubHexCode = toscalar(HexCodes) | mv-expand SubHexCode
查询输出
| Dates | HexCode | SubHexCode |
|---|---|---|
| 2023-03-01 | 0 | 0 |
| 2023-03-01 | 0 | 1 |
| 2023-03-01 | 0 | 2 |
目标需求:Data Flow中的Kusto查询
需要将Lookup输出的Dates、HexCode、SubHexCode参数化到Data Flow的Kusto查询中,目标查询如下:
set notruncation; let HexCode = "@{item().HexCode}@{item().SubHexCode}"; Test | where tolower(ContainerId) startswith HexCode | join kind=inner ( Test2 | extend ContainerId = VmId | where startofday(TIMESTAMP) == startofday(datetime(@{item().Dates})) | where tolower(ContainerId) startswith HexCode ) on ContainerId | project x, y, z
更新1:参数配置后的问题
已为管道设置参数并启用每个参数的「表达式」选项,但仍遇到查询错误。当前在Kusto查询中使用字符串插值拼接值:
let HexCode = "$HexCode$SubHexCode"; | where startofday(TIMESTAMP) == startofday(datetime($Dates))
同时不清楚如何查看最终生成的查询输出。
更新2:Kusto示例数据
| VmId | ContainerId | SampleCount | AverageCounterValue | CounterName | MinCounterValue | MaxCounterValue | PreciseTimeStamp |
|---|---|---|---|---|---|---|---|
| 11c1b7c1-d1b1-11fb-11a1-0c711fe111c0 | 11c1b7c1-d1b1-11fb-11a1-0c711fe111c0 | 20 | 13.57497519 | Percentage CPU | 3.420510085 | 26.68882703 | 2023-03-08T05:04:59.9999999Z |
| 11c1b7c1-d1b1-11fb-11a1-0c711fe111c0 | 11c1b7c1-d1b1-11fb-11a1-0c711fe111c0 | 20 | 10.97057026 | Percentage CPU | 6.853699734 | 20.73203381 | 2023-03-08T05:04:59.9999999Z |
| 11c1b7c1-d1b1-11fb-11a1-0c711fe111c0 | 11c1b7c1-d1b1-11fb-11a1-0c711fe111c0 | 40 | 4.061182971 | Percentage CPU | 3.075506647 | 5.650666854 | 2023-03-08T09:54:59.9999999Z |
内容的提问来源于stack exchange,提问作者Guilherme Matheus
相关产品推荐
相关产品推荐

