Kusto是否支持惰性求值?跨集群找多条件objectId优化方案
问题描述
需要找到一个同时满足三个跨集群条件的objectId,三个条件分别对应不同Kusto集群的表,仅需返回一个符合条件的结果即可。
三个筛选条件
条件A
let objectsThatSatCondA = cluster('clusterA').database('dA') .TableA | where componentName == "Orchestrator" and operationName == "Delete" | distinct objectId;
条件B
let objectsThatSatCondB = cluster('clusterB').database('dB') .TableB | where deleteStep in ("DeleteCompleted", "CleanupCompleted") | distinct objectId;
条件C
let objectsThatSatCondC = cluster('clusterC').database('dC') .TableC | where message has "CleanupDeletedObjectFamily" | distinct objectId;
本质是求这三个集合的交集,最初尝试用join实现,但该方法开销极大且超时:
let objectIdComputation = objectsThatSatCondA | join kind=innerunique objectsThatSatCondB on objectId | join kind=innerunique objectsThatSatCondC on objectId | take 1;
优化疑问
在仅需一个objectId的前提下,有哪些策略可以降低查询的内存和计算成本?
惰性求值方案探讨
参考Python中生成器的惰性求值思路:逐个生成满足条件A的objectId,检查是否满足B和C,找到第一个符合条件的就停止,Python代码示例:
def genObjectIdsThatSatCondA(): for objectId in objectIds: if satisifesCondA(objectId): yield objectId for objectId in genObjectIdsThatSatCondA(): if satisfiesCondB(objectId) and satisfiesCondC(objectId): print(objectId) break
想明确这种逻辑能否在Kusto中实现?是否可以惰性计算objectsThatSatCondA,将objectId逐个传入后续检查逻辑,避免全量计算条件A的结果?
Kusto函数方案设想
考虑编写Kusto函数satisfiesCondB(objectId)和satisfiesCondC(objectId),然后执行如下查询:
let objectsThatSatCondA = cluster('clusterA').database('dA') .TableA | where componentName == "Orchestrator" and operationName == "Delete" and satisfiesCondB(objectId) == True and satisfiesCondC(objectId) == True | take 1;
但不确定如何定义这类函数,以及跨集群调用是否存在限制。
另外,即便该方案可行,也想明确Kusto是否支持惰性求值,以备后续参考。
小左表方案
先对条件A的结果取前1000条:
let objectsThatSatCondA = objectsThatSatCondA | take 1000;
再执行innerunique join可以避免超时,但可能需要多次尝试才能找到符合条件的objectId。
内容的提问来源于stack exchange,提问作者joseville
相关产品推荐
相关产品推荐

