Polars LazyFrames自动缓存机制为何未自动优化重复Join操作?
Polars自动缓存未识别重复Join操作的原因
Polars为LazyFrames引入了自动缓存机制,按设计逻辑计划中存在重复LazyFrames时,无需手动执行缓存。但测试时发现该机制未达到预期最优效果:
无显式缓存示例
import polars as pl df1 = pl.DataFrame({'id': [0,5,6]}).lazy() df2 = pl.DataFrame({'id': [0,8,6]}).lazy() df3 = pl.DataFrame({'id': [7,8,6]}).lazy() df4 = df1.join(df2, on='id') print(pl.concat([df4.join(df3, on='id'), df1, df4]).explain())
对应的逻辑计划:
UNION PLAN 0: INNER JOIN: LEFT PLAN ON: [col("id")] INNER JOIN: LEFT PLAN ON: [col("id")] CACHE[id: a4bcf9591fefc837, count: 3] DF ["id"]; PROJECT 1/1 COLUMNS; SELECTION: "None" RIGHT PLAN ON: [col("id")] CACHE[id: 8cee8e3a6f454983, count: 1] DF ["id"]; PROJECT 1/1 COLUMNS; SELECTION: "None" END INNER JOIN RIGHT PLAN ON: [col("id")] DF ["id"]; PROJECT */1 COLUMNS; SELECTION: "None" END INNER JOIN PLAN 1: CACHE[id: a4bcf9591fefc837, count: 3] DF ["id"]; PROJECT 1/1 COLUMNS; SELECTION: "None" PLAN 2: INNER JOIN: LEFT PLAN ON: [col("id")] CACHE[id: a4bcf9591fefc837, count: 3] DF ["id"]; PROJECT 1/1 COLUMNS; SELECTION: "None" RIGHT PLAN ON: [col("id")] CACHE[id: 8cee8e3a6f454983, count: 1] DF ["id"]; PROJECT 1/1 COLUMNS; SELECTION: "None" END INNER JOIN END UNION
有显式缓存示例
import polars as pl df1 = pl.DataFrame({'id': [0,5,6]}).lazy() df2 = pl.DataFrame({'id': [0,8,6]}).lazy() df3 = pl.DataFrame({'id': [7,8,6]}).lazy() df4 = df1.join(df2, on='id').cache() print(pl.concat([df4.join(df3, on='id'), df1, df4]).explain())
对应的逻辑计划:
UNION PLAN 0: INNER JOIN: LEFT PLAN ON: [col("id")] CACHE[id: 290661b0780, count: 18446744073709551615] FAST_PROJECT: [id] INNER JOIN: LEFT PLAN ON: [col("id")] DF ["id"]; PROJECT 1/1 COLUMNS; SELECTION: "None" RIGHT PLAN ON: [col("id")] DF ["id"]; PROJECT 1/1 COLUMNS; SELECTION: "None" END INNER JOIN RIGHT PLAN ON: [col("id")] DF ["id"]; PROJECT */1 COLUMNS; SELECTION: "None" END INNER JOIN PLAN 1: DF ["id"]; PROJECT */1 COLUMNS; SELECTION: "None" PLAN 2: CACHE[id: 290661b0780, count: 18446744073709551615] FAST_PROJECT: [id] INNER JOIN: LEFT PLAN ON: [col("id")] DF ["id"]; PROJECT 1/1 COLUMNS; SELECTION: "None" RIGHT PLAN ON: [col("id")] DF ["id"]; PROJECT 1/1 COLUMNS; SELECTION: "None" END INNER JOIN END UNION
从结果可见,显式缓存时df1与df2的Join仅执行一次,逻辑计划更高效。为什么Polars的自动缓存机制无法检测到Join操作的重复使用并自动缓存?是我遗漏了什么细节吗?
内容的提问来源于stack exchange,提问作者barak1412
相关产品推荐
相关产品推荐

