如何基于多条件利用Spark DataFrame前一行值填充新列？
2026-8-14
生成Spark DataFrame新列Column H的问题
免责声明：此问题可能为重复问题，但我未找到完全匹配的解决方案。若确认重复，请标注该问题并在评论中提供原问题链接。
我仍在学习Python DataFrame操作，此问题或许存在简单解法，但我暂未理清。
我拥有一个约1200万行的Spark DataFrame，包含多列。现需基于另外两列的最大值及新列前一行的派生值生成新列Column H。
输入数据
| Column A | Column B | Column C | Column D | Column E | Column F | Column G |
|---|---|---|---|---|---|---|
| 1 | Apr | 2022 | 3 | 2 | 3 | 0 |
| 1 | May | 2022 | 3 | 2 | 3 | 0 |
| 1 | Jun | 2022 | 3 | 1 | 1 | 0 |
| 1 | Jul | 2022 | 3 | 2 | 4 | 0 |
| 1 | Aug | 2022 | 3 | 2 | -1 | 0 |
| 1 | Sep | 2022 | 3 | 8 | 3 | 0 |
| 1 | Oct | 2022 | 3 | 2 | 3 | 0 |
| 1 | Nov | 2022 | 3 | 2 | 0 | 0 |
| 1 | Dec | 2022 | 3 | 2 | 0 | 0 |
| 1 | Jan | 2023 | 3 | 2 | 6 | 0 |
| 1 | Feb | 2023 | 3 | 2 | 5 | 0 |
| 1 | Mar | 2023 | 3 | 2 | 6 | 0 |
| 1 | Apr | 2023 | 3 | 2 | 7 | 0 |
| 1 | May | 2023 | 0 | 0 | 0 | 0 |
| 2 | Apr | 2017 | 3 | 2 | 3 | 0 |
| 2 | Apr | 2017 | 3 | 2 | 3 | 0 |
| 2 | Apr | 2017 | 3 | 1 | 1 | 0 |
| 2 | Apr | 2017 | 3 | 2 | 4 | 0 |
| 2 | Apr | 2017 | 3 | 2 | -1 | 0 |
| 2 | Apr | 2017 | 3 | 7 | 3 | 0 |
输出数据
| Column A | Column B | Column C | Column D | Column E | Column F | Column G | Column H | Logic |
|---|---|---|---|---|---|---|---|---|
| 1 | Apr | 2022 | 3 | 2 | 3 | 0 | 3 | Max of Column E and F |
| 1 | May | 2022 | 3 | 2 | 3 | 0 | 3 | Max of Column E and F and output of Column H previous row |
| 1 | Jun | 2022 | 3 | 1 | 1 | 0 | 3 | Max of Column E and F and output of Column H previous row |
| 1 | Jul | 2022 | 3 | 2 | 4 | 0 | 4 | Max of Column E and F and output of Column H previous row |
| 1 | Aug | 2022 | 3 | 2 | -1 | 0 | 4 | Max of Column E and F and output of Column H previous row |
| 1 | Sep | 2022 | 3 | 8 | 3 | 0 | 8 | Max of Column E and F and output of Column H previous row |
| 1 | Oct | 2022 | 0 | 2 | 3 | 0 | 8 | Max of Column E and F and output of Column H previous row |
| 1 | Nov | 2022 | 0 | 2 | 0 | 0 | 8 | Max of Column E and F and output of Column H previous row |
| 1 | Dec | 2022 | 0 | 0 | 0 | 0 | 8 | Max of Column E and F and output of Column H previous row |
| 1 | Jan | 2023 | 0 | 0 | 6 | -3 | 6 | If Column G < 0 Max of Column E and F and (output of Column H previous row + Column G) |
| 1 | Feb | 2023 | 0 | 2 | 5 | 0 | 6 | Max of Column E and F and output of Column H previous row |
| 1 | Mar | 2023 | 0 | 2 | 6 | 0 | 6 | Max of Column E and F and output of Column H previous row |
| 1 | Apr | 2023 | 0 | 0 | 7 | 0 | 7 | Max of Column E and F and output of Column H previous row |
| 1 | May | 2023 | 0 | 0 | 0 | 0 | 7 | Max of Column E and F and output of Column H previous row |
| 2 | Apr | 2017 | 3 | 2 | 3 | 0 | 3 | Max of Column E and F |
| 2 | Apr | 2017 | 3 | 2 | 3 | 0 | 3 | Max of Column E and F and output of Column H previous row |
| 2 | Apr | 2017 | 3 | 1 | 1 | 0 | 3 | Max of Column E and F and output of Column H previous row |
| 2 | Apr | 2017 | 3 | 2 | 4 | 0 | 4 | Max of Column E and F and output of Column H previous row |
| 2 | Apr | 2017 | 3 | 2 | -1 | 0 | 4 | Max of Column E and F and output of Column H previous row |
| 2 | Apr | 2017 | 3 | 7 | 3 | 0 | 7 | Max of Column E and F and output of Column H previous row |
生成Column H的规则
- DataFrame中Column A的不同值按年月升序排列；
- 每个Column A不同值的首行，Column H取Column E与Column F的最大值；
- 当Column G >=0时，后续行的Column H取Column E、Column F的最大值与前一行Column H值的最大值；
- 当Column G <0时，Column H取Column E、Column F的最大值与（前一行Column H值 + Column G值）的最大值。
内容的提问来源于stack exchange，提问作者Raja
相关产品推荐
相关产品推荐
