如何遍历Spark Row数据?遍历行为与文档描述不符的问题
PySpark Row 对象遍历的疑惑
我在查看Spark 3.1.3官方文档中关于pyspark.sql.Row的说明时,看到文档指出key in row会搜索Row的键,但实际测试时,使用for thing in row遍历Row对象,得到的却是值而非键,和预期不符。
以下是我的测试代码:
data = [{"ID": ' ', "PostCode": ' A ', "Value": 121.44, "Truth": True}, {"ID": None, "PostCode": '', "Value": 300.01, "Truth": False}, {"ID": None, "PostCode": ' C', "Value": 10.99, "Truth": None}, {"ID": '', "PostCode": 'E ', "Value": 33.87, "Truth": True} ] df = spark.createDataFrame(data) df.show() cols = [f"any({col} is not null AND trim({col}) != '') as {col}_any_isnot_null_or_empty" for col in df.columns] rows = df.selectExpr(cols) rows.show() row = rows.collect()[0] print(type(row)) asdict = row.asDict() for key, val in asdict.items(): print(key, val) print() print() for thing in row: print(thing, row[thing])
输出结果:
<class 'pyspark.sql.types.Row'> ID_any_isnot_null_or_empty False PostCode_any_isnot_null_or_empty True Truth_any_isnot_null_or_empty True Value_any_isnot_null_or_empty True False False True True True True True True
通过asDict()方法遍历可以正常得到键值对,符合预期;但直接遍历Row对象时只能拿到值,这和文档描述的逻辑似乎矛盾,请问我哪里理解错了?
内容的提问来源于stack exchange,提问作者kyl
相关产品推荐
相关产品推荐

