Glue 5.0下DynamicFrame关联结果为空Schema且无数据问题

在AWS Glue 5.0环境中执行DynamicFrame关联操作时，出现无数据输出且Schema为空的问题，操作流程如下：

• 对数据执行展平处理
• 尝试将root_component.supporting_info与root进行关联

目前计划使用原生Spark DataFrame复现该问题，以下是示例代码及输出结果：

示例代码 #

unnested = dyf.relationalize("root", "s3://mybucket/temp/")
print("#####KEYS####")
print(unnested.keys())
print("#####KEYS####")
print("")
dyf_root = unnested.select("root")
print("#####ROOT DATA####")
dyf_root.toDF().show(vertical=True)
print("#####ROOT DATA####")
print("")
dyf_root_supporting_info = unnested.select("root_component.supporting_info")
print("#####ROOT SUPPORTING INFO####")
dyf_root_supporting_info.toDF().show(vertical=True)
print("#####ROOT SUPPORTING INFO####")
print("")
dyf_joined = dyf_root.join(paths1=["component.supporting_info"], paths2=["id"], frame2=dyf_root_supporting_info)
print("#####JOINED SCHEMA#####")
dyf_joined.printSchema()
print("#####JOINED SCHEMA#####")
print("")
print("#####JOINED DATA#####")
dyf_joined.toDF().show(vertical=True)
print("#####JOINED DATA#####")

输出结果 #

#####KEYS####
dict_keys(['root', 'root_component.tuple.licenses', 'root_component.supporting_info', 'root_image_tags', 'root_component.raw_tuple.licenses', 'root_component.original_tuple.licenses', 'root_component.supporting_info.val.cpes'])
#####KEYS####

#####ROOT DATA####
-RECORD 0-------------------------------------------------------------
 image_id                                      | 1127318              
 product_id                                    | 600                  
 scan_id                                       | 1727750              
 action_id                                     | 665a693e3d03044f1... 
 action_timestamp                              | 2024-06-01T00:20:... 
 image_tags                                    | 1                    
 component.found_by                            | protecode            
 component.image_id                            | 1127318              
 component.raw_tuple.component_type            | Open Source          
 component.raw_tuple.latest_known_version      | 2.7.3                
 component.raw_tuple.maven_id                  | xalan:xalan          
 component.raw_tuple.licenses                  | 1                    
 component.raw_tuple.name                      | xalan                
 component.raw_tuple.version                   | 2.7.2                
 component.raw_tuple.supplier                  | apache               
 component.raw_tuple.codetype                  | java                 
 component.raw_tuple.tuple_id                  | b9fff82793e1ddb9f... 
 component.raw_tuple.homepage                  | null                 
 component.reusable_module_id                  | 1127318              
 component.tuple.component_type                | Open Source          
 component.tuple.latest_known_version          | 2.7.3                
 component.tuple.maven_id                      | xalan:xalan          
 component.tuple.licenses                      | 1                    
 component.tuple.name                          | xalan                
 component.tuple.version                       | 2.7.2                
 component.tuple.supplier                      | apache               
 component.tuple.codetype                      | java                 
 component.tuple.tuple_id                      | b9fff82793e1ddb9f... 
 component.tuple.homepage                      | null                 
 component.origin_id                           | 665a693e3d03044f1... 
 component.original_tuple.component_type       | Open Source          
 component.original_tuple.latest_known_version | 2.7.3                
 component.original_tuple.maven_id             | xalan:xalan          
 component.original_tuple.licenses             | 1                    
 component.original_tuple.name                 | xalan                
 component.original_tuple.version              | 2.7.2                
 component.original_tuple.supplier             | apache               
 component.original_tuple.codetype             | java                 
 component.original_tuple.tuple_id             | b9fff82793e1ddb9f... 
 component.original_tuple.homepage             | null                 
 component.supporting_info                     | 1                    
 component.id                                  | 665a693e3d03044f1... 
 gc_image                                      | true                 
-RECORD 1-------------------------------------------------------------
 image_id                                      | 1127318              
 product_id                                    | 600                  
 scan_id                                       | 1727750              
 action_id                                     | 665a693e3d03044f1... 
 action_timestamp                              | 2024-06-01T00:20:... 
 image_tags                                    | 2                    
 component.found_by                            | protecode            
 component.image_id                            | 1127318              
 component.raw_tuple.component_type            | null                 
 component.raw_tuple.latest_known_version      | 6.9.0                
 component.raw_tuple.maven_id                  | org.apache.bcel:bcel 
 component.raw_tuple.licenses                  | 2                    
 component.raw_tuple.name                      | bcel                 
 component.raw_tuple.version                   | 2.7.2                
 component.raw_tuple.supplier                  | apache               
 component.raw_tuple.codetype                  | java                 
 component.raw_tuple.tuple_id                  | c4c44c1c3ad99644d... 
 component.raw_tuple.homepage                  | https://commons.a... 
 component.reusable_module_id                  | 1127318              
 component.tuple.component_type                | Open Source          
 component.tuple.latest_known_version          | 6.9.0                
 component.tuple.maven_id                      | org.apache.bcel:bcel 
 component.tuple.licenses                      | 2                    
 component.tuple.name                          | bcel                 
 component.tuple.version                       | 2.7.2                
 component.tuple.supplier                      | apache               
 component.tuple.codetype                      | java                 
 component.tuple.tuple_id                      | c4c44c1c3ad99644d... 
 component.tuple.homepage                      | https://commons.a... 
 component.origin_id                           | 665a693e3d03044f1... 
 component.original_tuple.component_type       | Open Source          
 component.original_tuple.latest_known_version | 6.9.0                
 component.original_tuple.maven_id             | org.apache.bcel:bcel 
 component.original_tuple.licenses             | 2                    
 component.original_tuple.name                 | bcel                 
 component.original_tuple.version              | 2.7.2                
 component.original_tuple.supplier             | apache               
 component.original_tuple.codetype             | java                 
 component.original_tuple.tuple_id             | c4c44c1c3ad99644d... 
 component.original_tuple.homepage             | https://commons.a... 
 component.supporting_info                     | 2                    
 component.id                                  | 665a693e3d03044f1... 
 gc_image                                      | true                 

#####ROOT DATA####

#####ROOT SUPPORTING INFO####
-RECORD 0-------------------------------------------------------------
 id                                            | 1                    
 index                                         | 0                    
 component.supporting_info.val.sha1            | 9ee6066b9f7152234... 
 component.supporting_info.val.path            | alu-sr-cli-8.49/p... 
 component.supporting_info.val.package_name    | xalan:xalan          
 component.supporting_info.val.confidence      | 0.67                 
 component.supporting_info.val.matching_method | signature            
 component.supporting_info.val.cpes            | 1                    
-RECORD 1-------------------------------------------------------------
 id                                            | 2                    
 index                                         | 0                    
 component.supporting_info.val.sha1            | 9ee6066b9f7152234... 
 component.supporting_info.val.path            | alu-sr-cli-8.49/p... 
 component.supporting_info.val.package_name    | org.apache.bcel:bcel 
 component.supporting_info.val.confidence      | 0.77                 
 component.supporting_info.val.matching_method | signature            
 component.supporting_info.val.cpes            | 2                    

#####ROOT SUPPORTING INFO####

#####JOINED SCHEMA#####
root

#####JOINED SCHEMA#####

#####JOINED DATA#####
(0 rows)

#####JOINED DATA#####

内容的提问来源于stack exchange，提问作者user2403018