Apache Pig求和计算报错：无法找到price字段，求排查解决

执行的代码历史 #

grunt> history

1   stockprice = load 'PG/TutorialA/input/stockprice.csv' using PigStorage(',') AS(Stock:chararray,price:int);

2   investor = load 'PG/TutorialA/input/investor.csv' using PigStorage(',') AS(id:int,first:chararray,last:chararray,stock:chararray,price:int);

3   investor_stockprice = join investor by stock, stockprice by Stock;

4   group_by_lastname = group investor_stockprice by last;

触发报错的代码 #

grunt> sum_of_shares = FOREACH group_by_lastname GENERATE investor_stockprice, SUM(investor_stockprice.price) as Sum;

错误日志 #

1275634 [main] ERROR org.apache.pig.tools.grunt.Grunt  - ERROR 1128: Cannot find field price in investor::id:int,investor::first:chararray,investor::last:chararray,investor::stock:chararray,investor::price:int,stockprice::Stock:chararray,stockprice::price:int
22/10/16 04:23:01 ERROR grunt.Grunt: ERROR 1128: Cannot find field price in investor::id:int,investor::first:chararray,investor::last:chararray,investor::stock:chararray,investor::price:int,stockprice::Stock:chararray,stockprice::price:int
Details at logfile: /mnt/var/log/pig/pig_1665892905780.log

关联的数据表 #

问题原因 #

报错核心是字段名冲突：investor和stockprice两个表都定义了price字段，join之后Pig无法直接识别你要引用的price属于哪张表，导致字段查找失败。从错误日志的字段列表也能看到，两个表的price字段都存在：investor::price:int、stockprice::price:int。

另外从数据表内容看，你要计算的总持股数对应的是investor表中的持股数量（也就是你定义的investor::price），需要明确指定字段所属的表。

修复方案 #

方案1：直接指定字段所属表 #

修改求和的代码，明确写出表别名加字段名：

sum_of_shares = FOREACH group_by_lastname GENERATE group as last_name, SUM(investor_stockprice.investor::price) as total_shares;

方案2：提前重命名冲突字段（更规范） #

在加载数据时就给冲突字段改个语义更清晰的名字，避免后续混淆：

stockprice = load 'PG/TutorialA/input/stockprice.csv' using PigStorage(',') AS(Stock:chararray,stock_price:int);
investor = load 'PG/TutorialA/input/investor.csv' using PigStorage(',') AS(id:int,first:chararray,last:chararray,stock:chararray,shares:int);
investor_stockprice = join investor by stock, stockprice by Stock;
group_by_lastname = group investor_stockprice by last;
sum_of_shares = FOREACH group_by_lastname GENERATE group as last_name, SUM(investor_stockprice.shares) as total_shares;

两种方案都能解决字段识别的问题，顺利计算出总持股数。

内容的提问来源于stack exchange，提问作者Wing