如何在Pig脚本中按分组计算错误总数及占比？

Got it! Let's tackle adding that error percentage column to your Pig script. Here's a step-by-step approach that builds on your existing grouping logic:

1. Start with your existing grouping logic #

First, let's recap your current script (adjust the load statement to match your actual input schema):

-- Load your raw error data
raw_errors = LOAD 'your_input_path' USING PigStorage(',') AS (error_code:chararray, name:chararray);

-- Group by Error Code and Name, count occurrences
grouped_errors = GROUP raw_errors BY (error_code, name);
error_counts = FOREACH grouped_errors GENERATE
    FLATTEN(group) AS (error_code, name),
    COUNT(raw_errors) AS error_count;

2. Calculate the total number of errors #

To compute the percentage, we first need the total number of errors across all groups:

-- Get total error count
total_errors = FOREACH (GROUP raw_errors ALL) GENERATE
    COUNT(raw_errors) AS total_count;

3. Join grouped counts with total count #

We'll do a cross join (using a dummy key) to attach the total count to every grouped record:

-- Join grouped counts with total count (dummy key '1' ensures all records pair with total)
with_total = JOIN error_counts BY 1, total_errors BY 1;

4. Compute and format the percentage #

Now calculate the percentage, convert it to a string, and append the '%' symbol. We'll cast to double to avoid integer division issues, then round to get a whole number percentage (adjust if you need decimal places):

-- Calculate percentage and format the result
final_output = FOREACH with_total GENERATE
    error_code,
    name,
    error_count,
    CONCAT(STRING(ROUND((error_count / (double)total_count) * 100)), '%') AS error_percentage;

5. Output the result #

Dump or store the final result:

DUMP final_output;
-- Or store to a file
-- STORE final_output INTO 'your_output_path' USING PigStorage(',');

Example Output #

This will give you exactly the format you want:

INVALID_PARAM,REQUEST_ERROR,10,20%
MISSING_FIELD,VALIDATION_FAILED,5,10%
...

Notes #

• If you need decimal precision (e.g., 20.5%), replace ROUND with FORMAT like this:

  FORMAT("%.1f%%", (error_count / (double)total_count) * 100) AS error_percentage
  

• Make sure your input data doesn't have null values for error_code/name, or add a FILTER step to exclude them first if needed.

内容的提问来源于stack exchange，提问作者skpraveen