如何度量与解读离散概率分布的多样性指数？含实例咨询

Great question! Let's walk through how to measure and make sense of diversity indices for discrete probability distributions—including handling multi-select cases and fixing that "hard to interpret" issue with Shannon entropy.

First, let's cover the most common metrics, how to calculate them, and how to turn them into intuitive percentages.

Shannon Entropy (and Normalization for Percentages) #

Shannon entropy is the go-to for measuring uncertainty/diversity in a distribution. The formula is:

H = -Σ(p_i * log2(p_i))

Where p_i is the probability of the i-th category.

The problem with raw entropy is it's tied to the number of categories—so a 2-category distribution maxes out at log2(2) = 1, while a 5-category distribution maxes at log2(5) ≈ 2.32. To convert this to a percentage (0-100% representing "how close we are to maximum possible diversity"), normalize it by the maximum entropy for your number of categories:

Normalized Shannon % = (H / log2(k)) * 100

Where k is the total number of distinct categories.

Example (your yes/no case): #

For distribution {0.9: yes, 0.1: no}:

• Raw H ≈ -(0.9log2(0.9) + 0.1log2(0.1)) ≈ 0.469
• Max entropy (k=2) is 1
• Normalized % ≈ (0.469 / 1) * 100 = 46.9%
• Interpretation: This response set has ~47% of the diversity we'd see if answers were perfectly split (50/50).

Simpson & Gini-Simpson Indices #

If you care more about dominant categories (rather than rare ones), Simpson's index is useful. It measures the probability that two randomly selected items belong to the same category:

D = Σ(p_i²)

The inverse—Gini-Simpson index—measures diversity directly (probability two items are from different categories):

Gini-Simpson = 1 - D

To convert this to a percentage relative to maximum diversity, normalize by the maximum possible Gini-Simpson value (1 - 1/k):

Normalized Gini-Simpson % = ((1 - D) / (1 - 1/k)) * 100

Example (your yes/no case): #

• D = 0.9² + 0.1² = 0.82
• Gini-Simpson = 1 - 0.82 = 0.18
• Max Gini-Simpson (k=2) is 0.5
• Normalized % ≈ (0.18 / 0.5) * 100 = 36%
• Interpretation: There's an 18% chance two random responses are different, or this set has 36% of the maximum possible diversity for two categories.

Effective Number of Categories #

Another intuitive way to frame diversity is the effective number of categories:

• For Shannon entropy: 2^H (or e^H if using natural log)
• For Simpson index: 1/D

Example (your yes/no case): #

• Shannon effective categories ≈ 2^0.469 ≈ 1.38
• Interpretation: This distribution is as diverse as if we had ~1.4 equally common categories (super close to just one dominant category, which makes sense here).

For multi-select data (where each respondent can pick multiple options), you first need to reframe the data into a category frequency distribution:

1. Count the total number of selections across all respondents (e.g., if 3 people pick both A and B, that's 6 total selections)
2. Calculate the relative frequency of each category: p_i = (number of times category i was selected) / total selections
3. Apply any of the indices above (Shannon, Gini-Simpson, etc.) to this new frequency distribution.

Example: #

Suppose 10 students:

• 3 pick both yes and maybe
• 5 pick only yes
• 2 pick only no
  Total selections = (3*2) + 5 + 2 = 6 + 5 + 2 = 13
• p_yes = (3 + 5)/13 ≈ 0.615
• p_maybe = 3/13 ≈ 0.231
• p_no = 2/13 ≈ 0.154
  Now calculate Shannon entropy or Gini-Simpson on these p_i values just like before.

• Use Shannon entropy if you want to emphasize the impact of rare categories (it weights low-probability options more heavily)
• Use Gini-Simpson if you care more about how dominant the top categories are
• Use normalized percentages or effective categories when you need to communicate results to non-technical stakeholders

内容的提问来源于stack exchange，提问作者Sh galal