网站流量领域：不同受试者前后测Likert量表数据对比咨询

Hey there! Let's walk through how to analyze and compare those pre- and post-redesign Likert scale scores (0-10 for task difficulty) from your independent survey groups. With 1800 pre-survey respondents and 1200 post-survey folks (no overlap, which makes sense given your traffic constraints), here's a structured approach to get meaningful insights:

First, get a clear picture of what each group's data looks like before jumping into significance tests:

• Calculate core metrics: Mean (overall trend), median (better for ordered data like Likert), and mode (most common score). The median is especially useful here because it avoids being skewed by extreme values.
• Break down frequency distributions: Group scores into meaningful buckets (e.g., 0-2 = "Very Difficult", 3-5 = "Moderately Difficult", 6-8 = "Relatively Easy", 9-10 = "Very Easy") and compare the percentage of respondents in each bucket across pre- and post-survey.
• Visualize the data: Use a boxplot to compare the spread, median, and outliers of the two groups, or a stacked bar chart to show the proportion of each score bucket. Visuals make it easy to spot obvious differences at a glance.

Your large sample sizes give you flexibility here—you can choose between two valid approaches depending on how strictly you want to treat the Likert scale:

Option A: Treat Scores as Approximate Continuous Data (Great for Large Samples) #

With sample sizes this big, the Central Limit Theorem kicks in, meaning the sampling distribution of the mean will be roughly normal even if the raw data isn't. This lets you use:

• Independent Samples t-test: Compare the means of the two groups. First check variance equality with a Levene's test; if variances are unequal, use Welch's t-test (the default in most stats tools for unpaired groups).
• Example R code:

  # Assume pre_scores and post_scores are vectors of your 0-10 ratings
  t.test(pre_scores, post_scores, alternative = "two.sided") # Uses Welch's by default
  

Option B: Treat Scores as Ordered Categorical Data (More Rigorous) #

If you want to stick strictly to the ordered nature of Likert scales (since 10 isn't exactly "twice as easy" as 5), use a non-parametric test:

• Mann-Whitney U Test (Wilcoxon Rank-Sum Test): This tests whether the two groups come from the same distribution, focusing on the rank order of scores rather than raw values. It doesn't require normality, which is a plus for ordered data.
• Example R code:

  wilcox.test(pre_scores, post_scores, alternative = "two.sided")
  

With large samples, even tiny differences can be statistically significant—but that doesn't mean they matter for your users. Always quantify the size of the difference:

• For t-tests: Calculate Cohen's d (d = 0.2 = small effect, 0.5 = medium, 0.8 = large).
• For Mann-Whitney U tests: Use Cliff's Delta (ranges from -1 to 1; absolute values closer to 1 mean bigger differences) or r (r = Z / sqrt(total sample size), where Z is the test statistic).

• Don't ignore context: A statistically significant difference of 0.3 points on the 0-10 scale might not translate to a noticeable change in user experience. Tie your findings back to what this means for your website's redesign goals.
• Check for outliers: Look for extreme scores (e.g., 0 or 10 from respondents who didn't actually complete the task) and decide whether to exclude them or analyze them separately.
• Stratify if needed: If you have data on user segments (e.g., new vs. returning users), run the same analyses for each group—differences might be more pronounced in one segment than another.

内容的提问来源于stack exchange，提问作者Clarinetist