如何去除JSON列表中的重复值？解决Set去重无效问题

问题背景 #

需要去除赛事JSON列表中的重复球队名称，当前数据里HomeTeam存在重复的Arsenal，AwayTeam存在重复的Liverpool，导致前端下拉列表出现重复选项。尝试在Spring Boot的CCController中将List<Team>转为HashSet去重，但仅改变了列表顺序，未去除重复球队。后续通过引入Jakarta JSON依赖，解析JSON数组结合Set实现了去重，现咨询更合适的去重方案。

JSON示例 #

[
    {
        "MatchNumber": 1,
        "RoundNumber": 1,
        "HomeTeam": "Crystal Palace",
        "AwayTeam": "Liverpool",
        "Group": null,
        "HomeTeamScore": 0,
        "AwayTeamScore": 2
    },
    {
        "MatchNumber": 16,
        "RoundNumber": 2,
        "HomeTeam": "Fulham",
        "AwayTeam": "Liverpool",
        "Group": null,
        "HomeTeamScore": 2,
        "AwayTeamScore": 2
    },
    {
        "MatchNumber": 11,
        "RoundNumber": 2,
        "HomeTeam": "Arsenal",
        "AwayTeam": "Leicester",
        "Group": null,
        "HomeTeamScore": 4,
        "AwayTeamScore": 2
    },
    {
        "MatchNumber": 191,
        "RoundNumber": 20,
        "HomeTeam": "Arsenal",
        "AwayTeam": "Leeds",
        "Group": null,
        "HomeTeamScore": null,
        "AwayTeamScore": null
    }
]

Controller代码 #

@Controller
public class CCController {
private static final String MAIN_PAGE = "main";
private static final String JSON_URL = "https://randomfootball.com/json/epl-2022";
  @GetMapping("/")
    public String main(Model model, @ModelAttribute Team team){
       List<Team> teams = (List<Team>) parsingService.parse(JSON_URL);
       Set<Team> targetSet = new HashSet<>(teams);
       model.addAttribute("teamList", targetSet);    
        return MAIN_PAGE;
    }
}

Team实体类代码 #

public class Team{
    
    public Team() {
    }
    
    Set<String> HomeTeam;
    
    public Set<String> getHomeTeam() {
        return HomeTeam;
    }

    public void setHomeTeam(Set<String> homeTeam) {
        HomeTeam = homeTeam;
    }

    public static Set<String> keySet() {
        return null;
    }
}

正确去重方案 #

1. 先修正Team实体类 #

原Team类字段定义错误：JSON中的HomeTeam和AwayTeam是字符串类型，而非Set<String>；同时要重写equals()和hashCode()方法（HashSet依赖这两个方法判断对象是否重复）。建议将类名改为Match更贴合语义：

import java.util.Objects;

public class Match {
    private Integer MatchNumber;
    private Integer RoundNumber;
    private String HomeTeam;
    private String AwayTeam;
    private String Group;
    private Integer HomeTeamScore;
    private Integer AwayTeamScore;

    // 无参构造、全参构造方法
    public Match() {}

    public Match(Integer matchNumber, Integer roundNumber, String homeTeam, String awayTeam, String group, Integer homeTeamScore, Integer awayTeamScore) {
        MatchNumber = matchNumber;
        RoundNumber = roundNumber;
        HomeTeam = homeTeam;
        AwayTeam = awayTeam;
        Group = group;
        HomeTeamScore = homeTeamScore;
        AwayTeamScore = awayTeamScore;
    }

    // getter/setter方法
    public Integer getMatchNumber() { return MatchNumber; }
    public void setMatchNumber(Integer matchNumber) { MatchNumber = matchNumber; }
    public Integer getRoundNumber() { return RoundNumber; }
    public void setRoundNumber(Integer roundNumber) { RoundNumber = roundNumber; }
    public String getHomeTeam() { return HomeTeam; }
    public void setHomeTeam(String homeTeam) { HomeTeam = homeTeam; }
    public String getAwayTeam() { return AwayTeam; }
    public void setAwayTeam(String awayTeam) { AwayTeam = awayTeam; }
    public String getGroup() { return Group; }
    public void setGroup(String group) { Group = group; }
    public Integer getHomeTeamScore() { return HomeTeamScore; }
    public void setHomeTeamScore(Integer homeTeamScore) { HomeTeamScore = homeTeamScore; }
    public Integer getAwayTeamScore() { return AwayTeamScore; }
    public void setAwayTeamScore(Integer awayTeamScore) { AwayTeamScore = awayTeamScore; }

    // 按球队名称判断对象重复的equals和hashCode
    @Override
    public boolean equals(Object o) {
        if (this == o) return true;
        if (o == null || getClass() != o.getClass()) return false;
        Match match = (Match) o;
        return Objects.equals(HomeTeam, match.HomeTeam) && Objects.equals(AwayTeam, match.AwayTeam);
    }

    @Override
    public int hashCode() {
        return Objects.hash(HomeTeam, AwayTeam);
    }
}

2. 提取所有不重复的球队名称（推荐） #

如果仅需下拉列表的不重复球队名称，无需将整个Match对象存入Set，直接提取HomeTeam和AwayTeam到Set即可：

@GetMapping("/")
public String main(Model model) {
    List<Match> matches = (List<Match>) parsingService.parse(JSON_URL);
    // 用循环提取
    Set<String> allTeams = new HashSet<>();
    for (Match match : matches) {
        allTeams.add(match.getHomeTeam());
        allTeams.add(match.getAwayTeam());
    }

    // 或者用Stream API更简洁
    // Set<String> allTeams = matches.stream()
    //         .flatMap(match -> Stream.of(match.getHomeTeam(), match.getAwayTeam()))
    //         .collect(Collectors.toSet());

    model.addAttribute("teamList", allTeams);
    return MAIN_PAGE;
}

3. 不修改实体类的直接解析去重 #

如果不想修正实体类，可使用Spring Boot默认依赖的Jackson直接解析JSON数组提取球队名称，无需额外引入Jakarta JSON：

import com.fasterxml.jackson.databind.JsonNode;
import com.fasterxml.jackson.databind.ObjectMapper;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Controller;
import org.springframework.ui.Model;
import org.springframework.web.bind.annotation.GetMapping;

import java.io.IOException;
import java.net.URL;
import java.util.HashSet;
import java.util.Set;

@Controller
public class CCController {
private static final String MAIN_PAGE = "main";
private static final String JSON_URL = "https://randomfootball.com/json/epl-2022";

    @Autowired
    private ObjectMapper objectMapper;

    @GetMapping("/")
    public String main(Model model) throws IOException {
        JsonNode rootNode = objectMapper.readTree(new URL(JSON_URL));
        Set<String> allTeams = new HashSet<>();
        if (rootNode.isArray()) {
            for (JsonNode node : rootNode) {
                allTeams.add(node.get("HomeTeam").asText());
                allTeams.add(node.get("AwayTeam").asText());
            }
        }
        model.addAttribute("teamList", allTeams);
        return MAIN_PAGE;
    }
}

原方案无效原因分析 #

之前用HashSet<Team>去重失败的核心问题：

• Team实体类的HomeTeam字段定义为Set<String>，与JSON中的字符串类型不匹配，导致解析后字段值异常
• 未重写equals()和hashCode()方法，HashSet无法判断两个Team对象是否代表同一球队

内容的提问来源于stack exchange，提问作者hellothere