如何扩展US ASCII字符集范围且兼容现有代码？文件读取报错排查

Let's break down your problem and walk through practical solutions step by step:

Why the error happens with the header intact #

Your code uses StandardCharsets.US_ASCII to read files, which only supports characters in the 0-127 byte range. Even if the file looks like plain ASCII, the header with square brackets likely contains hidden non-ASCII characters that trigger the exception:

• A UTF-8 BOM (byte sequence EF BB BF): This invisible marker is added by some text editors, and US_ASCII doesn't recognize it as a valid character.
• Full-width square brackets (「」 instead of standard []): These are Unicode characters outside the US_ASCII range, often introduced by copying text from word processors or non-English systems.
• Extended ASCII characters (e.g., accented letters in player names, or "smart quotes"): Even if they look similar to ASCII, they’re encoded outside the 0-127 range.

The Files.readString method uses a strict decoder by default—any invalid byte for the specified charset throws an exception, which your catch block catches and logs.

How to expand charset support without breaking existing ASCII functionality #

You have safe options that maintain compatibility with pure ASCII files while handling non-ASCII content:

1. Switch to UTF-8 (recommended) #

UTF-8 is fully backward-compatible with US_ASCII—all ASCII characters are encoded identically in UTF-8. It also supports every Unicode character, so it can handle any valid PGN file (including those with accented names or non-English comments). Just tweak your charset parameter:

private String getStringFromFile(File file) { 
    try { 
        return Files.readString(Paths.get(file.getPath()), StandardCharsets.UTF_8); 
    } catch (Exception e) { 
        System.out.println("Error while reading: " + file.getName()); 
        return ""; 
    } 
}

2. Use a lenient ASCII decoder #

If you want to stick with ASCII but avoid exceptions for invalid bytes, configure a decoder that replaces non-ASCII characters with a placeholder (like ?) instead of throwing an error. This keeps your existing ASCII logic intact while handling unexpected characters:

private String getStringFromFile(File file) { 
    try {
        // Create a lenient ASCII decoder that replaces invalid bytes
        CharsetDecoder asciiDecoder = StandardCharsets.US_ASCII.newDecoder()
                .onMalformedInput(CodingErrorAction.REPLACE)
                .replaceWith("?");
        return Files.readString(Paths.get(file.getPath()), asciiDecoder);
    } catch (Exception e) { 
        System.out.println("Error while reading: " + file.getName()); 
        return ""; 
    } 
}

3. Fall back to Latin-1 (ISO-8859-1) #

Latin-1 maps every byte (0-255) to a valid character, so it will never throw a charset-related exception. It’s compatible with ASCII, but note that it will interpret non-ASCII bytes as Latin-1 characters (which might not be ideal for full Unicode content). Use it if you only expect extended ASCII, not complex Unicode:

return Files.readString(Paths.get(file.getPath()), StandardCharsets.ISO_8859_1);

Is this error rare? #

Not at all—this is a common pitfall when dealing with text files that appear to be ASCII but have hidden encoding quirks. Typical scenarios include:

• Files saved with UTF-8 BOM by editors like Notepad++ or Microsoft Word
• PGN files downloaded from chess sites that include accented player names or non-English annotations
• Text copied from word processors that auto-replaces standard brackets/quotes with "smart" Unicode versions

内容的提问来源于stack exchange，提问作者tomashauser