如何读取大文件并避免java.lang.OutOfMemoryError异常？

我编写了一个Java程序，功能是读取文件内容、查找唯一字符串行并将其拆分为不相交的组。但处理大小≥1GB的大文件时，程序抛出堆内存溢出异常：

Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
    at java.base/java.util.Arrays.copyOf(Arrays.java:3537)
    at java.base/java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:228)
    at java.base/java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:582)
    at java.base/java.lang.StringBuilder.append(StringBuilder.java:179)
    at org.example.Main.main(Main.java:149).

我知道可以通过调整堆内存参数解决，但希望通过代码优化来处理这个问题，以下是我的代码：

public class HugeFileReader {

    public static void main(String[] args) throws IOException {
        String OutputFile = "File created";
        StringBuilder stringBuilder = new StringBuilder();
        LineIterator bufferedReader = FileUtils.lineIterator(new File(args[0]),"UTF-8");
        BufferedWriter bufferedWriter = new BufferedWriter(new FileWriter(OutputFile));
        List<Set<String>> numberOfGroups = new ArrayList<>();
        List<Map<String, Integer>> positionOfNumbers = new ArrayList<>();
        String line = bufferedReader.nextLine();

        while (bufferedReader.hasNext()) {
            String[] columns = getColumns(line);
            Integer numOfGroup = null;

            for (int i = 0; i < Math.min(positionOfNumbers.size(), columns.length); i++) {
                Integer numOfGroup2 = positionOfNumbers.get(i).get(columns[i]);

                if (numOfGroup2 != null) {
                    if (numOfGroup == null) {
                        numOfGroup = numOfGroup2;
                    } else if (!numOfGroup.equals(numOfGroup2)) {
                        for (String numbersOfGroup : numberOfGroups.get(numOfGroup2)) {

                            numberOfGroups.get(numOfGroup).add(numbersOfGroup);

                            for (int ii = 0; ii < getColumns(numbersOfGroup).length; ii++) {

                                if (getColumns(numbersOfGroup)[ii].isEmpty()) {
                                    continue;
                                }
                                if (ii < positionOfNumbers.size()) {
                                    positionOfNumbers.get(ii).put(getColumns(numbersOfGroup)[ii], numOfGroup);

                                } else {
                                    HashMap<String, Integer> map = new HashMap<>();

                                    map.put(getColumns(numbersOfGroup)[ii], numOfGroup);
                                    positionOfNumbers.add(map);
                                }
                            }
                        }
                        numberOfGroups.set(numOfGroup2, new HashSet<>());
                    }
                }
            }

            if (numOfGroup == null) {
                if (Arrays.stream(columns).anyMatch(s -> !s.isEmpty())) {
                    numberOfGroups.add(new HashSet<>(List.of(line)));

                    for (int ii = 0; ii < columns.length; ii++) {
                        if (columns[ii].isEmpty()) {
                            continue;
                        }

                        if (ii < positionOfNumbers.size()) {
                            positionOfNumbers.get(ii).put(columns[ii], numberOfGroups.size() - 1);
                        } else {
                            HashMap<String, Integer> map = new HashMap<>();
                            map.put(columns[ii], numberOfGroups.size() - 1);
                            positionOfNumbers.add(map);
                        }
                    }
                }
            } else {
                numberOfGroups.get(numOfGroup).add(line);

                for (int ii = 0; ii < columns.length; ii++) {
                    if (columns[ii].isEmpty()) {
                        continue;
                    }

                    if (ii < positionOfNumbers.size()) {
                        positionOfNumbers.get(ii).put(columns[ii], numOfGroup);
                    } else {
                        HashMap<String, Integer> map = new HashMap<>();
                        map.put(columns[ii], numOfGroup);
                        positionOfNumbers.add(map);
                    }
                }
            }
            line = bufferedReader.nextLine();
        }


        stringBuilder.append("group that contains the highest amount of elements ").append(numberOfGroups.stream().filter(s -> s.size() > 1).count());

        numberOfGroups.sort(Comparator.comparingInt(s -> -s.size()));
        int iterationOfGroups = 0;

        for (Set<String> perGroup : numberOfGroups) {
            iterationOfGroups++;
            stringBuilder.append("\n").append("Группа ").append(iterationOfGroups).append("\n");

            for (String setsOfNumbers : perGroup) {
                stringBuilder.append(setsOfNumbers).append("\n");
            }
        }

        bufferedWriter.write(stringBuilder.toString());
        bufferedWriter.close();
        bufferedReader.close();
    }


    private static String[] getColumns(String line) {
        for (int i = 1; i < line.length() - 1; i++) {
            if (line.charAt(i - 1) != ';' && line.charAt(i + 1) != ';' && line.charAt(i) == '"') {
                return new String[0];
            }
        }
        return line.replaceAll("\"", "").split(";");
    }
}

1. 全量输出缓存撑爆内存：用StringBuilder存储所有分组结果，大文件的输出内容会远超堆内存上限。
2. 内存存储全量文件行：numberOfGroups把所有文件行都存在内存中，Java字符串的内存开销远大于原始文本大小，1GB的文件可能需要数GB堆内存。
3. 重复解析行内容：合并分组时反复调用getColumns，生成大量临时字符串和数组，加重GC负担。
4. 无效内存占用：合并分组后仅将原集合设为空HashSet，空集合仍占用内存，未及时释放。

1. 取消全量输出缓存，边处理边写入 #

直接遍历分组时写入输出文件，避免把所有内容存在内存：

• 删除StringBuilder，遍历numberOfGroups时直接调用bufferedWriter.write()，每写若干组调用一次flush()减少内存缓存。

2. 减少内存中存储的无效数据 #

• 合并分组后直接移除原集合，而非设置为空集合，用numberOfGroups.remove(numOfGroup2)替代numberOfGroups.set(numOfGroup2, new HashSet<>())。
• 处理完成后清理空集合：numberOfGroups.removeIf(Set::isEmpty)。

3. 缓存行解析结果，避免重复计算 #

新增Map<String, String[]> lineColumnsCache，缓存每行对应的列数组，避免重复调用getColumns解析同一行。

4. 优化分组合并逻辑 #

用addAll批量合并集合，替代逐个add元素，提升效率同时减少内存操作：

Set<String> targetGroup = numberOfGroups.get(numOfGroup);
Set<String> sourceGroup = numberOfGroups.get(numOfGroup2);
targetGroup.addAll(sourceGroup);

5. 优化getColumns方法，减少临时对象 #

替换replaceAll和split为手动遍历处理，避免生成过多临时字符串。

import org.apache.commons.io.FileUtils;
import org.apache.commons.io.LineIterator;

import java.io.BufferedWriter;
import java.io.File;
import java.io.FileWriter;
import java.io.IOException;
import java.util.*;

public class HugeFileReader {

    public static void main(String[] args) throws IOException {
        String outputFile = "File created";
        LineIterator lineIterator = FileUtils.lineIterator(new File(args[0]), "UTF-8");
        // 使用try-with-resources自动关闭流
        try (BufferedWriter bufferedWriter = new BufferedWriter(new FileWriter(outputFile))) {
            List<Set<String>> numberOfGroups = new ArrayList<>();
            List<Map<String, Integer>> positionOfNumbers = new ArrayList<>();
            // 缓存行与列数组的映射，避免重复解析
            Map<String, String[]> lineColumnsCache = new HashMap<>();

            String line = lineIterator.nextLine();
            while (lineIterator.hasNext()) {
                String[] columns = lineColumnsCache.computeIfAbsent(line, HugeFileReader::getColumns);
                Integer numOfGroup = null;

                // 遍历已有的位置映射，查找所属组
                for (int i = 0; i < Math.min(positionOfNumbers.size(), columns.length); i++) {
                    Integer numOfGroup2 = positionOfNumbers.get(i).get(columns[i]);
                    if (numOfGroup2 != null) {
                        if (numOfGroup == null) {
                            numOfGroup = numOfGroup2;
                        } else if (!numOfGroup.equals(numOfGroup2)) {
                            // 批量合并两个组
                            Set<String> targetGroup = numberOfGroups.get(numOfGroup);
                            Set<String> sourceGroup = numberOfGroups.get(numOfGroup2);
                            targetGroup.addAll(sourceGroup);

                            // 更新位置映射
                            for (String groupLine : sourceGroup) {
                                String[] groupColumns = lineColumnsCache.computeIfAbsent(groupLine, HugeFileReader::getColumns);
                                for (int ii = 0; ii < groupColumns.length; ii++) {
                                    String col = groupColumns[ii];
                                    if (col.isEmpty()) continue;
                                    if (ii < positionOfNumbers.size()) {
                                        positionOfNumbers.get(ii).put(col, numOfGroup);
                                    } else {
                                        HashMap<String, Integer> map = new HashMap<>();
                                        map.put(col, numOfGroup);
                                        positionOfNumbers.add(map);
                                    }
                                }
                            }
                            // 移除原组，释放内存
                            numberOfGroups.remove(numOfGroup2);
                        }
                    }
                }

                if (numOfGroup == null) {
                    // 无所属组，新建组
                    if (Arrays.stream(columns).anyMatch(s -> !s.isEmpty())) {
                        HashSet<String> newGroup = new HashSet<>(List.of(line));
                        numberOfGroups.add(newGroup);
                        int groupIndex = numberOfGroups.size() - 1;
                        for (int ii = 0; ii < columns.length; ii++) {
                            String col = columns[ii];
                            if (col.isEmpty()) continue;
                            if (ii < positionOfNumbers.size()) {
                                positionOfNumbers.get(ii).put(col, groupIndex);
                            } else {
                                HashMap<String, Integer> map = new HashMap<>();
                                map.put(col, groupIndex);
                                positionOfNumbers.add(map);
                            }
                        }
                    }
                } else {
                    // 加入已有组
                    numberOfGroups.get(numOfGroup).add(line);
                    for (int ii = 0; ii < columns.length; ii++) {
                        String col = columns[ii];
                        if (col.isEmpty()) continue;
                        if (ii < positionOfNumbers.size()) {
                            positionOfNumbers.get(ii).put(col, numOfGroup);
                        } else {
                            HashMap<String, Integer> map = new HashMap<>();
                            map.put(col, numOfGroup);
                            positionOfNumbers.add(map);
                        }
                    }
                }
                line = lineIterator.nextLine();
            }

            // 清理空组和无效元素
            numberOfGroups.removeIf(Objects::isNull);
            numberOfGroups.removeIf(Set::isEmpty);

            // 直接写入统计信息
            long validGroupCount = numberOfGroups.stream().filter(s -> s.size() > 1).count();
            bufferedWriter.write("group that contains the highest amount of elements " + validGroupCount);
            bufferedWriter.newLine();

            // 按组大小降序排序
            numberOfGroups.sort((a, b) -> Integer.compare(b.size(), a.size()));
            int iterationOfGroups = 0;
            for (Set<String> perGroup : numberOfGroups) {
                iterationOfGroups++;
                bufferedWriter.write("Группа " + iterationOfGroups);
                bufferedWriter.newLine();
                for (String setsOfNumbers : perGroup) {
                    bufferedWriter.write(setsOfNumbers);
                    bufferedWriter.newLine();
                }
                // 每10组flush一次，减少内存缓存
                if (iterationOfGroups % 10 == 0) {
                    bufferedWriter.flush();
                }
            }
        } finally {
            lineIterator.close();
        }
    }

    private static String[] getColumns(String line) {
        // 检查行中是否存在不符合规则的引号
        for (int i = 1; i < line.length() - 1; i++) {
            if (line.charAt(i - 1) != ';' && line.charAt(i + 1) != ';' && line.charAt(i) == '"') {
                return new String[0];
            }
        }
        // 手动移除引号并分割列，减少临时对象
        int length = line.length();
        StringBuilder sb = new StringBuilder(length);
        for (int i = 0; i < length; i++) {
            char c = line.charAt(i);
            if (c != '"') {
                sb.append(c);
            }
        }
        String cleaned = sb.toString();
        List<String> cols = new ArrayList<>();
        int start = 0;
        int end = cleaned.indexOf(';');
        while (end != -1) {
            cols.add(cleaned.substring(start, end));
            start = end + 1;
            end = cleaned.indexOf(';', start);
        }
        cols.add(cleaned.substring(start));
        return cols.toArray(new String[0]);
    }
}

内容的提问来源于stack exchange，提问作者DAN