大规模图书馆应用内存处理与性能优化方案求助

核心瓶颈分析 #

当前实现里List<AppUserEmailBooks> appUserEmailBooksList = new ArrayList<>()的设计存在两大致命问题：

• 内存过载风险：全量用户匹配数据驻留内存，在千万/亿级用户场景下必然触发OOM
• 搜索效率暴跌：每次更新用户书籍列表都要遍历全列表，用户量增长后耗时呈指数级上升，极端情况可能需要数天完成

尝试过用Map替代List，但内存消耗反而更高，下面针对「书籍添加、用户订阅、每日新书通知+避免重复推送」的业务需求，给出内存高效、性能优异的优化方案。

优化方案 #

1. 数据库层面：用SQL直接完成订阅与新书匹配，砍掉内存计算 #

当前代码先拉取所有订阅，再在内存中逐个匹配新书，这是效率低下的核心根源。改为让数据库直接关联查询，只返回需要发送通知的用户与对应书籍，从源头减少数据传输量。

新增Repository查询方法

在SubscriptionRepository中添加分组查询，直接按用户邮箱聚合匹配的新书：

@Query("SELECT s.appUser.email, COLLECT(b) FROM Subscription s JOIN Book b ON " +
       "(b.author = s.bookAuthor OR b.category = s.bookCategory) " +
       "WHERE b.addedDate = CURRENT_DATE " +
       "GROUP BY s.appUser.email")
Page<Object[]> findUserEmailWithMatchedBooks(Pageable pageable);

注：若JPA不支持COLLECT，可改用原生SQL，或后续在内存中做轻量聚合

2. 内存优化：流式分批处理，避免全量数据驻留 #

重构定时任务逻辑，不需要把所有用户数据存到内存列表，而是查一批、处理一批、发送一批，处理完即释放内存，彻底解决内存占用问题。

修改EmailSchedule核心逻辑

@Component
@RequiredArgsConstructor
public class EmailSchedule {

    private final EmailService emailService;
    private final SubscriptionRepository subscriptionRepository;

    @Scheduled(cron = "${scheduled.email.notification.cron}")
    public void sendScheduledEmailNotification() {
        final int pageSize = 5000; // 根据数据库性能调整
        Pageable pageable = PageRequest.of(0, pageSize);
        Page<Object[]> pageResult;

        do {
            pageResult = subscriptionRepository.findUserEmailWithMatchedBooks(pageable);
            processCurrentPage(pageResult.getContent());
            pageable = pageResult.nextPageable();
        } while (pageResult.hasNext());
    }

    private void processCurrentPage(List<Object[]> userBooksEntries) {
        // 若查询已完成分组，此步骤可省略
        Map<String, List<Book>> userBooksMap = new HashMap<>();
        for (Object[] entry : userBooksEntries) {
            String email = (String) entry[0];
            Book book = (Book) entry[1];
            userBooksMap.computeIfAbsent(email, k -> new ArrayList<>()).add(book);
        }

        // 直接发送邮件，无需缓存全量数据
        for (Map.Entry<String, List<Book>> entry : userBooksMap.entrySet()) {
            emailService.sendNotificationIfNewBooks(entry.getKey(), entry.getValue());
        }
    }
}

3. 避免重复通知：新增通知记录表做幂等校验 #

新增NotificationRecord实体，记录用户邮箱、书籍ID、发送日期，通过数据库层面的唯一性约束，确保同一用户不会收到同一本书的重复通知。

实体与Repository示例

@Entity
public class NotificationRecord {
    @Id
    @GeneratedValue(strategy = GenerationType.IDENTITY)
    private Long id;
    private String userEmail;
    private Long bookId;
    private LocalDate sentDate;

    // getter、setter、构造方法省略
}

public interface NotificationRecordRepository extends JpaRepository<NotificationRecord, Long> {
    // 单条校验
    boolean existsByUserEmailAndBookIdAndSentDate(String userEmail, Long bookId, LocalDate sentDate);

    // 批量插入今日未发送的通知记录（自动去重）
    @Modifying
    @Query("INSERT INTO NotificationRecord(userEmail, bookId, sentDate) " +
           "SELECT s.appUser.email, b.id, CURRENT_DATE FROM Subscription s JOIN Book b ON " +
           "(b.author = s.bookAuthor OR b.category = s.bookCategory) " +
           "WHERE b.addedDate = CURRENT_DATE " +
           "AND NOT EXISTS (" +
           "   SELECT nr FROM NotificationRecord nr WHERE nr.userEmail = s.appUser.email AND nr.bookId = b.id AND nr.sentDate = CURRENT_DATE" +
           ")")
    int batchInsertNotificationRecords();

    // 查询今日待发送的用户与书籍
    @Query("SELECT nr.userEmail, b FROM NotificationRecord nr JOIN Book b ON nr.bookId = b.id WHERE nr.sentDate = :sentDate")
    Page<Object[]> findUserEmailWithBooksBySentDate(LocalDate sentDate, Pageable pageable);
}

定时任务中先执行batchInsertNotificationRecords()完成去重，再基于插入的记录发送邮件，彻底避免重复推送。

4. 性能最佳实践 #

• 分页参数调优：根据数据库性能调整分页大小（推荐5000-10000），避免过大导致数据库压力过载，过小导致请求次数过多
• 异步发送邮件：给EmailService的发送方法添加@Async注解，用线程池异步发送，避免同步发送阻塞任务执行
• 索引优化：
  • 给Book的addedDate、author、category字段加单独索引
  • 给Subscription的bookAuthor、bookCategory字段加索引
  • 给NotificationRecord的userEmail、bookId、sentDate加联合索引
• 避免N+1查询：在订阅查询中添加JOIN FETCH s.appUser，避免懒加载触发额外数据库请求

最终核心代码示例 #

优化后的EmailSchedule #

@Component
@RequiredArgsConstructor
public class EmailSchedule {

    private final EmailService emailService;
    private final NotificationRecordRepository notificationRecordRepository;

    @Scheduled(cron = "${scheduled.email.notification.cron}")
    public void sendScheduledEmailNotification() {
        // 先批量插入今日未发送的通知记录（自动去重）
        notificationRecordRepository.batchInsertNotificationRecords();

        final int pageSize = 5000;
        Pageable pageable = PageRequest.of(0, pageSize);
        Page<Object[]> pageResult;

        do {
            pageResult = notificationRecordRepository.findUserEmailWithBooksBySentDate(LocalDate.now(), pageable);
            processCurrentPage(pageResult.getContent());
            pageable = pageResult.nextPageable();
        } while (pageResult.hasNext());
    }

    private void processCurrentPage(List<Object[]> userBooksEntries) {
        Map<String, List<Book>> userBooksMap = new HashMap<>();
        for (Object[] entry : userBooksEntries) {
            String email = (String) entry[0];
            Book book = (Book) entry[1];
            userBooksMap.computeIfAbsent(email, k -> new ArrayList<>()).add(book);
        }

        for (Map.Entry<String, List<Book>> entry : userBooksMap.entrySet()) {
            emailService.sendNotificationIfNewBooks(entry.getKey(), entry.getValue());
        }
    }
}

内容的提问来源于stack exchange，提问作者Oskar