React应用批量获取大数据时的性能优化方案咨询

针对一次性请求950个项目导致的API响应慢、应用长期加载的问题，核心优化思路是控制并发请求数量，避免同时发起大量请求拖垮服务端和浏览器，同时渐进式渲染已加载的项目，让用户快速看到内容。

具体修改步骤 #

1. 新增分批请求状态 #

在组件中添加状态跟踪当前加载进度：

const [currentBatch, setCurrentBatch] = useState(0);
const [hasMore, setHasMore] = useState(true);
const [isBatchLoading, setIsBatchLoading] = useState(false);

2. 封装单个项目请求逻辑 #

把单个项目的请求抽成独立函数，支持单个项目缓存，提升复用性：

const fetchSingleProject = async (projectIdItem: any): Promise<Project | null> => {
  const projectId = projectIdItem.projectIds;
  // 先检查单个项目缓存
  const cachedProject = getFromApiCache(`project_${projectId}`);
  if (cachedProject) {
    return cachedProject;
  }

  try {
    const projectResponse = await axios.get(
      `${BASE_URL}/projects/${projectId}`,
      { headers: { Authorization: `Bearer ${TOKEN}` } }
    );
    const projectData = projectResponse.data;
    const gitlabUrl = projectData.web_url;

    const ciLintResponse = await axios.get(
      `${BASE_URL}/projects/${projectId}/ci/lint`,
      { headers: { Authorization: `Bearer ${TOKEN}` } }
    );
    const ciLintData = ciLintResponse.data;
    const imageArray = extractImagesFromYaml(ciLintData.merged_yaml);

    const newProject: Project = {
      projectId,
      projectName: projectData.name,
      gitlabUrl,
      images: imageArray,
    };

    // 缓存单个项目，而非全量列表
    updateApiCache(`project_${projectId}`, newProject);
    return newProject;
  } catch (error) {
    console.error(`Error fetching project ${projectId}`, error);
    return null;
  }
};

3. 实现分批加载逻辑 #

定义批次大小（可根据API承受能力调整），每次加载一批项目，加载完成后合并到现有列表：

const BATCH_SIZE = 10;

const fetchNextBatch = async () => {
  if (isBatchLoading || !hasMore) return;
  setIsBatchLoading(true);

  try {
    const startIndex = currentBatch * BATCH_SIZE;
    const endIndex = startIndex + BATCH_SIZE;
    const batchIds = projectIds.slice(startIndex, endIndex);

    if (batchIds.length === 0) {
      setHasMore(false);
      return;
    }

    // 并发请求当前批次的项目
    const batchPromises = batchIds.map(fetchSingleProject);
    const batchResults = await Promise.all(batchPromises);
    const validProjects = batchResults.filter((p): p is Project => p !== null);

    // 合并到现有列表，而非替换
    setProjects(prev => [...prev, ...validProjects]);
    setCurrentBatch(prev => prev + 1);
  } catch (error) {
    setError("加载项目批次失败");
  } finally {
    setIsBatchLoading(false);
  }
};

4. 调整初始化与滚动加载逻辑 #

初始化时加载第一批项目，同时添加滚动加载触发下一批：

useEffect(() => {
  const initProjects = async () => {
    setLoading(true);
    setError(null);

    // 优先读取全量缓存
    const cachedProjects = getFromApiCache("projects");
    if (cachedProjects) {
      setProjects(cachedProjects);
      setLoading(false);
      setHasMore(false);
      return;
    }

    // 加载第一批项目
    await fetchNextBatch();
    setLoading(false);
  };

  initProjects();
}, []);

// 滚动到底部自动加载下一批
useEffect(() => {
  const handleScroll = () => {
    if (
      window.innerHeight + document.documentElement.scrollTop >=
      document.documentElement.offsetHeight - 100 &&
      hasMore &&
      !isBatchLoading
    ) {
      fetchNextBatch();
    }
  };

  window.addEventListener("scroll", handleScroll);
  return () => window.removeEventListener("scroll", handleScroll);
}, [currentBatch, hasMore, isBatchLoading]);

// 全量加载完成后更新全量缓存
useEffect(() => {
  if (!hasMore && projects.length > 0) {
    updateApiCache("projects", projects);
  }
}, [hasMore, projects, updateApiCache]);

5. 更新UI加载状态 #

区分初始加载和批次加载的提示：

const renderBatchLoading = () => {
  if (isBatchLoading && hasMore) {
    return (
      <div className="d-flex justify-content-center my-3">
        <Spinner animation="border" variant="primary" />
      </div>
    );
  }
  if (!hasMore) {
    return <div className="text-center my-3">已加载全部项目</div>;
  }
  return null;
};

修改后的完整组件代码 #

import React, { useEffect, useState } from "react";
import axios from "axios";
import { Card, Container, Col, Button, Spinner } from "react-bootstrap";
import { BASE_URL, TOKEN, extractImagesFromYaml } from "./utils";
import projectIds from "./projectIds.json";
import { Project } from "./type";
import { useApiCache } from "./ApiContext";

const ProjectsPage: React.FC = () => {
  const { updateApiCache, getFromApiCache } = useApiCache();
  const [projects, setProjects] = useState<Project[]>([]);
  const [loading, setLoading] = useState<boolean>(true);
  const [error, setError] = useState<string | null>(null);
  const [visibleImages, setVisibleImages] = useState(2);
  const [searchTerm, setSearchTerm] = useState("");
  const [isSearchMode, setIsSearchMode] = useState<boolean>(false);
  // 分批请求相关状态
  const [currentBatch, setCurrentBatch] = useState(0);
  const [hasMore, setHasMore] = useState(true);
  const [isBatchLoading, setIsBatchLoading] = useState(false);

  const BATCH_SIZE = 10; // 可根据API性能调整

  const handleLoadMore = () => {
    setVisibleImages((prevVisibleImages) => prevVisibleImages + 3);
  };

  const handleGitLabButtonClick = (gitLabUrl: string) => {
    window.open(gitLabUrl, "_blank");
  };

  const handleSearch = (e: React.ChangeEvent<HTMLInputElement>) => {
    const value = e.target.value.toLowerCase();
    setSearchTerm(value);
    setIsSearchMode(value !== "");
  };

  const filteredProjects = projects.filter((project) => {
    const searchTermLower = searchTerm.toLowerCase();
    const projectNameMatch = project.projectName.toLowerCase().includes(searchTermLower);
    const imageMatch = project.images.some((imageObj) =>
      imageObj.image.toLowerCase().includes(searchTermLower)
    );
    return project.images.length > 0 && (projectNameMatch || imageMatch);
  });

  // 单个项目请求逻辑
  const fetchSingleProject = async (projectIdItem: any): Promise<Project | null> => {
    const projectId = projectIdItem.projectIds;
    const cachedProject = getFromApiCache(`project_${projectId}`);
    if (cachedProject) {
      return cachedProject;
    }

    try {
      const projectResponse = await axios.get(
        `${BASE_URL}/projects/${projectId}`,
        { headers: { Authorization: `Bearer ${TOKEN}` } }
      );
      const projectData = projectResponse.data;
      const gitlabUrl = projectData.web_url;

      const ciLintResponse = await axios.get(
        `${BASE_URL}/projects/${projectId}/ci/lint`,
        { headers: { Authorization: `Bearer ${TOKEN}` } }
      );
      const ciLintData = ciLintResponse.data;
      const imageArray = extractImagesFromYaml(ciLintData.merged_yaml);

      const newProject: Project = {
        projectId,
        projectName: projectData.name,
        gitlabUrl,
        images: imageArray,
      };

      updateApiCache(`project_${projectId}`, newProject);
      return newProject;
    } catch (error) {
      console.error(`Error fetching project ${projectId}`, error);
      return null;
    }
  };

  // 加载下一批项目
  const fetchNextBatch = async () => {
    if (isBatchLoading || !hasMore) return;
    setIsBatchLoading(true);

    try {
      const startIndex = currentBatch * BATCH_SIZE;
      const endIndex = startIndex + BATCH_SIZE;
      const batchIds = projectIds.slice(startIndex, endIndex);

      if (batchIds.length === 0) {
        setHasMore(false);
        return;
      }

      const batchPromises = batchIds.map(fetchSingleProject);
      const batchResults = await Promise.all(batchPromises);
      const validProjects = batchResults.filter((p): p is Project => p !== null);

      setProjects(prev => [...prev, ...validProjects]);
      setCurrentBatch(prev => prev + 1);
    } catch (error) {
      setError("加载项目批次失败");
    } finally {
      setIsBatchLoading(false);
    }
  };

  // 初始化加载逻辑
  useEffect(() => {
    const initProjects = async () => {
      setLoading(true);
      setError(null);

      const cachedProjects = getFromApiCache("projects");
      if (cachedProjects) {
        setProjects(cachedProjects);
        setLoading(false);
        setHasMore(false);
        return;
      }

      await fetchNextBatch();
      setLoading(false);
    };

    initProjects();
  }, []);

  // 滚动加载下一批
  useEffect(() => {
    const handleScroll = () => {
      if (
        window.innerHeight + document.documentElement.scrollTop >=
        document.documentElement.offsetHeight - 100 &&
        hasMore &&
        !isBatchLoading
      ) {
        fetchNextBatch();
      }
    };

    window.addEventListener("scroll", handleScroll);
    return () => window.removeEventListener("scroll", handleScroll);
  }, [currentBatch, hasMore, isBatchLoading]);

  // 全量加载完成后更新缓存
  useEffect(() => {
    if (!hasMore && projects.length > 0) {
      updateApiCache("projects", projects);
    }
  }, [hasMore, projects, updateApiCache]);

  if (loading) {
    return (
      <div className="d-flex justify-content-center align-items-center" style={{ minHeight: "100vh" }}>
        <h3>Loading...</h3>
        <Spinner animation="border" variant="primary" />
      </div>
    );
  }

  if (error) {
    return (
      <div className="d-flex justify-content-center align-items-center" style={{ minHeight: "100vh" }}>
        Error: {error}
      </div>
    );
  }

  // 批次加载提示
  const renderBatchLoading = () => {
    if (isBatchLoading && hasMore) {
      return (
        <div className="d-flex justify-content-center my-3">
          <Spinner animation="border" variant="primary" />
        </div>
      );
    }
    if (!hasMore) {
      return <div className="text-center my-3">已加载全部项目</div>;
    }
    return null;
  };

  function highlightSearchTerm(text: string, searchTerm: string) {
    if (!searchTerm) {
      return text;
    }

    const regex = new RegExp(`(${searchTerm})`, "gi");
    const parts = text.split(regex);

    return parts.map((part, index) => {
      if (part.toLowerCase() === searchTerm.toLowerCase()) {
        return (
          <span key={index} style={{ backgroundColor: "red" }}>
            {part}
          </span>
        );
      }
      return part;
    });
  }

  return (
    <Container>
      <div className="d-flex justify-content-between align-items-center mb-2">
        <h1>Projects</h1>
        <div className="search-input">
          <input
            type="text"
            className="form-control"
            placeholder="Search by project name or image"
            onChange={handleSearch}
          />
        </div>
      </div>
      {filteredProjects.length === 0 && isSearchMode ? (
        <div className="d-flex justify-content-center align-items-center">
          <h3>No projects found</h3>
        </div>
      ) : (
        <div>
          {filteredProjects.map((project) => (
            <Col key={project.projectId} xs="auto" style={{ marginBottom: "1rem" }}>
              <Card
                bg="success"
                text="white"
                style={{ width: "auto" }}
                className="mb-2"
              >
                <Card.Body>
                  <Card.Title>
                    {highlightSearchTerm(project.projectName, searchTerm)}
                  </Card.Title>
                  <Card.Title>{project.projectId}</Card.Title>
                  <Card.Text className="mb-1">Use image:</Card.Text>
                  <ul className="mb-0 pb-0">
                    {project.images
                      .slice(0, visibleImages)
                      .map((image: any, i: any) => (
                        <li key={i}>
                          {highlightSearchTerm(image.image, searchTerm)}
                        </li>
                      ))}
                  </ul>
                  <div style={{ marginTop: "1rem" }}>
                    {visibleImages < project.images.length && (
                      <Button
                        variant="secondary"
                        onClick={handleLoadMore}
                        style={{ marginRight: "0.5rem" }}
                      >
                        Load More
                      </Button>
                    )}
                    <Button
                      variant="primary"
                      onClick={() => handleGitLabButtonClick(project.gitlabUrl)}
                    >
                      Repo
                    </Button>
                  </div>
                </Card.Body>
              </Card>
            </Col>
          ))}
          {renderBatchLoading()}
        </div>
      )}
    </Container>
  );
};

export default ProjectsPage;

额外优化建议 #

• 调整批次大小：根据API的QPS限制修改BATCH_SIZE，平衡加载速度和服务器压力。
• 添加错误重试：为单个项目请求添加重试逻辑（如失败后重试1-2次），提升请求成功率。
• 搜索优先加载：若用户在初始加载时发起搜索，可暂停批次加载，优先过滤已加载项目，同时针对未加载项目优先请求匹配搜索关键词的内容（需后端支持搜索接口）。

内容的提问来源于stack exchange，提问作者user12494839