NextJS中动态API Fetch结果定期更新与存储方案咨询

Hey there! As someone who’s tackled similar dynamic API fetching and storage challenges in Next.js, let’s break down the best options for your use case—plus step-by-step implementation ideas tailored for a beginner.

Your core needs boil down to:

• Regularly fetch new articles and append them to an existing list
• Short-term storage (1-2 weeks max, no long-term retention)
• Keep things straightforward while avoiding production pitfalls

Let’s compare storage options and pick the optimal path.

1. Memory Arrays (Not Recommended for Production) #

A simple in-memory array works for quick local testing, but it has critical flaws:

• If your Next.js server restarts (e.g., deployments, scaling), all stored data vanishes.
• Multiple server instances (common in production) will each have their own separate list—leading to inconsistent data across users.
• No built-in way to auto-expire old entries.

Stick with this only for throwaway dev tests, not production.

2. SQLite + Prisma (Great for Small to Mid-Size Projects) #

If you want a self-contained, zero-external-service solution, SQLite paired with Prisma (Next.js’s go-to ORM) is perfect. It’s file-based, so no need to spin up a separate database server, and you can easily add logic to auto-clean old entries.

Implementation Steps: #

1. Set Up Prisma and SQLite

   • Install dependencies: npm install prisma @prisma/client
   • Initialize Prisma: npx prisma init --datasource-provider sqlite
   • Update your prisma/schema.prisma to define an Article model:

     model Article {
       id          String   @id @default(cuid())
       externalId  String   @unique // Unique ID from your API to avoid duplicates
       title       String
       content     String
       apiCreatedAt DateTime // The article's publish time from the API
       createdAt   DateTime @default(now())
     }
     

   • Run the migration to create the database: npx prisma migrate dev --name init

2. Build the Sync Logic
   Create a reusable function to fetch new articles, avoid duplicates, and clean old entries:

   // lib/syncArticles.js
   import { PrismaClient } from '@prisma/client';
   
   const prisma = new PrismaClient();
   
   export async function syncArticles() {
     try {
       // Fetch latest articles from your API
       const apiResponse = await fetch('YOUR_DYNAMIC_API_ENDPOINT');
       const newArticles = await apiResponse.json();
   
       // Upsert articles (add new ones, skip existing duplicates)
       for (const article of newArticles) {
         await prisma.article.upsert({
           where: { externalId: article.id },
           update: {}, // Leave empty if you don't want to update existing entries
           create: {
             externalId: article.id,
             title: article.title,
             content: article.content,
             apiCreatedAt: new Date(article.publishedAt)
           }
         });
       }
   
       // Clean up articles older than 2 weeks
       const twoWeeksAgo = new Date();
       twoWeeksAgo.setDate(twoWeeksAgo.getDate() - 14);
       await prisma.article.deleteMany({
         where: { apiCreatedAt: { lt: twoWeeksAgo } }
       });
   
       console.log('Sync completed successfully');
     } catch (error) {
       console.error('Error syncing articles:', error);
     } finally {
       await prisma.$disconnect();
     }
   }
   

3. Set Up Scheduled Sync

   • If deploying to Vercel, use Vercel Cron Jobs to trigger the sync at regular intervals (e.g., every hour). Create an API Route to run the sync:

     // pages/api/sync-articles.js
     import { syncArticles } from '../../lib/syncArticles';
     
     export default async function handler(req, res) {
       await syncArticles();
       res.status(200).json({ message: 'Article sync finished' });
     }
     

   • For self-hosted Next.js, use a package like node-schedule to run the sync function on a timer.

4. Fetch and Display Articles
   Use a Next.js Server Component to fetch stored articles and render them:

   // app/articles/page.js
   import { PrismaClient } from '@prisma/client';
   
   const prisma = new PrismaClient();
   
   export default async function ArticlesPage() {
     const articles = await prisma.article.findMany({
       orderBy: { apiCreatedAt: 'desc' } // Show newest first
     });
   
     await prisma.$disconnect();
   
     return (
       <div className="container mx-auto p-4">
         <h1 className="text-2xl font-bold mb-4">Latest Articles</h1>
         <ul className="space-y-4">
           {articles.map(article => (
             <li key={article.id} className="border p-4 rounded">
               <h2 className="text-xl font-semibold">{article.title}</h2>
               <p className="mt-2">{article.content}</p>
             </li>
           ))}
         </ul>
       </div>
     );
   }
   

3. Redis (Best for High-Performance, Auto-Expiring Storage) #

If you need faster read/write speeds and want automatic expiration (no manual cleanup), Redis is ideal. It’s an in-memory data store that supports TTL (time-to-live) for keys, so you can set your article list to auto-delete after 2 weeks.

Implementation Steps: #

1. Set Up Redis

   • Install the Redis client: npm install ioredis
   • Create a Redis client instance (store it in a reusable file):

     // lib/redis.js
     import Redis from 'ioredis';
     
     const redis = new Redis(process.env.REDIS_URL); // Use your Redis connection string
     
     export default redis;
     

2. Build the Sync Logic
   Create a function to fetch new articles, append them to the stored list, and set the TTL:

   // lib/syncArticles.js
   import redis from './redis';
   
   export async function syncArticles() {
     try {
       // Fetch new articles from the API
       const apiResponse = await fetch('YOUR_DYNAMIC_API_ENDPOINT');
       const newArticles = await apiResponse.json();
   
       // Get existing articles from Redis
       const storedArticlesStr = await redis.get('articles');
       const storedArticles = storedArticlesStr ? JSON.parse(storedArticlesStr) : [];
   
       // Filter out duplicates (using article ID)
       const existingIds = new Set(storedArticles.map(art => art.id));
       const articlesToAdd = newArticles.filter(art => !existingIds.has(art.id));
   
       // Update the list
       const updatedArticles = [...storedArticles, ...articlesToAdd];
   
       // Store back in Redis with 2-week TTL (1209600 seconds = 14 days)
       await redis.set('articles', JSON.stringify(updatedArticles), 'EX', 1209600);
   
       console.log(`Added ${articlesToAdd.length} new articles`);
     } catch (error) {
       console.error('Sync failed:', error);
     }
   }
   

3. Scheduled Sync & Frontend Display
   Follow the same scheduled sync steps as the SQLite option (Vercel Cron or node-schedule). For frontend display, use a Server Component to fetch from Redis:

   // app/articles/page.js
   import redis from '../lib/redis';
   
   export default async function ArticlesPage() {
     const storedArticlesStr = await redis.get('articles');
     const articles = storedArticlesStr ? JSON.parse(storedArticlesStr) : [];
   
     return (
       <div className="container mx-auto p-4">
         <h1 className="text-2xl font-bold mb-4">Latest Articles</h1>
         <ul className="space-y-4">
           {articles.map(article => (
             <li key={article.id} className="border p-4 rounded">
               <h2 className="text-xl font-semibold">{article.title}</h2>
               <p className="mt-2">{article.content}</p>
             </li>
           ))}
         </ul>
       </div>
     );
   }
   

• Go with SQLite + Prisma if you’re a beginner and want a zero-config, self-contained solution. It’s easy to set up, maintain, and doesn’t require external services.
• Choose Redis if you need better performance or already use caching in your project—auto-expiration saves you from writing custom cleanup logic.

Avoid in-memory arrays for production, as they’re unreliable for even short-term persistent storage.

内容的提问来源于stack exchange，提问作者Moshe