Yii2每日批量创建5000-10000条数据的技术咨询

Alright, let's tackle this problem where you need to create 5000-10000 daily Loghour3d entries based on your existing Sku3d table. Your current code generates one record per SKU, which probably won't reach that target volume. Here's a practical, efficient approach to adjust your logic:

Step 1: Fetch Base Data & Define Target Record Count #

First, grab the SKU data (we'll only select the fields we need to save memory) and randomly pick how many records to generate today:

// Fetch only necessary SKU fields to reduce memory usage
$skus = Sku3d::select(['sku', 'modeler', 'team', 'handover', 'totalhours'])->get();

// Exit early if there are no SKUs to avoid errors
if ($skus->isEmpty()) {
    return;
}

// Randomly determine the number of records to create (5000-10000)
$totalRecords = rand(5000, 10000);
$today = date('Y-m-d'); // Simplified date formatting

Step 2: Build Batch Record Data #

Loop through the target count, randomly selecting a SKU each time to build your Loghour3d entries:

$logRecords = [];

for ($i = 0; $i < $totalRecords; $i++) {
    // Pick a random SKU from the collection
    $randomSku = $skus->random();

    $logRecords[] = [
        'sku' => $randomSku->sku,
        'modeler' => $randomSku->modeler,
        'team' => $randomSku->team,
        'day' => $today,
        'handover' => $randomSku->handover,
        'hour' => $randomSku->totalhours,
        // Add timestamps if your model uses them
        'created_at' => now(),
        'updated_at' => now()
    ];
}

Step 3: Batch Insert for Performance #

Instead of saving each record individually (which is slow for thousands of entries), use batch insertion. We'll split the records into chunks to avoid memory issues:

$chunkSize = 1000; // Adjust based on your server's memory limits

// Insert records in chunks
foreach (array_chunk($logRecords, $chunkSize) as $chunk) {
    Loghour3d::insert($chunk);
}

Optional Optimizations #

• Weighted SKU Selection: If you want certain SKUs to generate more records, add a weight field to your Sku3d table. You can then select SKUs based on their weight to skew the distribution:

  // Example: Use SKU weights to influence selection probability
  $weightedPool = [];
  foreach ($skus as $sku) {
      // Repeat the SKU in the pool based on its weight
      $weightedPool = array_merge($weightedPool, array_fill(0, $sku->weight, $sku));
  }
  
  // Pick a random SKU from the weighted pool
  $randomSku = $weightedPool[array_rand($weightedPool)];
  

• Automate Daily Execution: Use Laravel's Task Scheduler to run this code automatically every day. Add this to app/Console/Kernel.php:

  protected function schedule(Schedule $schedule)
  {
      // Run at 1 AM every day
      $schedule->call(function () {
          // Place all the generation code here
      })->dailyAt('01:00');
  }
  

内容的提问来源于stack exchange，提问作者Sao Ho