Laravel队列创建及SOAP WSDL房产数据导入优化技术问询

Hey there! Let's walk through a practical, step-by-step solution to refactor your bulk property import into a scalable Laravel Queue workflow. Here's how to approach it:

1. Create a Dedicated Import Job #

First, generate a job class to handle individual (or small-batch) property imports. This keeps your controller clean and offloads heavy processing to the queue.

Run this Artisan command to generate the job:

php artisan make:job ImportPropertyFromSoap

Then update the job's handle() method with your core import logic. Here's a tailored example:

<?php

namespace App\Jobs;

use App\Models\Property;
use App\Models\Office;
use Illuminate\Bus\Queueable;
use Illuminate\Contracts\Queue\ShouldQueue;
use Illuminate\Foundation\Bus\Dispatchable;
use Illuminate\Queue\InteractsWithQueue;
use Illuminate\Queue\SerializesModels;
use SoapClient;

class ImportPropertyFromSoap implements ShouldQueue
{
    use Dispatchable, InteractsWithQueue, Queueable, SerializesModels;

    protected $propertyData;
    // Set retry attempts for transient errors (adjust as needed)
    public $tries = 3;

    public function __construct(array $propertyData)
    {
        $this->propertyData = $propertyData;
    }

    public function handle()
    {
        // Resolve the associated Office (use your unique identifier from the WSDL feed)
        $office = Office::firstOrCreate(
            ['external_id' => $this->propertyData['office_external_ref']],
            ['name' => $this->propertyData['office_name'], 'location' => $this->propertyData['office_location']]
        );

        // Upsert the Property: update if it exists, create if not
        Property::updateOrCreate(
            ['external_id' => $this->propertyData['property_external_ref']], // Unique key from WSDL
            [
                'title' => $this->propertyData['title'],
                'address' => $this->propertyData['full_address'],
                'price' => $this->propertyData['asking_price'],
                'office_id' => $office->id,
                // Add all other required fields here
            ]
        );
    }

    // Optional: Handle failed jobs for debugging
    public function failed(\Throwable $exception)
    {
        logger()->error('Property import failed', [
            'property_data' => $this->propertyData,
            'error_message' => $exception->getMessage(),
            'trace' => $exception->getTraceAsString()
        ]);
    }
}

2. Split Bulk Data into Manageable Batches #

Instead of processing all WSDL data at once, split it into smaller batches to avoid timeouts and improve queue efficiency. Use an Artisan command (better than a controller for long-running tasks):

Generate the command:

php artisan make:command ImportProperties

Write the command logic to fetch WSDL data and dispatch jobs in batches:

<?php

namespace App\Console\Commands;

use App\Jobs\ImportPropertyFromSoap;
use Illuminate\Console\Command;
use SoapClient;

class ImportProperties extends Command
{
    protected $signature = 'import:properties';
    protected $description = 'Import properties from SOAP WSDL feed via Laravel Queue';

    public function handle()
    {
        $this->info('Fetching property data from SOAP WSDL...');

        // Initialize SOAP client (pull URL from .env for flexibility)
        $soapClient = new SoapClient(env('SOAP_WSDL_URL'));
        // Replace with your actual SOAP method to retrieve data
        $rawSoapResponse = $soapClient->getAllProperties();

        // Convert SOAP response to a usable array (adjust based on your WSDL structure)
        $properties = json_decode(json_encode($rawSoapResponse), true);

        $totalProperties = count($properties);
        $this->info("Found {$totalProperties} properties to process.");

        // Split into batches of 50 (tune this size based on your server's capacity)
        collect($properties)->chunk(50)->each(function ($batch) {
            foreach ($batch as $property) {
                ImportPropertyFromSoap::dispatch($property);
            }
        });

        $this->info("Dispatched all {$totalProperties} import jobs to the queue!");
    }
}

3. Configure and Start the Queue Worker #

Set your preferred queue connection in .env (Redis is ideal for performance; use Database as a fallback):

QUEUE_CONNECTION=redis

Start the queue worker to process jobs in the background:

php artisan queue:work

For production, use a process manager like Supervisor to ensure the worker runs continuously even after server restarts.

4. Additional Optimizations #

• Batch Tracking: Use Laravel's Batch Jobs to monitor progress and handle failures across a group of jobs.
• Rate Limiting: If your SOAP API has rate limits, add delays to jobs with ->delay(now()->addSeconds(3)) when dispatching.
• Client Reuse: Avoid initializing a new SOAP client for every job—pass a pre-configured instance via the job constructor or use a singleton.
• Monitoring: Use Laravel Horizon (for Redis) to track queue performance, job throughput, and failed jobs.

5. Kick Off the Import #

Run the Artisan command to start the workflow:

php artisan import:properties

This will fetch data from the WSDL feed, split it into batches, and dispatch all import jobs to the queue for background processing.

内容的提问来源于stack exchange，提问作者Max Pooley