HangFire定时任务并发多次执行问题求助

我有一个使用HangFire的C# Worker，配置和代码如下：

Program.cs 配置 #

var sqliteOptions = new SQLiteStorageOptions();
builder.Services.AddHangfire(configuration => configuration
    .UseSimpleAssemblyNameTypeSerializer()
    .UseRecommendedSerializerSettings()
    .UseFilter(new DisableConcurrentExecutionAttribute(300))
    .UseSQLiteStorage($"Filename={hangfireDbPath};", sqliteOptions));

builder.Services.AddHangfireServer(options =>
{
    options.WorkerCount = 1; // Limit to one worker
});

Worker.cs 代码 #

protected override async Task ExecuteAsync(CancellationToken stoppingToken)
{
    _fileMonitor.StartMonitoring();

    _logger.LogInformation("Backup service is starting.");

    // Start the Hangfire server
    using (var server = new BackgroundJobServer())
    {
        _backupService.ScheduleInitialJobs();

        _logger.LogInformation("Backup service has scheduled all initial backup jobs.");

        await Task.Delay(Timeout.Infinite, stoppingToken);
    }
}

BackupService 代码 #

public void ScheduleInitialJobs()
{
    var backupPlans = _backupPlanService.GetAllBackupPlans();

    foreach (var plan in backupPlans)
    {
        if (plan.IsEnabled)
        {
            var cronExpression = GetCronExpression(plan);
            var options = new RecurringJobOptions
            {
                TimeZone = TimeZoneInfo.Local
            };
            try
            {
                _recurringJobManager.AddOrUpdate(
                    plan.Name,
                    () => ExecuteBackupAsync(plan, CancellationToken.None),
                    cronExpression,
                    options);
            }
            catch (Exception ex)
            {
                // 空异常处理
            }
        }
    }
}

ExecuteBackupAsync 方法 #

[MaximumConcurrentExecutions(1)]
public async Task ExecuteBackupAsync(BackupPlan plan, CancellationToken cancellationToken)
{
    // 备份逻辑
}

日志信息 #

2024-05-15 23:05:09.297 -06:00 [DBG] 1 recurring job(s) processed by scheduler.
2024-05-15 23:05:09.302 -06:00 [DBG] 1 recurring job(s) processed by scheduler.
2024-05-15 23:05:24.113 -06:00 [DBG] Server ava:90924:57316309 heartbeat successfully sent
2024-05-15 23:05:24.165 -06:00 [INF] Executing backup for plan: Plan01
2024-05-15 23:05:24.171 -06:00 [INF] Completed backup for plan: Plan01
2024-05-15 23:05:24.378 -06:00 [DBG] Server ava:90924:0204f2f5 heartbeat successfully sent
2024-05-15 23:05:24.381 -06:00 [INF] Executing backup for plan: Plan01
2024-05-15 23:05:24.382 -06:00 [INF] Completed backup for plan: Plan01
2024-05-15 23:05:24.386 -06:00 [INF] Executing backup for plan: Plan01
2024-05-15 23:05:24.387 -06:00 [INF] Completed backup for plan: Plan01
2024-05-15 23:05:24.400 -06:00 [INF] Executing backup for plan: Plan01
2024-05-15 23:05:24.401 -06:00 [INF] Completed backup for plan: Plan01
2024-05-15 23:05:24.403 -06:00 [INF] Executing backup for plan: Plan01
2024-05-15 23:05:24.404 -06:00 [INF] Completed backup for plan: Plan01
2024-05-15 23:05:24.419 -06:00 [INF] Executing backup for plan: Plan01
2024-05-15 23:05:24.420 -06:00 [INF] Completed backup for plan: Plan01
2024-05-15 23:05:24.587 -06:00 [INF] Executing backup for plan: Plan01
2024-05-15 23:05:24.588 -06:00 [INF] Completed backup for plan: Plan01
2024-05-15 23:05:24.590 -06:00 [INF] Executing backup for plan: Plan01
2024-05-15 23:05:24.591 -06:00 [INF] Completed backup for plan: Plan01
2024-05-15 23:05:24.594 -06:00 [INF] Executing backup for plan: Plan01
2024-05-15 23:05:24.595 -06:00 [INF] Completed backup for plan: Plan01
2024-05-15 23:05:24.601 -06:00 [INF] Executing backup for plan: Plan01
2024-05-15 23:05:24.602 -06:00 [INF] Completed backup for plan: Plan01
2024-05-15 23:05:54.125 -06:00 [DBG] Server ava:90924:57316309 heartbeat successfully sent
2024-05-15 23:05:54.387 -06:00 [DBG] Server ava:90924:0204f2f5 heartbeat successfully sent

我已确认HangFire数据库中仅添加了一次定时任务，但日志显示任务被重复执行多次，求解决办法。

核心原因 #

从日志中可以看到两个不同的Server ID（ava:90924:57316309 和 ava:90924:0204f2f5）在发送心跳，说明应用中启动了两个独立的HangFire Server实例：

1. Program.cs中通过builder.Services.AddHangfireServer()注册的Server（由DI容器管理生命周期）
2. Worker.cs中手动new BackgroundJobServer()创建的Server

两个Server会同时监听并执行定时任务，导致任务重复执行。

修复步骤 #

1. 移除Worker.cs中手动创建的BackgroundJobServer
   修改Worker.cs的ExecuteAsync方法，删除手动创建Server的代码，因为AddHangfireServer()已经会自动启动并管理Server的生命周期：

   protected override async Task ExecuteAsync(CancellationToken stoppingToken)
   {
       _fileMonitor.StartMonitoring();
   
       _logger.LogInformation("Backup service is starting.");
   
       // 移除手动创建的BackgroundJobServer实例
       _backupService.ScheduleInitialJobs();
   
       _logger.LogInformation("Backup service has scheduled all initial backup jobs.");
   
       await Task.Delay(Timeout.Infinite, stoppingToken);
   }
   

2. 确保ScheduleInitialJobs只执行一次
   由于Worker服务默认是单例，ExecuteAsync只会执行一次，所以ScheduleInitialJobs不会重复调用。如果后续有动态添加计划的需求，可以单独处理，避免重复调用AddOrUpdate。

3. 验证修复效果
   重启应用后，查看日志应该只会出现一个Server的心跳信息，定时任务也只会执行一次。如果启用了HangFire Dashboard，可以在Dashboard的Servers页面确认只有一个Server在运行。

额外优化建议 #

• 去掉ScheduleInitialJobs中的空异常捕获，至少记录异常信息，方便排查问题：

  catch (Exception ex)
  {
      _logger.LogError(ex, "Failed to schedule backup plan {PlanName}", plan.Name);
  }
  

• 避免在AddOrUpdate的委托中直接引用循环变量plan，可能会导致闭包捕获问题，建议先赋值给局部变量：

  var currentPlan = plan;
  _recurringJobManager.AddOrUpdate(
      currentPlan.Name,
      () => ExecuteBackupAsync(currentPlan, CancellationToken.None),
      cronExpression,
      options);
  

内容的提问来源于stack exchange，提问作者VAAA