Hangfire启动时创建两个服务器实例问题排查求助

问题现象 #

应用启动后，Hangfire会创建两个服务器实例，其中一个实例（C7518906-D440-4113-85AD-0C1A97E01ACC）在调试日志中无任何启动记录，但心跳更新早于日志中显示的实例。已尝试清空所有Hangfire相关表、启动前删除所有服务器记录，问题仍复现。

配置代码 #

服务注册阶段 #

services.AddHangfire((provider, hangfireConfig) => hangfireConfig
            .UseDatabase(storageSettings.StorageProvider, storageSettings.ConnectionString, config)
            .UseConsole());

services.AddHangfireServer(options => config.GetSection("HangfireSettings:Server").Bind(options));

仪表盘配置（builder.Build()后） #

var dashboardOptions = config.GetSection("HangfireSettings:Dashboard").Get<DashboardOptions>();

dashboardOptions.Authorization = new[]
{
   new HangfireCustomBasicAuthenticationFilter
   {
        User = config.GetSection("HangfireSettings:Credentials:User").Value,
        Pass = config.GetSection("HangfireSettings:Credentials:Password").Value
   }
};

app.UseHangfireDashboard(config["HangfireSettings:Route"], dashboardOptions);

调试日志 #

[09:01:20 INF] Starting Hangfire Server using job storage: 'SQL Server: xxxx'
[09:01:20 INF] Using the following options for SQL Server job storage: Queue poll interval: 00:00:01.
[09:01:20 INF] Using the following options for Hangfire Server:
    Worker count: 5
    Listening queues: 'default', 'default', 'notdefault', 'background', 'export', 'provisioning'
    Shutdown timeout: 00:00:15
    Schedule polling interval: 00:00:15
[09:01:20 DBG] Execution loop BackgroundServerProcess:6446fe1e has started in 14,3531 ms
[09:01:20 INF] Application started. Press Ctrl+C to shut down.
[09:01:20 INF] Hosting environment: NotDevelopment
[09:01:20 INF] Server 55b4c516-d5a9-4516-be31-29b9a4e83f1c successfully announced in 189,132 ms
[09:01:20 INF] Server 55b4c516-d5a9-4516-be31-29b9a4e83f1c is starting the registered dispatchers: ServerWatchdog, ServerJobCancellationWatcher, ExpirationManager, CountersAggregator, SqlServerHeartbeatProcess, Worker, DelayedJobScheduler, RecurringJobScheduler...
[09:01:20 DBG] Execution loop ServerHeartbeatProcess:bab9cda3 has started in 7,4454 ms
[09:01:20 DBG] Execution loop ServerWatchdog:d2946b36 has started in 7,503 ms
[09:01:20 DBG] Execution loop ServerJobCancellationWatcher:8325617f has started in 11,7685 ms
[09:01:20 DBG] Execution loop ExpirationManager:e95adc5c has started in 10,9894 ms
[09:01:20 DBG] Execution loop CountersAggregator:cedbdca3 has started in 12,2819 ms
[09:01:20 DBG] Removing outdated records from the 'AggregatedCounter' table...
[09:01:20 DBG] Execution loop SqlServerHeartbeatProcess:f57fa439 has started in 11,9128 ms
[09:01:20 DBG] Aggregating records in 'Counter' table...
[09:01:20 DBG] Execution loop Worker:e40656de has started in 12,6065 ms
[09:01:20 DBG] Execution loop Worker:aa196c8a has started in 18,7138 ms
[09:01:20 DBG] Execution loop Worker:f73ee2ec has started in 24,7742 ms
[09:01:20 DBG] Execution loop Worker:6c8a5c72 has started in 30,6857 ms
[09:01:20 INF] Server 55b4c516-d5a9-4516-be31-29b9a4e83f1c all the dispatchers started
[09:01:20 DBG] Execution loop Worker:21626abb has started in 37,1683 ms
[09:01:20 DBG] Execution loop DelayedJobScheduler:e419d23f has started in 12,6418 ms
[09:01:20 DBG] Execution loop RecurringJobScheduler:677d3276 has started in 12,5518 ms
[09:01:20 DBG] Removing outdated records from the 'Job' table...
[09:01:20 DBG] Removing outdated records from the 'List' table...
[09:01:20 DBG] Removing outdated records from the 'Set' table...
[09:01:20 DBG] Removing outdated records from the 'Hash' table...
[09:01:50 DBG] Server 55b4c516-d5a9-4516-be31-29b9a4e83f1c heartbeat successfully sent
[09:02:20 DBG] Server 55b4c516-d5a9-4516-be31-29b9a4e83f1c heartbeat successfully sent
[09:02:50 DBG] Server 55b4c516-d5a9-4516-be31-29b9a4e83f1c heartbeat successfully sent
[09:03:20 DBG] Server 55b4c516-d5a9-4516-be31-29b9a4e83f1c heartbeat successfully sent
[09:03:50 DBG] Server 55b4c516-d5a9-4516-be31-29b9a4e83f1c heartbeat successfully sent
[09:04:20 DBG] Server 55b4c516-d5a9-4516-be31-29b9a4e83f1c heartbeat successfully sent
[09:04:50 DBG] Server 55b4c516-d5a9-4516-be31-29b9a4e83f1c heartbeat successfully sent
[09:05:20 DBG] Server 55b4c516-d5a9-4516-be31-29b9a4e83f1c heartbeat successfully sent
[09:05:50 DBG] Server 55b4c516-d5a9-4516-be31-29b9a4e83f1c heartbeat successfully sent
[09:06:20 DBG] Aggregating records in 'Counter' table...
[09:06:20 DBG] Server 55b4c516-d5a9-4516-be31-29b9a4e83f1c heartbeat successfully sent
[09:06:50 DBG] Server 55b4c516-d5a9-4516-be31-29b9a4e83f1c heartbeat successfully sent
[09:07:20 DBG] Server 55b4c516-d5a9-4516-be31-29b9a4e83f1c heartbeat successfully sent

Hangfire.Server表数据 #

Id  Data    LastHeartbeat
55b4c516-d5a9-4516-be31-29b9a4e83f1c

{"$type":"Hangfire.SqlServer.Entities.ServerData, Hangfire.SqlServer","WorkerCount":5,"Queues":{"$type":"System.String[], System.Private.CoreLib","$values":["default","default","notdefault","background","export","provisioning"]},"StartedAt":"2023-03-08T08:01:20.073434Z"} 

2023-03-08 08:24:51.687

c7518906-d440-4113-85ad-0c1a97e01acc    

{"$type":"Hangfire.SqlServer.Entities.ServerData, Hangfire.SqlServer","WorkerCount":5,"Queues":{"$type":"System.String[], System.Private.CoreLib","$values":["default","default","notdefault","background","export","provisioning"]},"StartedAt":"2023-03-08T08:01:32.2476057Z"}    

2023-03-08 08:24:32.387

排查建议 #

• 检查是否在Program.cs/Startup.cs之外的地方重复调用了AddHangfireServer，比如在某个模块、中间件或者依赖注入扩展方法中二次注册
• 确认UseDatabase方法内部是否自动创建了服务器实例：部分自定义存储扩展可能会隐式启动Hangfire Server，导致和显式调用的AddHangfireServer重复
• 检查应用启动流程：是否存在两次构建WebApplication实例的情况，比如在启动前做测试或者预初始化时重复构建
• 启用Hangfire的最详细日志级别（Trace），查看是否有隐藏的服务器启动记录，确认第二个实例的启动触发点
• 检查是否使用了热重载功能，热重载可能会导致应用域重启，进而创建新的服务器实例（但通常旧实例会被标记为离线）
• 验证配置绑定是否存在异常：检查HangfireSettings:Server配置是否被多次读取，导致重复注册

内容的提问来源于stack exchange，提问作者user1264137