Hangfire执行任务时出现Win32Exception(258)超时问题求助

问题现象 #

使用Hangfire执行任务时，偶尔触发SQL超时异常，错误信息如下：

System.Data.SqlClient.SqlException (0x80131904): Timeout expired.  The timeout period elapsed prior to completion of the operation or the server is not responding.
 ---> System.ComponentModel.Win32Exception (258): The wait operation timed out.
   at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)
   at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose)
   at System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady)
   at System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString)
   at System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean async, Int32 timeout, Task& task, Boolean asyncWrite, SqlDataReader ds)
   at System.Data.SqlClient.SqlCommand.InternalExecuteNonQuery(TaskCompletionSource`1 completion, Boolean sendToPipe, Int32 timeout, Boolean asyncWrite, String methodName)
   at System.Data.SqlClient.SqlCommand.ExecuteNonQuery()
   at Dapper.SqlMapper.ExecuteCommand(IDbConnection cnn, CommandDefinition& command, Action`2 paramReader)
   at Dapper.SqlMapper.ExecuteImpl(IDbConnection cnn, CommandDefinition& command)
   at Hangfire.SqlServer.SqlServerDistributedLock.Acquire(IDbConnection connection, String resource, TimeSpan timeout)
   at Hangfire.SqlServer.SqlServerConnection.AcquireLock(String resource, TimeSpan timeout)
   at Hangfire.DisableConcurrentExecutionAttribute.OnPerforming(PerformingContext filterContext)
   at Hangfire.Profiling.ProfilerExtensions.InvokeAction[TInstance](InstanceAction`1 tuple)
   at Hangfire.Profiling.SlowLogProfiler.InvokeMeasured[TInstance,TResult](TInstance instance, Func`2 action, String message)
   at Hangfire.Profiling.ProfilerExtensions.InvokeMeasured[TInstance](IProfiler profiler, TInstance instance, Action`1 action, String message)
   at Hangfire.Server.BackgroundJobPerformer.InvokePerformFilter(IServerFilter filter, PerformingContext preContext, Func`1 continuation)
ClientConnectionId:cf3220e9-bbf5-40f2-8dc5-0d7836736771
Error Number:-2,State:0,Class:11

当前配置 #

任务方法配置 #

[AutomaticRetry(Attempts = 0)]
[DisableConcurrentExecution(30 * 60)]
public async System.Threading.Tasks.Task ExecuteAsync(int id, PerformContext context, CancellationToken cancellationToken)
{
    ...
}

Hangfire SQL Server配置 #

builder.Services.AddHangfire((provider, config) => config
    .SetDataCompatibilityLevel(CompatibilityLevel.Version_170)
    .UseSimpleAssemblyNameTypeSerializer()
    .UseRecommendedSerializerSettings()
    .UseSqlServerStorage(databaseConnectionStr, new SqlServerStorageOptions
    {
        CommandBatchMaxTimeout = TimeSpan.FromMinutes(5),
        CommandTimeout = TimeSpan.FromSeconds(60),
        SlidingInvisibilityTimeout = TimeSpan.FromMinutes(5),
        QueuePollInterval = TimeSpan.Zero,
        UseRecommendedIsolationLevel = true,
        DisableGlobalLocks = true,
        PrepareSchemaIfNecessary = false
    })
    .WithJobExpirationTimeout(TimeSpan.FromDays(30 * 6))
    .UseFilter(provider.GetRequiredService<SendEMailOnFailureFilter>())
    .UseConsole());

额外测试情况 #

• 任务实际在37秒左右超时失败（部分实例27秒），远低于配置的30分钟并发锁超时和60秒SQL命令超时
• 手动重新排队相同任务可正常执行
• 测试包含70秒延迟的模拟任务，DisableConcurrentExecution工作正常，无超时问题

原因分析 #

从异常栈可以明确，超时发生在获取分布式锁的SQL操作阶段，而非任务执行过程中，核心原因包括：

1. SQL服务器本身负载过高或存在锁竞争，导致获取分布式锁的SQL语句执行缓慢，触发命令超时
2. DisableConcurrentExecutionAttribute默认按任务类型生成锁资源，同类型任务会共用同一锁，若某任务长期运行或异常未释放锁，会导致后续任务等待锁时触发SQL超时
3. 部分旧版本Hangfire.SqlServer存在锁获取逻辑未正确应用CommandTimeout配置的问题

解决办法 #

1. 优化SQL命令超时与Hangfire版本 #

• 进一步提升CommandTimeout配置，给锁获取操作留足缓冲时间：

.UseSqlServerStorage(databaseConnectionStr, new SqlServerStorageOptions
{
    CommandTimeout = TimeSpan.FromSeconds(120), // 提升至120秒
    // 其他配置保持不变
})

• 升级Hangfire.SqlServer到最新稳定版本，修复潜在的锁配置不生效问题

2. 自定义细粒度分布式锁 #

默认DisableConcurrentExecution按任务类型生成锁，会导致同类型所有任务排队。可自定义属性，将任务参数（如id）纳入锁资源标识，降低锁竞争范围：

public class DisableConcurrentExecutionByIdAttribute : JobFilterAttribute, IServerFilter
{
    private readonly int _timeoutInSeconds;

    public DisableConcurrentExecutionByIdAttribute(int timeoutInSeconds)
    {
        _timeoutInSeconds = timeoutInSeconds;
    }

    public void OnPerforming(PerformingContext filterContext)
    {
        var id = filterContext.Job.Args.FirstOrDefault() as int?;
        if (!id.HasValue)
        {
            throw new InvalidOperationException("任务必须包含id参数");
        }
        // 按任务类型+id生成唯一锁资源
        var resource = $"Job:{filterContext.Job.Type.FullName}:{id.Value}";
        var timeout = TimeSpan.FromSeconds(_timeoutInSeconds);
        
        filterContext.Connection.AcquireLock(resource, timeout);
    }

    public void OnPerformed(PerformedContext filterContext)
    {
        // Hangfire会自动释放锁，无需手动处理
    }
}

替换原属性使用：

[AutomaticRetry(Attempts = 0)]
[DisableConcurrentExecutionById(30 * 60)]
public async Task ExecuteAsync(int id, PerformContext context, CancellationToken cancellationToken)
{
    ...
}

3. 排查数据库锁与性能问题 #

• 检查Hangfire的[HangFire].[Lock]表，清理长期未释放的无效锁记录
• 查看SQL服务器等待统计，排查是否存在锁等待、IO等待等性能瓶颈
• 确认[HangFire].[Lock]表的索引已正确创建（Hangfire默认会生成，可通过SQL Server Management Studio验证）

4. 临时恢复重试策略 #

暂时恢复任务重试，避免单次超时直接导致任务失败：

[AutomaticRetry(Attempts = 3, DelaysInSeconds = new[] { 10, 30, 60 })]
[DisableConcurrentExecution(30 * 60)]

内容的提问来源于stack exchange，提问作者roli09