TPL Dataflow实现消费者动态订阅/取消订阅分类消息的优化方案
解决方案
核心优化思路
首先纠正分发逻辑的设计:抛弃Publisher遍历所有消费者匹配分类的实现,改为在Publisher中维护 ConcurrentDictionary<int, List<Consumer>> 类型的分类-订阅者映射表,收到消息后直接通过分类ID取对应订阅者列表投递,分发效率从O(消费者数量)降到O(1),完全满足你当前的性能约束。
对应问题的修复方案
- 取消订阅后队列残留无效消息的问题
消费者在执行业务处理逻辑前,先校验当前消息的分类是否仍在自身订阅集合中,若已取消订阅直接跳过处理即可,单次校验为O(1)操作,几乎无性能损耗。 - 集合修改导致的枚举异常问题
- 所有订阅/取消订阅操作统一通过Publisher提供的接口执行,修改映射表和消费者订阅集合时加锁,或直接使用线程安全集合
- Publisher投递消息时,先对订阅者列表拍快照再遍历,避免遍历过程中集合变化
- 消费者自身的订阅集合所有读写操作都加锁,禁止边遍历边修改
- 分发逻辑的合理性问题
优化后的映射表分发逻辑完全合理,避免了全量广播给所有消费者的性能浪费,也避免了遍历所有消费者的开销,是进程内消息分发的常规实现。
优化后代码示例
public class Message { public Message(int categoryId, string data) { CategoryId = categoryId; Data = data; } public int CategoryId { get; } public string Data { get; } } public class Publisher { // 分类ID -> 订阅该分类的消费者列表,线程安全字典 private readonly ConcurrentDictionary<int, List<Consumer>> _categorySubscribers = new(); // 全局锁,用于修改订阅关系 private readonly object _lockObj = new(); public void Subscribe(Consumer consumer, int categoryId) { lock (_lockObj) { if (!_categorySubscribers.ContainsKey(categoryId)) { _categorySubscribers[categoryId] = new List<Consumer>(); } if (!_categorySubscribers[categoryId].Contains(consumer)) { _categorySubscribers[categoryId].Add(consumer); consumer.AddSubscribeCategory(categoryId); } } } public void Unsubscribe(Consumer consumer, int categoryId) { lock (_lockObj) { if (_categorySubscribers.TryGetValue(categoryId, out var subscribers)) { subscribers.Remove(consumer); consumer.RemoveSubscribeCategory(categoryId); if (subscribers.Count == 0) { _categorySubscribers.TryRemove(categoryId, out _); } } } } public void Publish(Message message) { // 取到当前分类的订阅者列表,拍快照后遍历投递 if (_categorySubscribers.TryGetValue(message.CategoryId, out var subscribers)) { List<Consumer> snapshot; lock (_lockObj) { snapshot = subscribers.ToList(); } foreach (var consumer in snapshot) { consumer.AddMessage(message); } } } } public class Consumer { private readonly HashSet<int> _categoryIds = new(); private readonly ActionBlock<Message> _queue; private readonly object _categoryLock = new(); // 注入Publisher实例用于后续取消订阅 private readonly Publisher _publisher; public Consumer(string name, Publisher publisher) { Name = name; _publisher = publisher; _queue = new ActionBlock<Message>(async m => { // 处理前先校验是否还订阅该分类 bool isSubscribed; lock (_categoryLock) { isSubscribed = _categoryIds.Contains(m.CategoryId); } if (!isSubscribed) return; await Foo(m); }, new ExecutionDataflowBlockOptions { MaxDegreeOfParallelism = 1, SingleProducerConstrained = true }); } public string Name { get; } public void AddMessage(Message message) { bool accepted = _queue.Post(message); if (!accepted) { Console.WriteLine("Message has been rejected!"); } } // 订阅/取消订阅改为内部方法,由Publisher统一调用 internal void AddSubscribeCategory(int categoryId) { lock (_categoryLock) { _categoryIds.Add(categoryId); } } internal void RemoveSubscribeCategory(int categoryId) { lock (_categoryLock) { _categoryIds.Remove(categoryId); } } private async Task Foo(Message message) { // 业务处理逻辑 await Task.Delay(10); if (message.Data.Contains("test")) { // 统一走Publisher接口取消订阅 _publisher.Unsubscribe(this, message.CategoryId); } Console.WriteLine($"[{Name}] - category id: [{message.CategoryId}] data: [{message.Data}]"); } } // 调用示例 public class Test { static void Main() { var publisher = new Publisher(); var consumer1 = new Consumer("Consumer1", publisher); publisher.Subscribe(consumer1, 1); publisher.Subscribe(consumer1, 2); var consumer2 = new Consumer("Consumer2", publisher); publisher.Subscribe(consumer2, 2); publisher.Subscribe(consumer2, 3); publisher.Subscribe(consumer2, 4); var consumer3 = new Consumer("Consumer3", publisher); publisher.Subscribe(consumer3, 3); publisher.Subscribe(consumer3, 4); var message1 = new Message(1, "message1 test"); var message2 = new Message(2, "message2"); var message3 = new Message(1, "message3"); var message4 = new Message(3, "message4 test"); var message5 = new Message(4, "message5"); var message6 = new Message(3, "message6"); var messages = new[] {message1, message2, message3, message4, message5, message6}; foreach (var message in messages) { publisher.Publish(message); } Console.ReadLine(); } }
性能说明
按照你的业务规模,500QPS的消息量,每次投递只需要一次字典查询+遍历少量订阅者(大部分分类订阅者数量不多),进程内执行完全无压力,不需要引入额外的Actor框架。如果后续消费者数量进一步上涨,还可以把分类映射表的锁改成细粒度锁,每个分类单独加锁,进一步提升并发性能。
内容的提问来源于stack exchange,提问作者chunk1ty
相关产品推荐
相关产品推荐

