HAProxy/任务等待队列

用 EBtree 索引任务期限,并延后不必要的重排

HAProxy 把任务等待节点嵌入任务对象,用 EBtree 按期限组织;延长期限时可以保留旧树位置,到扫描时再校正。

fcba844c5调度与资源分配核验 2026-10-12
相关概念

工程问题#

网络任务可能频繁刷新超时,但很多任务会在真正超时前完成。每次刷新都先删除再插入树,会为大量最终不会到期的任务反复维护索引。

实际实现#

task_queue 仅在未入队或新期限早于索引键时立即重排。__task_queue 将 task->wq 按 expire 插入线程的 EBtree。wake_expired_tasks 从环绕时间范围内的候选开始,检查真实 expire:确实到期则唤醒;只是旧键过期则移除并按新期限重排。树位置允许暂时偏早,但不会仅凭旧键认定任务超时。

处理流程#

  1. 比较真实期限与已登记的树键,决定是否需要立即重排
  2. 把任务内嵌的 wq 节点按期限加入 EBtree
  3. 扫描时用真实 expire 判断是否到期,并处理 32 位时间环绕
  4. 唤醒到期任务;对延后或禁用的期限校正或移除旧索引

源码片段#

固定版本 · 原文摘录
include/haproxy/task.hfcba844c55ebbab61a888472d5a162453e87679a · L400–L420
查看原文件
400	 * has been disabled, it's useless to leave the queue as well. We'll401	 * rely on wake_expired_tasks() to catch the node and move it to the402	 * proper place should it ever happen. Finally we only add the task403	 * to the queue if it was not there or if it was further than what404	 * we want.405	 */406	if (!tick_isset(task->expire))407		return;408409	BUG_ON(task->tid >= 0 && task->tid != tid);410411	if (!task_in_wq(task) || tick_is_lt(task->expire, task->wq.key)) {412		if (likely(caller)) {413			caller = HA_ATOMIC_XCHG(&task->caller, caller);414			BUG_ON((ulong)caller & 1);415#ifdef DEBUG_TASK416			HA_ATOMIC_STORE(&task->debug.prev_caller, caller);417#endif418		}419		__task_queue(task);420	}
_task_queue

延期可保留旧位置;未登记或期限提前时才进入重排分支。

src/task.cfcba844c55ebbab61a888472d5a162453e87679a · L327–L339
查看原文件
327	if (likely(task_in_wq(task)))328		__task_unlink_wq(task);329330	/* the task is not in the queue now */331	task->wq.key = task->expire;332#ifdef DEBUG_CHECK_INVALID_EXPIRATION_DATES333	if (tick_is_lt(task->wq.key, now_ms))334		/* we're queuing too far away or in the past (most likely) */335		return;336#endif337338	eb32_insert(&th_ctx->timers, &task->wq);339}
__task_queue

移除已有等待节点,把真实 expire 写为键,再插入当前线程的 EBtree。

include/import/eb32tree.hfcba844c55ebbab61a888472d5a162453e87679a · L277–L298
查看原文件
277		old_node_bit = old->node.bit;278279		/* Stop going down when we don't have common bits anymore. We280		 * also stop in front of a duplicates tree because it means we281		 * have to insert above.282		 */283284		if ((old_node_bit < 0) || /* we're above a duplicate tree, stop here */285		    (((new->key ^ old->key) >> old_node_bit) >= EB_NODE_BRANCHES)) {286			/* The tree did not contain the key, so we insert <new> before the node287			 * <old>, and set ->bit to designate the lowest bit position in <new>288			 * which applies to ->branches.b[].289			 */290			new->node.node_p = old->node.node_p;291			up_ptr = &old->node.node_p;292			break;293		}294295		/* walk down */296		side = (newkey >> old_node_bit) & EB_NODE_BRANCH_MASK;297		troot = side ? old->node.branches.b[1] : old->node.branches.b[0];298		root = &old->node.branches;
__eb32_insert

按公共高位决定是否继续下降,分歧处插入;重复键另有子树处理。

src/task.cfcba844c55ebbab61a888472d5a162453e87679a · L345–L365
查看原文件
345void wake_expired_tasks()346{347	struct thread_ctx * const tt = th_ctx; // thread's tasks348	int max_processed = global.tune.runqueue_depth;349	struct task *task;350	struct eb32_node *eb;351352	while (1) {353		if (max_processed-- <= 0)354			goto leave;355356		eb = eb32_lookup_ge(&tt->timers, now_ms - TIMER_LOOK_BACK);357		if (!eb) {358			/* we might have reached the end of the tree, typically because359			* <now_ms> is in the first half and we're first scanning the last360			* half. Let's loop back to the beginning of the tree now.361			*/362			eb = eb32_first(&tt->timers);363			if (likely(!eb))364				break;365		}
wake_expired_tasks

限制本轮处理量,并从回绕时间窗口定位候选,必要时回到树首。

src/task.cfcba844c55ebbab61a888472d5a162453e87679a · L385–L394
查看原文件
385		task = eb32_entry(eb, struct task, wq);386		if (tick_is_expired(task->expire, now_ms)) {387			int set_running = 0;388389			/* expired task, wake it up */390			__task_unlink_wq(task);391			/*392			 * If it's a shared task, see whether we should hand it393			 * to a less loaded thread.394			 */
wake_expired_tasks

从内嵌节点取回任务,只在真实期限已到时移除等待节点并准备唤醒。

src/task.cfcba844c55ebbab61a888472d5a162453e87679a · L432–L449
查看原文件
432			if (set_running)433				task_drop_running(task, TASK_WOKEN_TIMER);434			else435				_task_wakeup(task, TASK_WOKEN_TIMER, 0);436		}437		else if (task->expire != eb->key) {438			/* task is not expired but its key doesn't match so let's439			 * update it and skip to next apparently expired task.440			 */441			__task_unlink_wq(task);442			if (tick_isset(task->expire))443				__task_queue(task);444		}445		else {446			/* task not expired and correctly placed. It may not be eternal. */447			BUG_ON(task->expire == TICK_ETERNITY);448			break;449		}
wake_expired_tasks

到期任务进入唤醒路径;旧键失效则重新排队或移除,遇到正确的未到期项时停止。

摘录分别保留调用入口、核心机制及边界;完整上下文见各段源码链接。

收益与取舍#

工程分析

带来的收益

  • 刷新为更晚期限时可省去当次树的删除与插入
  • 任务节点嵌入对象,树维护无需为每次登记单独分配包装节点

付出的代价

  • 读索引必须重新检查真实期限,不能把最小树键直接当作确定到期的任务
  • 32 位时间比较依赖有限的有效时间跨度和专门的 tick 规则,不能当作任意整数时间线
  • 每轮扫描受 runqueue_depth 限制,并涉及线程归属与状态同步;此优化不承诺精确唤醒延迟

可比较的选择

每次刷新立即重排索引键始终等于真实期限,扫描更直接,但频繁延期增加维护操作。

最小堆加过期记录可以保留旧记录并在弹出时校验,不过需要处理重复记录数量和对象生命周期。

版本与适用范围#

固定到提交 fcba844c55eb。EBtree 属于按位组织的压缩二叉索引,不能标成红黑树或普通二叉堆。此例解释期限懒更新;并发任务转交的完整状态机不在摘录范围,实际调用仍须满足线程归属规则。

参考来源#

源码、官方文档与原始论文
01
HAProxy · _task_queue

include/haproxy/task.h · _task_queue

源码fcba844c55ebbab61a888472d5a162453e87679a
02
HAProxy · __task_queue

src/task.c · __task_queue

源码fcba844c55ebbab61a888472d5a162453e87679a
03
HAProxy · __eb32_insert

include/import/eb32tree.h · __eb32_insert

源码fcba844c55ebbab61a888472d5a162453e87679a

相关实现与比较#

4 个相关案例
nginx·geo 地址匹配release-1.26.3

nginx geo 的 CIDR 模式使用二进制 radix tree,根据 IP 地址的最长匹配前缀确定变量值。

附源码片段 · 索引与查找
19 个匹配案例