# Flows, Finally: Bringing BullMQ Parity to AnvilMQ

I started AnvilMQ because I wanted a queue engine I could run locally, inside a regional Kubernetes cluster, without standing up Redis next to every workload.

It is a small thing by design. One Rust binary. An embedded libSQL (SQLite + WAL) database running in-process. gRPC in front, Prometheus metrics out the side. No external broker, no separate datastore to babysit.

For a while, that smallness came with a catch: AnvilMQ could not do everything BullMQ could. And the workloads I actually care about lean on BullMQ features I had been quietly working around.

So the goal for this stretch was simple to say and large to build:

> Reach functional parity with the BullMQ features a cron-and-bulk workload actually depends on — and then add flows.

Both are done now. This is what shipped.

## Why Parity, Not Everything

BullMQ is a big surface. Chasing all of it would have been a great way to never finish.

So I did not define parity as "every BullMQ feature." I defined it as "the features a representative cron-and-bulk workload reaches for." That gave me a short, honest list instead of an open-ended one.

- timezone-aware cron schedulers
- bulk enqueue
- idempotent / singleton enqueue
- non-retryable failures
- backoff jitter
- job result storage
- flows

Everything above the last line is the unglamorous middle of the list. It landed first, because flows needed some of it as a foundation.

## The Supporting Cast

None of these are headline features on their own. Together they are the difference between "a queue" and "the queue my app already assumes."

**Timezone-aware cron schedulers.** Declarative schedulers that reconcile, backed by a background producer loop. You describe the schedule and the timezone; the broker fires the jobs. Cron-heavy workloads lean on this more than anything else, so it was the critical path.

**Bulk enqueue.** `AddJobBulk` submits a batch of jobs in one atomic call instead of N round trips. Fan-out stops being a loop of RPCs.

**Release-on-finish dedup.** Idempotency-key receipts you can release, look up, and delete by key. That is what gives you singleton enqueue — "only one of these in flight" — without bolting coordination on top.

**Non-retryable failures.** Some failures are never going to succeed on retry. Now you can say so, and the broker stops wasting attempts on them.

**Backoff jitter.** Randomized retry backoff, so a wave of failures doesn't retry in lockstep and stampede whatever just recovered.

**Job result storage.** The broker now retains each job's return value. Useful on its own for debugging, and — not coincidentally — the foundation flows needed to let a parent read its children's results.

That last point is the segue.

## The Headline: Flows

Flows are parent jobs that wait for their children. It is the single most invasive feature I have added, and the one that most changes what AnvilMQ can model. It is the same shape BullMQ exposes through `FlowProducer`.

`AddFlow` creates one parent/child dependency tree, atomically, in a single transaction. Leaf nodes land as immediately claimable work. Internal (parent) nodes park in a new state: `WaitingChildren`.

The important part is the gate.

> A parent in `WaitingChildren` is never handed to a worker until every one of its children reaches a terminal state.

That gate lives in dequeue, not in some external coordinator. Each child completion decrements the parent's pending-children count, and when it hits zero the parent becomes runnable on its own. No polling, no sweeper job, no client-side orchestration.

A few more pieces make it actually useful:

- `GetChildrenValues` lets a parent read the return values of its completed children. This is built directly on the job result storage above — results are stored for *all* jobs, not just flow children, so this was mostly wiring once results existed.
- `failParentOnFailure` propagates a child's terminal failure up through ancestors that opted in, while siblings keep running and ancestors that did not opt in are left alone. That matches BullMQ's behavior rather than inventing my own.

## Bulk Trees and Merging Into Existing Ones

Two operations extend flows past the single-tree case.

`AddFlowBulk` builds many independent trees in one atomic, all-or-nothing transaction. Either every tree in the batch is created, or none is.

`AddFlowMerge` is the one I went back and forth on. It attaches new child subtrees under an *existing* flow node — located by job name plus idempotency key — and re-gates that parent so it waits on the newly attached work too.

The design question was whether "re-send a flow and have it attach" should be implicit. I decided it should not. `AddFlow` still rejects a reused idempotency key, because its job is to create fresh trees. Merging is a separate, explicit RPC. Surprising people by silently mutating an existing tree is exactly the kind of convenience I did not want.

## Finite Trees, On Purpose

A dependency tree is a great way to accidentally build something enormous. So every tree is bounded, and the bounds are checked atomically — an oversized tree is rejected with nothing written, rather than half-created and then unwound.

| Knob | Default | Bounds |
| --- | --- | --- |
| `ANVILMQ_MAX_FLOW_DEPTH` | 10 | Tree height |
| `ANVILMQ_MAX_FLOW_NODES` | 1000 | Node count per tree |
| `ANVILMQ_MAX_FLOW_BULK` | 100 | Trees per bulk call |

Because a flow tree is created in one atomic action and is finite, every node can share a single tree depth instead of accumulating ancestry per node. That turned out to be a cleaner model than the per-node lineage checks I had been carrying for runaway-chain protection — flows quietly subsumed that older safety work.

And because I never regret adding observability early, flows ship with their own metrics: `anvilmq_flows_created_total`, `anvilmq_flow_nodes_created_total`, `anvilmq_flow_waiting_children`, `anvilmq_flow_parent_failures_total`, `anvilmq_flows_rejected_total`, `anvilmq_flow_merges_total`, and `anvilmq_flow_merge_nodes_total`.

## Did I Just Make It Slower?

This is the question I always distrust myself on.

Flows added columns to the jobs table, a new job state, a new index, and a handful of metric series — all on the hot enqueue/claim/complete path. It would be easy to tell myself "it's probably fine" and move on.

So instead I re-ran the isolated arrival-rate benchmark against the build that carries all of it: 4 producers, 4 workers, `synchronous=NORMAL`, 1 KiB payloads, zero handler work, a clean database per repetition, three repetitions per rate.

| Target rate | Completion/s (window) | Backlog slope | End-to-end p99 | Sustained |
| --- | --: | --: | --: | :-: |
| 250 jobs/s | ~249.9 | ~0 | ~40 ms | 3/3 |
| 350 jobs/s | ~349.9 | ~0 | ~37 ms | 3/3 |
| 400 jobs/s | ~399.8 | ~0 | ~158 ms | 3/3 |

Completion tracked arrival at every rate, backlog stayed flat, end-to-end latency stayed well under a second. No regression on the hot path from the flow work. If anything, 400 jobs/s held comfortably where an earlier baseline had tipped into congestion collapse — though I am crediting that to a quieter host and a newer Docker, not to my code being magic.

> These are local, exploratory numbers on shared developer hardware, not production SLOs. Treat them as directional and re-run on your own hardware before making any sizing decision.

That caveat is not false modesty. The dominant cost in this engine is a single embedded writer with per-commit durability, and that moves a lot with real handler work, larger payloads, and different storage.

## Where This Leaves AnvilMQ

AnvilMQ is still the small thing I wanted: one Rust binary, an embedded transactional database, gRPC ingress, Prometheus metrics. It just does a lot more of what my workloads already assume — schedulers, bulk enqueue, idempotent enqueue, and now real parent/child flows.

Parity was never the end goal. It was the point where AnvilMQ stops being "the interesting experiment" and starts being "the thing I could actually run."