# Metrics Are Not Enough: Adding Searchable History to BullMQ A while back I wrote [*Queue Depth Is Not Enough: Adding Workload Metrics to BullMQ*](https://shankproof.dev/queue-depth-is-not-enough-adding-workload-metrics-to-bullmq). The argument was simple: knowing how many jobs are waiting, active, failed, and completed tells you about *backlog*, but not about *behavior*. So I added workload metrics to [bull-der-dash](https://github.com/kofno/bullderdash) — completion counts, duration percentiles, and failure ratios by queue and job name — by reading BullMQ's event streams incrementally instead of scanning retained jobs. That post ended on a principle I still believe: > The easiest metric to compute is not always the safest metric to collect. This is the follow-up, because workload metrics turned out to not be enough either. ## The Questions Prometheus Can't Answer Aggregate metrics are wonderful. They are also, by design, lossy. Prometheus can tell me that the `send-device-action` job failed 14 times in the last five minutes. It cannot tell me *which* 14 jobs, *what* they contained, or *why* they failed. It can show me a p99 duration climbing. It cannot hand me the single slow job so I can read its payload. Once you are running real workloads, the questions get specific in a way that time-series data simply cannot hold: * Show me *that* failed job. What was in it? What was the error? * This job errored, retried, and eventually succeeded — can I still find it? * A flow kicked off a cascade of child jobs. What actually happened across the whole tree? * Which exact job names ran last night? Not the top 100 — all of them. That last one stung the most, because it was a limitation I had shipped on purpose. ## The `__other__` Debt In the first version, adding a `name` label to metrics was useful but dangerous. Prometheus cardinality explodes if a label carries unbounded values, so the collector caps the number of distinct job names per queue. Everything past the cap gets bucketed under a single catch-all: ```text bullmq_jobs_finished_total{queue, name="__other__", result} ``` That guardrail was correct. It was also a slow leak of truth. On a busy production queue, `__other__` filled up fast, and the exact tail of job names — often the interesting part during an incident — disappeared into an aggregate I could never expand. I did not want to remove the cardinality cap. Removing it is how you turn a monitoring tool into a Prometheus outage. I wanted a *second* source of truth that was allowed to be exact, because it lived somewhere that didn't punish cardinality. That somewhere is a local database. ## The Part Where I Argue With My Past Self If you read the first post, you know the problem. I spent several paragraphs explaining why scanning retained jobs in Redis is the wrong shape for a production monitoring path: ```text bad model: metric cost ~= retained jobs better model: metric cost ~= newly finished jobs ``` And now here I am, adding *persistent history*. Record-level history. The exact thing that sounds like "scan all the jobs and keep them." So the design question was not "should I add history?" It was: > Can I add durable, searchable, record-level history without making Redis pay for it? The answer is yes — because the history store is not fed by scanning Redis. It is fed by the event stream the collector was *already* reading. ## One Event Stream, Two Consumers The workload collector already watches BullMQ's per-queue event streams and reacts to terminal (`completed` / `failed`) events. For each one, it reads only the fields it needs from the job hash: ```redis HMGET bull:{queue}:{jobId} name processedOn finishedOn ... ``` In the first version, that data flowed one way: into in-memory Prometheus counters and histograms. In v0.2.0, the same terminal event fans out to a second consumer. The collector hands a record to an embedded SQLite store: ```go type JobStore interface { Enqueue(store.Record) } ``` The Redis cost does not change. It is still proportional to *newly finished jobs*, not to retained history. Redis never gets scanned for history, and `/metrics` never touches the database. The history lives on local disk, and querying it is somebody else's budget — the database's, not the queue's. Same stream. Two sinks. One cheap, aggregate, and in-memory; one durable, exact, and on disk. ## Writing History Without Blocking Anything `Enqueue` is deliberately non-blocking. The collector loop must never stall waiting on a disk write, so writes go through a buffered channel drained by a background writer that batches them into SQLite transactions: ```text STORE_WRITE_BUFFER = 4096 # in-memory channel depth STORE_BATCH_SIZE = 256 # max records per write transaction STORE_FLUSH_MILLIS = 500 # max time a partial batch waits ``` If the buffer ever fills — a write burst outrunning the disk — records are *dropped and counted*, never queued into backpressure that could reach the collector. A monitoring tool is allowed to lose a little telemetry under extreme load. It is not allowed to become the bottleneck. That was the whole thesis last time, and it still holds. In production this costs almost nothing. The store flushes batches in well under 25ms, and the whole process sits around **8 MiB RSS** — the same low-footprint Go story from the first post, now with a database attached. ## What Actually Gets Stored (and What Doesn't) The store only persists **terminal** events: `completed` and `failed`. Transient states — `active`, `waiting`, `delayed` — are never written. That keeps the live queue completely out of the history write path, and it means the store is a settled, append-mostly record of work that finished. There's a subtlety that aggregate metrics can't express but a record store can. BullMQ emits a `failed` event only once a job has exhausted its retries. A job that threw on attempt one but succeeded on attempt two is, correctly, `Completed` — yet it still carried an error and burned an extra attempt. The store keeps both the final state *and* the last error and true attempt count, so you can ask a question Prometheus can't: ```text GET /v1/search?errored=true ``` > `errored=true` means "last error present **or** more than one attempt" — the retried, troubled, eventually-fine jobs that a pure success/failure count renders invisible. ## Flows Get a Trace and a Depth — For Free BullMQ's `FlowProducer` builds parent/child job trees. When a terminal event belongs to a flow, the collector resolves two things by walking the job's native parent chain in Redis: * a **trace id** — the id of the flow root * a **depth** — how far down the tree this job sits No payload convention is required. There's an optional `STORE_TRACE_KEYS` override if you already stamp a correlation id into job data, but by default the lineage comes straight from BullMQ's own flow links. The walk is bounded (a cycle or a pathologically deep flow can never stall the loop) and memoized, so flow-heavy workloads don't re-pay the cost: ```text trace = flow-root job id depth = parent-chain distance from the root ``` The result: in the console, legitimate trace ids with real reported depth line up exactly with flow jobs, and you can pull an entire cascade back out of history by its trace. ## The Console All of this surfaces in a new full-text search console at `/console`, available whenever the store is enabled. It is backed by a small JSON API that is safe to call directly: ```text GET /v1/search?q=&name=&state=&trace_id=&errored=&since_ms=&limit= GET /v1/jobs/{id} GET /v1/stats/job-names?queue=&state=&since_ms=&limit= ``` `state` only matches the settled states the store actually records (`Completed` / `Failed`), and `since_ms` is a standalone "finished within" window. Reads are concurrency-limited (`STORE_READ_CONCURRENCY`, default 16) and shed load with a `503` rather than letting search traffic swamp the writer — the same "protect the hot path" instinct, applied to the read side. ## Paying Off the `__other__` Debt That `/v1/stats/job-names` endpoint is the exact fix for the cardinality compromise. It reads the **raw, uncapped** job name recorded for every job, grouped and counted in SQL: ```text [ { "name": "send-device-action", "queue": "actions", "state": "Completed", "count": 4127 }, { "name": "reconcile-room", "queue": "sync", "state": "Completed", "count": 318 }, ... ] ``` No cap. No `__other__`. Ever. The Prometheus metric keeps its cap because cardinality safety is non-negotiable on the time-series path — but the dashboard's "Top job names" table now reads from history instead, so it reports the real distribution. In Grafana that's a single [Infinity datasource](https://github.com/yesoreyeram/grafana-infinity-datasource) panel pointed at the endpoint, scoped to the dashboard's time range. Two sources of truth, each honest about what it's for: Prometheus for cheap aggregate trends, SQLite for exact record-level answers. ## Keeping It Bounded A database that only grows is a future outage. History is swept on a tiered TTL so it reaches a steady state instead of filling the disk: ```text STORE_COMPLETED_TTL_HOURS = 24 # completed history kept 1 day STORE_FAILED_TTL_HOURS = 336 # failed history kept 14 days STORE_SWEEP_SECONDS = 300 # sweep runs every 5 minutes STORE_MAX_ROWS = 0 # optional hard row cap, newest kept ``` Failures live far longer than successes, because a two-week-old failure is still worth reading and a two-week-old success almost never is. There's an optional absolute row cap for environments where you'd rather bound by size than by time. A sweep counter (`bullderdash_store_sweep_deleted_total`) makes it observable that retention is actually doing its job. ## The Architecture Rule, Refined Once More Last time I drew a line between *stateless HTTP serving* and *ephemeral state for telemetry collection*. v0.2.0 moves that line again, carefully: * The request path stays cheap. The dashboard, job inspection, health checks, and `/metrics` never touch the history database. * The collector stays incremental. Redis cost tracks newly finished jobs, not retained ones. * The new thing — **durable, bounded, local history** — is written asynchronously, off the hot path, and is allowed to drop under pressure. Because SQLite is a single-writer database, the deployment runs a single instance with the volume mounted `ReadWriteOnce` and a `Recreate` rollout strategy, so the disk detaches cleanly on every upgrade. The same single-replica honesty from the first post: no leader election, no distributed coordination, because the simplest correct design is one writer. Running multiple replicas would mean double-reading events and double-counting — still a good future problem, still not one I needed to solve on day one. ## Store Metrics, Because Of Course The store exports its own Prometheus metrics, so the thing watching your queues is itself watchable: ```text bullderdash_store_writes_total bullderdash_store_writes_dropped_total bullderdash_store_write_errors_total bullderdash_store_write_batch_duration_seconds bullderdash_store_query_duration_seconds{operation} bullderdash_store_sweep_deleted_total ``` If writes start dropping, you see it. If sweeps stop deleting, you see it. If query latency climbs, you see it. The monitoring tool does not get a pass on being monitored. ## Rolling It Out Like last time, the rollout was intentionally boring. Everything is behind a flag: ```text STORE_ENABLED=true ``` Same binary, store disabled by default, enabled per environment through the Helm chart alongside a persistent volume. The sequence was the familiar one: enable in a local simulator, confirm writes and sweeps, deploy to demo, build the Grafana panels, promote to production. Incremental, visible, easy to back out. ## What I Can See Now Before, I could see how my queues *behaved*. Now I can also: * search completed and failed history by text, name, state, or time window * pull a specific job's full payload and error back out of history * find retried-but-eventually-fine jobs that aggregate counts hide * reconstruct an entire flow by its trace id, with real depth * read the exact distribution of job names, with no `__other__` bucket * watch the history store's own write, drop, and sweep health Queue depth told me about backlog. Workload metrics told me about behavior. History tells me about *specifics* — the one job, the one flow, the one failure I actually need to look at. ## Project Links * [bull-der-dash on GitHub](https://github.com/kofno/bullderdash) * [Helm chart and deployment docs](https://github.com/kofno/bullderdash#helm-install-ghcr) * [Workload metrics usage examples](https://github.com/kofno/bullderdash#workload-metrics) If you run BullMQ and want lightweight visibility — queue depth, job inspection, Prometheus workload metrics, and now durable, searchable history — I'd love issues and feedback. ## Closing Thought The lesson from the first post was that the easiest metric to compute is not always the safest one to collect. The lesson from this one is its mirror image: > The data you most want to keep is exactly the data most likely to hurt you if you keep it carelessly. Durable history *sounds* like the thing I warned against. It only became safe once I refused to pay for it in the one place that mattered — the live queue. Persist the events you already read, write them off the hot path, bound their lifetime, and let them drop before they ever push back. Observability should make production clearer. It still should not make Redis sad.