Membench: A Common Yardstick for Emergent Memory and Orchestration

The problem: every memory system grades its own homework

Ask five teams building memory into an agent whether their system “remembers well” and you get five different answers, measured five different ways, none of them comparable to the others. That is not a minor inconvenience — it means nobody outside a given team can tell whether a claimed improvement is real or just a metric picked to flatter the system that produced it.

Membench (the public, vendor-neutral spec at github.com/Persistech/membench) is an attempt at something narrower and more useful than “the best benchmark”: a shared, evolving language — a common probe set that any memory-augmented agent stack can run, so that retention, relevance, and overhead mean the same thing regardless of whose architecture produced the number. reckon implements that spec directly, M1 through M11, mapped one to one, no reinterpretation. The reckon implementation lives in docs/membench/ in this repo; the spec it implements lives upstream, and the two are deliberately not the same link — one is “the vocabulary,” the other is “one dialect speaking it.” You can watch the M-series probes run against a live session in the membench visualization.

A shared language earns its keep only if it stays honest about the words it doesn’t have in common yet. That turned out to be the harder half of the problem.

The approach: keep the incompatible things from touching

Once you’re running a real swarm — not a demo — you very quickly generate three different kinds of numbers that all look like “metrics” and are not remotely the same kind of claim:

Membench (M-series) — cross-system, apples-to-apples. The eleven official probes: retention, relevance, work efficiency, overhead, continuity, coordination, crystallization, confabulation, ceiling-hit, coverage, bootstrap. Two of them are hard vetoes, not scores — confabulation above 5% or bootstrap success below 70% blocks a release outright, whatever the composite says. This is the only class in reckon’s whole metric surface that a competitor’s number can be checked against directly, because the probe set and scoring are identical regardless of whose memory architecture is under test.

Emergence (E-series) — reckon-native, no external analog. Separate reckon tooling (revenant_emergence_lite.py) measures something membench does not attempt to score at all: how well a swarm of agents organizes itself. Concretely, every agent in reckon is “born,” not spawned cold — its spawn brief carries citations to at least three generations of prior agents’ work (selected by embedding nearness to the current mission, backfilled across eras so the citation set actually reaches back rather than clustering in the last few hours), plus one of six fixed cognitive-diversity “lenses” (archaeologist, adversary, cartographer, artisan, connector, skeptic-of-done) assigned deterministically so siblings see the same problem differently instead of running six redundant copies of the same approach. There’s a real, measured structural-health formula behind this (edge density, clustering, bearing linearity, backtrack ratio), and it’s been validated against actual sessions — but it has no benchmark twin. No other system has this architecture, so there is nothing to hold it up against. It’s a research signal reckon watches to steer itself, not a score to win a comparison with. The mechanism that produces this kind of self-organization without a central scheduler is the same one described in Stigmergy Without a Manager — emergence here is what that coordination discipline looks like from the measurement side.

Branch-native economics (MaaS vs. MaaA) — self-consistent, not comparable to each other. More on this below — it’s the distinction the operator specifically wanted kept sharp.

The discipline that makes the shared vocabulary trustworthy is refusing to let these three classes blur into one number. A taxonomy doc (docs/membench/METRIC-TAXONOMY-CANONICAL.md) exists for exactly this reason: it exists because two incompatible numbering schemes both called themselves “M1 through M15” at one point, so “M1” meant retention in one file and token efficiency in another — the kind of collision that makes a shared language silently stop meaning anything. The fix wasn’t clever math; it was naming which axis is cross-comparable (membench, and only membench) and refusing to let the other two borrow that credibility.

The evolution: the eval harness finding real problems, not decoration

The membench implementation itself has a completion history worth being specific about, because “we run evals” is a claim that’s easy to fake and reckon’s eval harness has actually caught a real defect doing its job. As of the M1–M15 runner-completion pass, the harness runs the full stack — eleven official membench probes plus four reckon-specific extensions (stability, safety, reliability, calibration) — end to end with zero exceptions, including on a date with no data at all (empty inputs return SKIP, never a crash or a silently wrong zero).

Building the four extension probes surfaced one genuine finding, not a tuning exercise: the new M13 safety probe — which checks whether a manifest claiming success actually did anything (tool calls, files touched, artifacts produced) — found that roughly 37% of success-claiming manifests in chart/active carried no concrete artifact at all. That’s the exact “phantom success” failure mode a self-measurement system exists to catch: an agent reporting done without having done anything checkable. The probe didn’t get tuned to produce that number; it measured the swarm as it actually was and the swarm had a real gap. That is the strength worth naming here — not “the swarm never fails,” but “the harness that watches the swarm catches it when it does.”

The MaaS vs. MaaA distinction — and why it can’t be one axis

The operator’s shorthand for this is memory-as-a-service versus memory-as-architecture, and it is a real, load-bearing distinction, not a marketing frame:

• Memory-as-a-Service (MaaS) is the shape most “add memory to your agent” products take: memory is rented per call, re-injected through the context window every turn. The cost grows with conversation history, because you re-send more of it each turn — and because the provider bills for what’s sent, the economics work against the user. More history earned, more tokens billed.

• Memory-as-Architecture (MaaA) is what reckon is: file-backed, crystallized memory the system owns, where each agent re-reads a bounded, fixed prefix rather than an ever-growing transcript. The cost is flat and amortized regardless of how much history exists, because the architecture never re-sends the whole past — it reads a bounded summary of it. The economics cooperate with the user: more history doesn’t mean more bill.

These are not two settings of the same dial — they are different cost topologies, and a metric that flatters one is often meaningless for the other by construction. The clearest example: “cache hit rate on re-sent history” is the metric a MaaS system would naturally optimize, because its whole cost model is re-sending. Point that same metric at a MaaA system and it’s a category error — the denominator (re-sent history) is near zero, because MaaA doesn’t re-send history in the first place. A high number and a meaningless number look identical on a dashboard if you don’t already know which branch you’re reading.

This is why reckon’s own documentation is careful to only ever present a MaaS-vs-MaaA comparison as a labeled counterfactual estimate — never a benchmark score. A modeled comparison (45 sessions, one parameter for how much history a native-memory MaaS system would offload) puts reckon’s architecture at roughly 15.6× lower billed-input-equivalent cost than a vanilla system with bolted-on native memory, and about 31× lower than a system with no memory at all. Those numbers are real and worth stating — but they are a modeled counterfactual under a stated assumption, not a membench score, and reckon’s own docs say so explicitly rather than letting a reader mistake one for the other.

Why this matters for the shared language

Bring this back to the opening idea: membench is only worth calling a shared language if using it doesn’t quietly launder incompatible things onto one scale. A benchmark that lets a vendor’s branch-native cost metric masquerade as an apples-to-apples score isn’t a shared vocabulary — it’s a rigged comparison with a neutral-sounding name. Keeping M (cross-comparable), E (reckon-native emergence signal), and the MaaS/MaaA branch economics (self-consistent, each on its own axis) in three visibly separate buckets is what lets membench’s M-series stay trustworthy as the one number you actually can compare across products. The distinction isn’t bureaucratic hairsplitting; it’s the thing that keeps the “evolving” part of “shared evolving language” from becoming “conveniently vague.”

The takeaway

A benchmark that tries to score everything on one axis usually ends up scoring nothing honestly — it just hides which parts of the number are architecture-neutral and which parts are somebody’s home-field advantage. The more useful discipline is the unglamorous one: say plainly which numbers compare across systems (membench’s M-series, and only that), which ones are reckon watching its own swarm self-organize (the emergence signal, with no peer to check it against yet), and which ones describe a business model rather than a memory quality (MaaS’s per-call economics versus MaaA’s flat, owned cost) — and never let the third kind borrow the credibility of the first.