DefraDB.rs: Seven Months Later

In February I wrote about porting DefraDB from Go to Rust in a month, using the Go test suite as an automated specification. 730 commits, 196,000 lines, 96% test parity. In that post, I made a handful of claims about how high-velocity, AI-driven coding should work.

Seven months later, the August 24 snapshot offers enough history to check those principles: 755 pull requests from nine contributors, an agent-assisted security audit and verification, and a handoff to the long-term maintainers.

Meanwhile I was building gents, an agent runtime that uses DefraDB as its control plane. Fleets, machine pairing, peer discovery, and replication across hosts put new pressure on the database and drove development on the defradb.rs side.

Rust lines are physical lines in tracked files under crates/. Crates are direct packages under that directory. Test counts include standard, Tokio, and wasm test attributes. August data ends at the August 24 commit snapshot.

Expand and contract#

In my earlier post I described a rhythm I called expand and contract.

• Expand: let the LLMs run and generate new features. Accept a feature when it works; perfection comes later.
• Contract: audit ownership and interfaces, split large files to make the code easier to read, and remove cruft, excessive comments, dead code, and abandoned ideas.

The commit log shows the shift. January is the only month when feature-prefixed commits outnumber fixes; by April the feat:fix ratio fell from 1.85 to 0.12, a 15x inversion. Commit prefixes are imperfect, but the work clearly changed. In January I was generating a database wholesale. By spring I was fixing it.

The repo’s structure held up: storage, CRDTs, query, networking, and their interfaces were in roughly the right places. Plenty inside those boundaries was wrong. I accepted that trade because structural mistakes compound; behavioral failures are easier to isolate when a test, reference implementation, or other oracle exposes them.

Contraction reached the crate graph too. After peaking at 41 in June, the count returned to 32 when we folded the six db-* crates into db, three query crates into query, and five benchmark packages into one harness. The subsystem boundaries survived; the graph around them had expanded past its usefulness.

As the crate count fell, recognized Rust test attributes kept rising. We revised the structure without losing the accumulated evidence of correctness.

Repeated contraction also left a measurable file distribution. Across the 1,505 Rust files at the cutoff:

median   185 lines
p90      562
p99    1,301
max    3,966
files over 2,000 lines:  4
files over 5,000 lines:  0

That distribution emerged from the contraction passes; nobody enforced a file-size standard.

Requirements from gents#

Most of the next seven months went into that repair queue, with gents setting the order.

One agent calls another by writing an AgentRequest document that replicates over P2P. No package test described a fleet of those agents running across my hardware. Operating one exposed failures at scale and gave me a reason to chase them.

I reproduced the worst failure with 19 empty database nodes in a P2P mesh. Activity hockey-sticked until the host ran out of resources. Three failures compounded: the retry ledger missed collection-commit document IDs, symmetric peers dropped each other’s gossip, and capacity rejection synchronized retries across the mesh. Stabilizing the fleet required the Regolith storage backend, performance and P2P work in defradb.rs, and less write load from gents. That got the fleet stable.

Tightening the definition of ‘conformance’#

February treated DefraDB’s 210,000-line Go test suite as a machine-readable definition of correct. Running it against Rust through FFI made comparison mechanical: same test, inputs, and expected outputs; different implementation.

That held. We added two stricter layers.

Tier one — ported suite. Go’s tests compile against the Rust shared library and still get re-run against upstream.

Tier two — differential execution. The same commands run against both binaries and their results are compared. Its twelve main areas contained 493 cross-binary tests across 190 Rust files and 50,292 physical lines. The snapshot still recorded failures and ignored cases, especially around iroh.

Tier three — live peer. Go and Rust nodes talk over libp2p and iroh: replication, KMS interop, gossipsub. This exposed failures the first two tiers could not, including fix(p2p): keep one connection per peer to stop gossipsub dying against Go. The bug only exists when both implementations are in a room together. This is the layer gents depends on: nodes staying connected and agreeing over time.

Go kept moving, so parity work continued. By August the specification was something the implementations did to each other continuously, rather than a suite we had copied once.

The audit#

Conformance measured agreement between Rust and Go. Safety required a different test. Before 1.0 I ran an agent-assisted security audit across seven workstreams:

It found a hand-rolled DER parser, Merkle verification that did not fail closed on crypto errors, and ingestion paths that skipped CID verification or collection-level access checks. A green conformance suite had never asked those questions.

The 303 exclude summaries and triage documents but include verified-safe observations. The triage summary classified roughly 35 as 1.0 blockers. Remediation was followed by a verification pass.

Then @fredcarle, @iverc, and @JesseAbram reviewed the code. Their concentrated push landed April 15–22. The commits are blunt: delete merkle_proof module, per-peer Bitswap ACP filter, derive secp256k1 peer DIDs via crypto crate.

April has the lowest feat:fix ratio in the project’s history — 14 feature-prefixed commits against 114 fixes. The human review is the dip in the middle of the contraction chart.

When Go had no answer#

The February post states a precondition plainly:

This worked because DefraDB’s Go codebase is well-tested… It would not have worked on a codebase with poor test coverage or unclear interfaces.

That precondition was too narrow. The project needed some way to define correctness; for the ported surface, most of it happened to be written already.

We hit the limit first with iroh, roughly 8,400 lines of Rust-only P2P transport. gents needed fleet replication over a transport the Go stack had never provided, leaving the conformance infrastructure nothing to compare against.

You can see what that cost in the shape of the differential suite:

P2P accounts for 296 of 493 tests, and p2p_iroh alone has more than every non-P2P area combined. That is the cost of specifying the Rust-only transport ourselves.

CRDT semantics exposed a different gap.

Go’s tests record behavior case by case without stating the rule behind it. That mattered little when a query either returned the right rows or did not. CRDT semantics were different: Rust and Go disagreed under untested interleavings, while the intended rule lived only in the shape of Go’s code.

On June 2 the repo gained 88 TLA+ models with 131 configurations and 21 Lean files, mostly covering CRDT semantics, convergence, materialization, and replication failures.

We encoded broken designs as red models and the real designs as green ones. The checker must reject red and accept green. TwoStoreCounter.tla, for example, compares a buggy split state with Go’s single-key counter.

Then, while transcribing Go’s merge queue, we found a red state in the reference implementation:

FailMode = "Closed"  - real Rust: exhausted retries -> Err -> NOT marked done   [GREEN]
         = "Open"    - Go merge.go bug: exhausted retries -> return nil ->
                       caller treats block as done -> silent drop               [RED]

On the Go path, a merge that exhausts its retries returns nil. The caller reads that as success, marks the block merged, and prevents another delivery. Under the wrong failure conditions, the reference can silently lose data.

Rust already returned an error and left the CID eligible for another delivery, a choice made for unrelated reasons. The model showed it was also the fix. MergeQueue_DESIGN.md records the result: the Go path violates INV_NoSilentDrop.

After seven months of treating Go as the specification, making its rules explicit found a bug in it.

I expected formal methods to help where Go ran out, as it had for iroh. Their greater value was finishing a specification rich in examples and quiet about meaning. Tests covered written cases; models stated the rules behind them and gave us enough distance to find a bug in the reference.

proofs/verify-all.sh checks the TLA+ models, builds the Lean proofs, and runs contract and conformance tests against Rust types or a release binary. It makes some kinds of drift fail like any other broken test; a refinement proof would be stronger.

The tools stayed put#

February’s other claim was that you have to build your own instruments:

Claude can generate a module, but it can’t tell me which of 2,839 tests are failing or which worktree has stale results when running all the tests takes 5 hours.

ffi-test remains, joined by the differential harness, lens host, pg-compat harness, and verify-all.sh. Each turns a question about the system into an executable check.

Claude Code did February’s work. Since then Codex, Grok, Gemini, and newer Claude models have rotated through. They could be swapped mid-flight because the tools survived.

For the last stretch gents, running GLM-5.2, took on dependency cleanup, security upgrades, and build scoping. The runtime that forced DefraDB’s P2P layer into production is now committing back to the database underneath it.

Handing it to people#

February had exactly one contributor.

By August 24, I was a minority contributor to a codebase I generated: 117 of 370 commits. @vertexclique and @edjroz landed 199 together, and August had the highest volume since February. Commit counts are rough, but the handoff is clear.

The February post argued that AI makes judgment portable. I meant across languages: my distributed systems experience transferred to Rust. By August, the tests and tools were carrying that judgment between people too.

February left the FFI dashboard at 96%. By August I was no longer the person closing the gaps it exposed.

@edjroz made his first commit on August 5. By the cutoff he was closing CLI flag parity, Go-compatible mount paths, and restart ownership invariants. The differential suite made every “not yet” concrete.

Contraction had accumulated small files, 5,121 test attributes, three conformance levels, an audit trail, and formal models for hidden invariants. We built them to keep moving; together they made the project possible to hand off.

The handoff separates application ownership from database maintenance. gents can keep pushing DefraDB without making DefraDB’s future depend on the person building on top of it.

Seven months later#

The February post framed expand and contract as a rhythm for refactoring AI-generated code. Seven months later, I would add one condition: expansion at this speed only works when failures have somewhere reliable to appear. The Go test suite gave us that at first. When it ran out, we added differential tests, live peers, an audit, and formal models; gents supplied the production failures and priorities.

I also called the last 10% a grind. I missed what defining correctness was producing: a codebase that no longer depended on the person who opened the port.

The 31 days got the shape right. The seven months made it true. I had underestimated how much of the work came after the port could already pass for finished.