Repository navigation
[Rewrite] Umbrella: nostream 4.0 — Rust + LMDB, factory-managed #860
Description
Purpose
Umbrella issue for the complete rewrite of nostream from TypeScript/Node.js/PostgreSQL to Rust with an LMDB-based storage backend (strfry-style). Led by a factory manager agent from 0 to 100%.
Ground rules (frozen)
- Capture point:
242a3001d77149e4d70151ec03060c4ef44a1547(nostream@3.4.0, 2026-10-13T19:31:52Z). The plan absorbs every change onmainthrough 2026-10-15T00:00:00Z — measured: 24 commits between capture and tip, including nostream@3.5.0/3.5.1 (NIP-77 negentropy wire protocol, settings URL validation, security bumps). Protocol currency is judged against the NIP registry as of the capture date, then re-checked at cutover. - Out of scope: the website (
website/, the Astro marketing site) — it stays TypeScript. - Protocol compliance: the rewrite must support every NIP the relay advertises at capture (
supportedNips: 1,2,3,4,9,11,12,13,14,15,16,17,20,22,25,28,33,40,42,43,44,45,50,65,77 — 25 NIPs) plus any NIP change merged through the capture window; Phase 0 diffs the registry against this list and adds issues for the gaps. - Costs: every issue carries an Expected Cost (USD) on the project board; on completion the Actual Cost (USD) is recorded from the factory's own spans (measured baseline: job 001 — a ~165-line diff across 6 files — cost 2.2 h factory wall time).
- Hardware: the factory runs on chappie (this host: orchestration, reviews, LMDB benchmark rig); tauceti (Ada's host) takes heavy Rust release builds and long integration suites via a shared worktree per phase. The split is recorded per issue in the project's Host field.
- Oracle: the existing test suite (33,053 lines of TS tests) is the porting oracle — a Rust module is done when it reproduces the TypeScript module's observable behaviour, verified by the parity tests in Phase 6 and, where they exist, the original spec tests re-expressed in Rust.
Factory manager's mandate
- Owns the board: moves issues through Todo → Costed → In progress → In review → Done/Blocked, sets Expected/Actual Cost and Host fields.
- Monitors spend per phase against the plan; when a phase exceeds its budget by >25% or a milestone slips, adjusts the project plan, milestones and timelines on the board and records the adjustment in this issue's comments.
- Runs each issue through the factory loop (recon → spec → build → review → approve) with the six lenses; escalates per the factory's needs-human rules.
Project board
https://github.com/users/cameri/projects/4
Issue tree
See the phase issues #860–#868 (created below as sub-issues of this one).
Labels: rewrite on everything; rust, lmdb, mcp, factory by area.
Activity
chappie-daemon commentedon Oct 17, 2026
Phase 0 (job
001-nip-audit, #861) — delivered, awaiting review.rounds-exhaustedafter the review was halted at round 5. Eight round-4 findings were fixed in the escalated head and no lens reviewed that head;decision.mdin the job folder records what was and was not reviewed.One planning input from this job. The rewrite plan's premise that jobs branch from
rewrite/maindoes not hold: norewrite/mainexists in this repository (47 remote heads checked), so this job branched frommain's tip811fbb5. The PR body carries the integration note — the audit file must be carried across, or the cut re-pointed, at the moment Phase 1 code jobs start. Worth settling before the first code job, not after.Cost data suggests the per-issue estimates are conservative by roughly an order of magnitude for documentation-shaped work; no change made to the board's expected costs, since one sample is not a basis for rescaling the plan.
chappie-daemon commentedon Oct 17, 2026
Cost calibration — first measured job.
#861's Phase 0 job landed at 0.046× its planning anchor ($3.71 measured against $80 expected). The anchor was set before any job had run under this process, so it was a placeholder, not an estimate.
Method, from here on: every phase's expected cost is recalibrated from that phase's first measured job, not from a global fudge factor applied to the original anchors —
with scope stated in the units the measurement is per (sessions and requests, split by stage), and the anchor replaced rather than scaled. Phase 0's calibration input: 30 sessions, 1,362 requests, $0.124/session across recon/spec/build/review-scoping/lenses, of which lenses are 60.2%.
No board change made yet: one job calibrates Phase 0 only, and Phase 1 has no measured job. When Phase 1's first job closes, its anchor is replaced by the same formula and the replacement is recorded here. This is now standing cost duty — every completed issue is reported actual vs expected, and an anchor is only allowed to stand while its phase has no measurement.
chappie-daemon commentedon Oct 19, 2026
Phase 1 anchor replaced — first measured job.
#862 (job
005-rust-foundation) closed at actual $14.7016 against a $1,200 anchor — 1.2% of the anchor — over 15 sessions and 1,437 requests (recon 1, spec 2, build 1, review and confirmations 11). Stage split and thefactory-spansinvocation:projects/nostream/.factory/jobs/005-rust-foundation/cost.md.The rule recorded above applies unchanged:
Phase 1 input: 15 sessions, $14.7016 → $0.9801/session.
Project 4 changes, applied 2026-10-19:
nostream-coreSo, plainly: the $1,200 Phase 1 anchor is retired. The $900 and $200 figures were placeholders of the same family — neither was ever an estimate — and neither survives on the board. #862 keeps $1,200 as its expected only because that is the anchor it was actually run against.
Caveat recorded with the replacement. #863 carries more acceptance surface than #862 did: a 1,000-event differential fixture and 328 TypeScript unit tests re-expressed against an 842-line crate carrying 23 stubs. $14.70 is the calibrated floor, not a forecast with headroom. A budget overrun past 25% escalates per the operating rule, and the same formula re-applies from #863's measurement when it closes.
Still unreplaced: #864 ($2,400), #865 ($2,000), #866 ($1,600), #870 ($450), #867 ($1,400), #868 ($1,200), #869 ($600) — each is replaced when its phase's first job closes, not before.
Two housekeeping notes: the Phase 0 item (#861) still shows Expected $80 against Actual $3.71, which is the same placeholder situation — the earlier note deferred it and nothing has replaced it yet; and this umbrella's own Project item also carries Expected Cost USD 1,200. I have not touched either: the umbrella is not a phase, so its field should be cleared or set to the sum of its children, and that is Cameri's call.
chappie-daemon commentedon Oct 19, 2026
Phase 1, second measured job — #863 closed at
b324ef5e(PR #907), and the cost figure is not measurable on this host.The pricing gap is the finding: the run was served by
opencode-deepseek-v4.1-flash, which has no price configured, so everyusagerecord in all 19 of the job's transcripts carriescost: 0.factory-spansappended its line honestly ($0.00) — that is an absence of price, not a free job, and the Project item's Actual Cost USD is deliberately left empty rather than filled with a guess.What was measured: 19 sessions, 1,572 requests, 2.11M input / 1.43M output tokens, 250.3M cache-read, wall ~1h50m of agent time (recon 9m, spec 10m, build 19m46s + three resumes, four review rounds ≈20/12/8/7m, approve 5m30s).
Proxy for the calibration, labelled as one: at #862's blended $0.0102/request, 1,572 requests ≈ $16 — inside #863's $55 forecast and its $40–70 band. The formula cannot be re-applied from this job until the served model carries a price; when it does, this comment is the input to re-derive
expected(Phase 1)from two jobs instead of one.Two harness defects recorded with it: missing pricing needs an explicit "unpriced" marker so $0.00 is never read as a result, and
factory-spans' lower bound anchored on the job's firstplanline — which excluded this job's own recon, spec and first builder run (three sessions, ~500 requests).chappie-daemon commentedon Oct 19, 2026
Program-order adjustment: #864 (Phase 3, storage seam + LMDB) runs before #865 (Phase 2, relay handlers), on measured evidence.
The milestones put #865 in Phase 2, so the board order opened it first. Recon for #865 (
007-relay-handlers, stopped before the spec, no branch left behind) measured why that order cannot hold:EventRepositorydeclares 4 of 11 methods andUserRepository3 of 7 —upsert,find_by_filters,delete_by_pubkey_except_kinds,is_vanished/set_vanished/revoke_admissionandreport.create_manyare missing — so even a test double cannot serve REQ/EVENT. [Rewrite P3] Storage seam + LMDB backend (strfry-style) + PG→LMDB migration #864 is a hard prerequisite for [Rewrite P2] nostream-relay: wire handlers, strategies, rate limits #865's acceptance as written, and soft only for a handler-only implementation.DATABASE_URLunset — and the Cucumber suite'sBeforeAllneeds the same database.So: #864 opens next (its own anchor $2,400 stands only because Phase 3 has no measured job; it is replaced, not scaled, from its measurement). #865 is not cancelled — it resumes after the storage seam exists, with the frame fixture and the 774-case oracle already identified. This is a disposition change, recorded here before the job opens, per the charter.
chappie-daemon commentedon Oct 19, 2026
Phase 3 scope split (#864), recorded before the build — expected cost replaced, not absorbed.
The issue bundles five surfaces under a $2,400 anchor set when no Phase-3 job had run. Recon measured what each surface is (traits 51 methods across nine interfaces; 37 knex migrations; LMDB store 2.5–4k LOC; NIP-50 300–600; migration tool 600–1,000; benchmark rig with no corpus existing), and the spec delivered the first slice:
migrate pg-to-lmdb(follow-up)The $2,400 was replaced on the Project item with $1,170 for this job, and the four remainders are named, sized and preconditioned in
spec.md§6. They will be carried as their own items with their own expected costs rather than folded back into this one. Phase 3 still has no measured job — this job's close supplies the first, and the same formula then replaces these figures.}chappie-daemon commentedon Oct 24, 2026
Plan adjustment, 2026-10-24 — costs re-estimated from measured jobs
The per-issue dollar figures this plan was scoped with were wrong by two to three orders of magnitude. They priced the work; a factory job is billed by builder passes and review rounds, which barely track the size of the feature. Six measured jobs give the units, and the whole open plan re-costs to $46.10 against $10,629.40 on the board (231× out, after #595's build pass; the first figure here was $45.10, carried from the planning pass after four rows had been raised - #955, #956 and #957 by $0.10 each and #963 by $0.40 - so the total lagged its own table by $0.70. No row was missing. It now equals the sum of its rows, 19 of them, checked mechanically).
Measured, from
factory-spans.tsover each job's own transcripts:cost: 0cost: 0The units the model is built on: $0.0027 per request; $0.30 per small build pass and $2.72 per large one; $0.63 for a cold five-lens round, $0.39 scoped close-out, $0.11 for two lenses; $0.18 per approval; $0.03 per scoping. Full table, per-ticket basis and the per-issue dispositions:
.factory/estimates.md(committed to the workspace repo).Board changes made in this pass
Expected costline and a one-line basis were updated to match.src/routes/admin+src/routes/callbacks+ admissions/invoices routes"), so there is no work of its own to hold. Both descriptions say why, and reopening either is one click.rewritealready had its phase; [Rewrite] Umbrella: nostream 4.0 — Rust + LMDB, factory-managed #860 is the umbrella and spans all of them. The 74 unassigned issues outside the rewrite set were left untouched, as instructed — the 2.x backlog, the website, the peering programme (Relay peering: advertise terms in NIP-11, authenticate peers, handle feeds distinctly from clients #931–Peering: owner-attested peering requests — the operator co-signs, staking their reputation #940, Produce the relay peering launch film #942), the 2.x payment-processor test gaps, the 2.x upgrade defects and the TypeScript admin-console phases.Six issues need a decision, not an estimate — #717, #718, #719, #720 (NIP-86/67/96/85: new NIPs the relay does not advertise at capture, so they are neither parity nor the audit's gap list), #595 (a Prometheus endpoint — Phase 6 has no observability work, and the soak may need one to be observable), #902 (scriptc single-binary POC — distribution experiment, not parity). They are unassigned until answered.
Six decisions, applied 2026-10-24. #718 NIP-67 folded into #870 (Phase 2,
blocked_by #870, left open); #595 Prometheus/metricsinto Phase 6 as a build pass inside #868, whose estimate moves $3.00 → $3.30 — the soak is unfalsifiable without it, which is why it is the one of the six that cannot wait; #717 NIP-86, #719 NIP-96 and #720 NIP-85 deferred past cutover with no milestone (each is a new surface or engine rather than parity, and a "Post-cutover" milestone would be a phase with no phase); #902 labelledown-trackand out of the critical path, free to fail without consequence. All four deferred items are unestimated in the "Deferred past cutover" section — deferral is not a price.Total now $46.10, 19 rows, and the file's own
.factory/check-estimatesrecomputes rows-against-header rather than trusting the prose; it was shown failing on a deliberately broken header before it was relied on.chappie-daemon commentedon Oct 24, 2026
Plan adjustment, 2026-10-24 — #865 re-costed from $4.00 to $5.00 on job 011's spec
Recorded here because the charter asks for every plan change to land on this issue, and this one moved an estimate after it was published.
What changed the number. The row was priced as "fourteen strategy bodies". Job 011's spec found that the arms cannot be implemented honestly without three core stubs they call:
src/utils/nip03.ts, 396 TS lines) — thetimestamparm would otherwise be a stub that rejects every valid proof;src/utils/nip56.ts102 +src/utils/report-scoring.ts16), three Phase 1 stubs no code path calls today;limits::Limiter, which the report and vanish arms need.With the 802 lines of strategy TypeScript that is ~1,300 TS lines, plus a census-backed test target of roughly 1,600 Rust lines (one test per census title, 1:1). Job 008 — $6.91 — ported 3,700 lines over five review rounds, so this is a smaller port needing two or three rounds.
New figures: #865 $5.00 (range $4.00–6.00), #860 umbrella $47.10, board and body updated together, and
.factory/check-estimatespasses: 19 rows, header, headline, per-phase subtotals and the umbrella row all agree.What did not change. No scope was cut to protect the estimate: all fourteen arms stay in the job, and the option to park
timestamp(13 census titles, ~400 lines) was available and refused, because parking it would ship an arm that silently rejects valid proofs — a worse outcome than the extra dollar.