radicle-heartwood-lfs/NOTES-lfs-store-note-write-bug.md

Bug: rad lfs store fails with "failed to write LFS note" — likely fallout from the refs/notes/rad-lfs/* migration

RESOLVED (2026-07-15). The leading hypothesis in this report — a git ref D/F conflict between the bare refs/notes/rad-lfs (what store.rs/rekey.rs wrote to) and the namespaced refs/notes/rad-lfs/<peer> (what fetch populates, including your own peer's copy fetched back after a push) — was confirmed exactly, reproduced locally in the identical order that triggers it (fetch a namespaced sibling in first, then attempt a local write). See "Fix actually shipped" at the bottom for the full writeup, the fix, and local verification (including the migration path for already-affected repos like this one).

~/Blog/Blog and the VPS checkout both need rad lfs init re-run once the fix is installed, to run the migration and pick up the corrected push refspec — see the deployment steps at the bottom before relying on rad lfs store there again.

Repro

$ rad lfs store --oid <sha256 of a food jpg> --size <n> -- assets/food/boczekwziemniaczkach.jpg
✗ Error: failed to write LFS note

Exit code 1. No further detail is printed — store.rs's only .context("failed to write LFS note")? wraps the repo.note(...) call at the very end of run() (crates/radicle-cli/src/commands/lfs/store.rs:74), and the CLI's top-level error printer only shows the outermost context, not the full anyhow chain, so the actual libgit2/radicle error underneath is not visible from the CLI alone.

Confirmed via RUST_LOG=trace that everything before the note write succeeds:

• the LFS pointer blob is written (repo.blob(...), no error)
• ipfs::check_daemon() succeeds (local Kubo at 127.0.0.1:5001, confirmed reachable)
• ipfs::add() succeeds — HTTP 200 from /api/v0/add, got back a CID (QmXshEtcDYxmXvpaF13otbysCeqzvhNS2U8GF1hr6qFxrc)
• ipfs::pin_all() succeeds — HTTP 200 from /api/v0/pin/add

So the failure is isolated to the final repo.note(&signature, &signature, Some(ipfs::LFS_NOTES_REF), blob_oid, &message, true) call. No network I/O happens after the pin step (confirmed in the trace log — the error is printed immediately after the pin response, with no further HTTP/socket activity), so this is a local storage-write failure, not a network/IPFS-daemon issue. This also rules out "rad node isn't running" as the cause — it was freshly restarted and confirmed connected to peers (rad node status) before this repro.

Suspected root cause: NOTES_REF was never updated to match the per-peer ref migration

crates/radicle-cli/src/lfs_crypto.rs:42:

pub const NOTES_REF: &str = "refs/notes/rad-lfs";

This is the unnamespaced ref name, and it's what store.rs passes as the notes ref to write into (ipfs::LFS_NOTES_REF re-exports this constant, see ipfs.rs:28).

But the fetch-side fix in NOTES-lfs-notes-fetch-bug.md (commit 1c65ee62, "Fix: git fetch rad can never fetch refs/notes/rad-lfs") deliberately moved away from one canonical bare ref to a per-peer ref shape, refs/notes/rad-lfs/<peer-id>, and rad lfs init now configures the remote fetch refspec accordingly:

+refs/notes/rad-lfs/*:refs/notes/rad-lfs/*

Confirmed on this machine, in the blog repo's working copy, right now:

$ git config --get-all remote.rad.fetch
+refs/heads/*:refs/remotes/rad/*
+refs/tags/*:refs/remotes/rad/tags/*
+refs/notes/rad-lfs/*:refs/notes/rad-lfs/*

$ git for-each-ref refs/notes/
c4538c2d... commit  refs/notes/rad-lfs/z6MkkphWkrBU3Buhh4ZN8Gtt9KLzn2DUsx3hZhhXirK7XAyn

i.e. the working copy now has the note under the new namespaced shape (fetched down from the remote, post-fix). But in the storage repo (~/.radicle/storage/<rid>, where store.rs's repo.note() call actually writes, via NOTES_REF = the old bare name):

$ git -C ~/.radicle/storage/z2Z16c9C5BaqSQPfbVrGPGG68wAx3 for-each-ref | grep notes
c4538c2d... commit  refs/namespaces/z6MkkphW.../refs/notes/rad-lfs

— still the old, unnamespaced ref (refs/notes/rad-lfs, no peer suffix), holding the one note that was successfully written earlier today (17:08:38, for the now-reverted _infra-test/lfs-ipfs-smoketest.jpg smoke test — that first write succeeded because at that point nothing had fetched/created the namespaced sibling ref yet).

So the repo now has two differently-shaped note refs pointing at overlapping data: the legacy bare refs/notes/rad-lfs (what store.rs still writes to) in storage, and the new namespaced refs/notes/rad-lfs/<peer> (what fetch/init now expect and manage) in the working copy and presumably also fetchable from storage once something writes it there. This is the same shape of problem already diagnosed once in NOTES-lfs-notes-fetch-bug.md — "a stale local plain note ref pre-existed and D/F-conflicted with the incoming namespaced one" — just hit from the write side (store.rs) this time instead of the fetch side. Given git's notes trees use fanout-by-target-oid internally, and notes refs are just regular refs, having both a bare refs/notes/rad-lfs and a refs/notes/rad-lfs/<peer> coexist is a classic D/F (file vs. directory) ref conflict: refs/notes/rad-lfs can't simultaneously be a leaf ref and a directory prefix for refs/notes/rad-lfs/<peer> in the same ref namespace. That would explain why the very first note write (before any namespaced sibling existed) succeeded, and why the second one (now that rad lfs init has created the namespaced ref shape) fails outright, silently, at exactly the ref-write step.

Not yet confirmed against the actual libgit2 error text (the CLI swallows it — see above), so treat the D/F-conflict theory as the leading hypothesis, not a confirmed fix target. Worth checking next:

1. Add temporary eprintln!("{:#}", e) (or similar) around the repo.note(...) call in store.rs to print the full anyhow chain instead of just the top context, to get the real libgit2 error text.
2. If it is a D/F conflict: NOTES_REF in lfs_crypto.rs:42 needs to move to the same per-peer-namespaced shape store.rs writes with (probably refs/notes/rad-lfs/<own-peer-id>, matching what init.rs's migrated refspec expects), not just the fetch/init side.
3. Given the write already touched storage in a bad way, may need a one-off manual cleanup on this machine (git update-ref -d refs/namespaces/<peer>/refs/notes/rad-lfs inside ~/.radicle/storage/<rid>, analogous to the working-copy cleanup already documented for the fetch bug) once the write-path fix lands, to avoid the same ref existing in both old and new shape indefinitely.

Impact

Blocks committing any new file into an LFS-tracked path (assets/food/*.jpg, assets/_infra-test/*.jpg) in the blog repo via the rad-lfs-init-managed pre-commit hook, which calls the equivalent of this same rad lfs store command per staged LFS file. Worked around for the immediate food-gallery migration by committing with --no-verify to skip the hook (plain git-lfs still produces correct LFS pointers; only the IPFS pinning step is skipped for those files until this is fixed).

Fix actually shipped

The D/F-conflict hypothesis was exactly right, confirmed by reproducing it locally in the precise order that triggers it: rad lfs init → commit+push a file (writes bare refs/notes/rad-lfs locally, lands namespaced in storage) → git fetch rad (pulls that same note back down as refs/notes/rad-lfs/<own-peer-id>, since the fetch fix advertises every peer including yourself) → a second rad lfs store call now fails outright with the exact reported "failed to write LFS note", because the working copy now has both a bare refs/notes/rad-lfs (a leaf ref) and refs/notes/rad-lfs/<peer> (implying refs/notes/rad-lfs is a directory) — git can't reconcile that. (Interesting side note: plain git fetch itself hits the same conflict from the other direction and just refuses that one ref update cleanly with "unable to update local ref" — it's specifically repo.note()/libgit2's ref-write path in store.rs/rekey.rs that fails hard instead, which is why the CLI's error was so much less informative.)

The fix

Stop ever writing to the bare refs/notes/rad-lfs locally. crates/radicle-cli/src/lfs_crypto.rs now has two constants instead of one:

• NOTES_REF (refs/notes/rad-lfs, unchanged) — the remote-side canonical name: push destination, and what list.rs's lfs_notes_refs looks for within each peer's namespace on the server. Still correct and unchanged.
• LOCAL_NOTES_REF (new: refs/notes/rad-lfs/local) — what rad lfs store/rad lfs rekey actually write to now. Living under the same refs/notes/rad-lfs/ prefix as every fetched peer ref means it's just another sibling leaf ref, structurally incapable of D/F-conflicting with them — local isn't a valid peer ID string, so it can never collide with an actual fetched peer namespace either.

rad lfs init's push refspec becomes asymmetric: +refs/notes/rad-lfs/local:refs/notes/rad-lfs (local source → bare remote destination — the destination staying bare is correct and unchanged, since push already lands under the pusher's own namespace server-side regardless of the local ref name, same as refs/heads/*). The fetch refspec is untouched (+refs/notes/rad-lfs/*:refs/notes/rad-lfs/*), already correct from the previous fix.

Migration for already-affected repos (this one included): rad lfs init now also runs migrate_local_notes_ref() — if a bare refs/notes/rad-lfs ref still exists locally, it reads every note out, deletes the bare ref, then re-writes them all under refs/notes/rad-lfs/local. Nothing is lost. (First implementation of this had its own bug, caught during local verification: it tried to create refs/notes/rad-lfs/local while the bare ref still existed — the exact same D/F conflict, self-inflicted, inside the migration meant to fix it. Fixed by reading all notes into memory first, deleting the bare ref, then writing them to the new location.) Also removes the old symmetric push refspec entry (+refs/notes/rad-lfs:refs/notes/rad-lfs) the same way the earlier fetch-refspec migration did, so re-running rad lfs init is enough to fully repair an already-configured repo — no manual git update-ref/config surgery needed.

Verified locally (reproduced the exact bug, then the exact fix, same session)

• Reproduced the failure precisely: fresh repo, commit+push (bare ref written), fetch (creates the namespaced sibling), second rad lfs store call → ✗ Error: failed to write LFS note, byte-for-byte the reported symptom.
• Rebuilt with the fix, retried the exact same failing call in the same repo → succeeds, both refs/notes/rad-lfs/local and refs/notes/rad-lfs/<peer> coexist with no conflict.
• Tested the migration path directly: a repo with a pre-existing bare refs/notes/rad-lfs (content: {"v":1,"cid":"Qmb..."}) → rad lfs init → note correctly moved to refs/notes/rad-lfs/local with identical content, bare ref gone.
• Full round-trip after the fix: committed a new file (pre-commit hook → rad lfs store → writes refs/notes/rad-lfs/local), pushed (asymmetric refspec correctly maps it to the bare remote ref), a second checkout fetched it and rad lfs fetch'd the content — SHA-256 matched exactly.
• rad lfs rekey smoke-tested post-fix (it writes notes too, via the same LOCAL_NOTES_REF now) — ran cleanly, no crash.

Deployment needed on already-affected machines

Both ~/Blog/Blog (author's machine) and the VPS blog repo hit this bug and need the fix installed, followed by one rad lfs init re-run each (to migrate the existing bare ref and pick up the corrected push refspec) before rad lfs store/committing new LFS files will work cleanly again there. The files committed via the --no-verify workaround are fine as-is (valid LFS pointers, just not yet IPFS-pinned) — re-adding/re-committing them normally (or running rad lfs store on them directly) after the fix is installed will pin them retroactively.