Found live while migrating the blog's food-gallery images into LFS,
diagnosed in NOTES-lfs-store-note-write-bug.md, confirmed and fixed here.
Root cause: store.rs/rekey.rs wrote local notes to the bare
refs/notes/rad-lfs, but the earlier fetch-bug fix (1c65ee62) has the
fetch refspec populate refs/notes/rad-lfs/<peer> siblings locally --
including your own peer's copy, fetched back after your own push. A bare
refs/notes/rad-lfs ref can't coexist with refs/notes/rad-lfs/<peer> in
the same git ref namespace (a leaf ref vs. a directory prefix at the same
path) -- so the very first commit+push+fetch cycle on a repo left it
permanently unable to `rad lfs store` again, failing with "failed to
write LFS note" and no further detail (git itself hits the same conflict
from the fetch direction and just refuses that one ref cleanly; it's
specifically libgit2's ref-write path used here that fails hard instead).
Fix: stop writing to the bare ref locally at all. New
lfs_crypto::LOCAL_NOTES_REF (refs/notes/rad-lfs/local) is what
store.rs/rekey.rs write to now -- living under the same
refs/notes/rad-lfs/ prefix as every fetched peer ref makes it just
another sibling leaf ref, structurally incapable of conflicting with
them ("local" can never collide with an actual peer ID). NOTES_REF
(bare) stays exactly as it was for the remote-side/push-destination
role, unchanged. rad lfs init's push refspec becomes asymmetric
(+refs/notes/rad-lfs/local:refs/notes/rad-lfs) instead of symmetric;
fetch refspec is untouched.
Added a migration step (migrate_local_notes_ref, run by rad lfs init)
for already-affected repos: reads any existing bare-ref notes out,
deletes the bare ref, then re-writes them under the new local ref --
also removes the old symmetric push refspec config entry the same way
the earlier fetch-refspec migration did, so re-running rad lfs init
fully repairs an already-broken repo with no manual git surgery needed.
(First version of the migration itself had the identical bug -- tried to
write the new ref before deleting the old one, hit the same conflict
self-inflicted -- fixed by reading all notes into memory before deleting
the bare ref.)
Verified locally: reproduced the exact failure in the precise order that
triggers it, confirmed the fix resolves that exact same failing call in
the same repo, verified the migration path preserves note content
exactly, and confirmed a full commit->push->fetch->decrypt round trip
plus `rad lfs rekey` all still work correctly afterward.
Bug #2 (ls-remote vs fetch disagreeing on the VPS) turned out to be two
separate things, neither a bug in the fetch fix itself: a stale,
unrelated system-wide git-remote-rad binary being silently used in
ad-hoc manual testing (PATH had no ~/.radicle/bin), plus a genuine
cross-device rename bug in rad-lfs-transfer's download path (fixed
separately, ac4a27c) that only surfaced once the real pipeline actually
ran with the fetch fix for the first time. Verified live: triggered
blog-deploy.service for real, full success end-to-end, SHA-256-verified.
Found in a real deployment (mbator.pl blog pipeline): rad lfs init
configures a client-side fetch refspec for refs/notes/rad-lfs, but
git-remote-rad's for_fetch() never advertised any such ref -- notes only
exist per-peer, under refs/namespaces/<peer>/refs/notes/rad-lfs, with no
canonicalization step the way refs/heads/refs/tags get one. Requesting an
unadvertised refspec fails the *entire* git fetch, not just that one ref,
so any repo with `rad lfs init` run could no longer git fetch/pull at
all, LFS or not.
Fix: expose each peer's refs/notes/rad-lfs individually, as
refs/notes/rad-lfs/<peer-id> (list.rs's for_fetch), matching how patch
refs already work rather than trying to canonicalize -- notes are
per-peer contributions (any peer can commit an LFS-tracked file, not
just delegates), so unlike heads there's no single "correct" value to
resolve to; canonicalizing would silently drop non-delegate contributors'
objects. rad lfs init's fetch refspec becomes a wildcard
(+refs/notes/rad-lfs/*:refs/notes/rad-lfs/*); push is unchanged (already
worked -- for_push() turns out to be purely informational for git's
list-for-push handshake, not an enforcement gate, so the actual send-pack
call was never restricted to heads/tags despite what for_push() lists).
rad lfs store/fetch/rekey and seed/unseed's pinning (ipfs.rs::lfs_cids)
now read across every notes ref instead of assuming one canonical ref,
merging recipient lists for notes that share a CID (e.g. after a rekey
performed by a different peer than the original committer) while keeping
genuinely different ciphertexts (from two peers independently encrypting
byte-identical content) as separate candidates to try in turn.
Also added a migration step so re-running `rad lfs init` on an
already-configured repo actually fixes it, by removing the old
non-wildcard fetch refspec rather than just adding the wildcard
alongside a still-broken entry.
Verified through the real git-remote-rad transport (not the storage-copy
shortcut used for earlier testing, since this bug lives specifically in
that transport): fresh identities, a real rad:// remote matching exactly
what `rad init` configures, a genuine `git fetch rad` that used to fail
with "fatal: couldn't find remote ref refs/notes/rad-lfs" now succeeds
and pulls the notes ref, and `git lfs pull` afterward correctly decrypts
private content end-to-end (SHA-256 verified). Re-verified the rekey
flow through the same real path too.
Maciek "mab122" Bator