mab122
/
radicle-heartwood-lfs
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Compare commits

merge into: mab122:master
Branches Tags
mab122:rad-lfs-ipfs
mab122:master
...
pull from: mab122:rad-lfs-ipfs
Branches Tags
mab122:rad-lfs-ipfs
mab122:master

29 Commits
master ... rad-lfs-ip

Author SHA1 Message Date
Maciek "mab122" Bator 76e3d6ced8 docs: note package/publish gaps blocking the 10MB-fix release 
Code fix (35f1f972) is done, tested, and pushed. Package build + Forgejo
package-registry publish is blocked on missing packaging conventions
here (no existing publish script/credentials/target-scope decision) --
see NOTES-package-publish-todo.md for what's needed to finish this.
 2026-07-19 12:51:54 +02:00
Maciek "mab122" Bator 35f1f97290 Fix: rad lfs fetch fails on any LFS object over 10MB from IPFS 
`ipfs::cat()` used ureq's `Body::read_to_vec()`, which defaults to a
10MB response-size limit meant for typical HTTP API responses. Any LFS
object larger than that (e.g. the ~29MB thumbnail.xcf in the blog repo)
failed with "the response body is larger than request limit: 10485760",
breaking `rad lfs fetch`/`rad lfs store`-driven smudges for large files.

Raise the cap to 1GiB via ureq's `with_config().limit(...)`, since LFS
content is exactly the out-of-band-large-file case this whole feature
exists for. Verified: reproduced the exact production error with a 15MB
round-trip test against a live Kubo daemon, confirmed it fails without
the fix and passes with it, and confirmed live against the actual
29MB thumbnail.xcf object that broke deploy.
 2026-07-19 12:49:41 +02:00
Maciek "mab122" Bator 4ae9756a3d docs: explain the fetch-batch worker in LFS-IPFS.md 2026-07-17 23:43:18 +02:00
Maciek "mab122" Bator a8c8dbf3dd Bump radicle-lfs-transfer submodule: batched multi-object fetch worker 2026-07-17 23:42:40 +02:00
Maciek "mab122" Bator 7c6d342ef9 lfs: add fetch-batch, a long-lived worker for multi-object downloads 
`rad lfs fetch` still exists for single-object/manual use, but
`radicle-lfs-transfer` used to spawn a fresh one per object during
`git lfs pull`/checkout -- for a private repo, each one independently
loaded the signer, so N files meant N passphrase prompts. Same bug
class `rad lfs precommit` already fixed for commits, unfixed here
until now.

Unlike precommit (which knows every staged file upfront and can read
stdin until EOF), fetch can't collect a batch before starting:
git-lfs's custom-transfer protocol requests objects one at a time.
`rad lfs fetch-batch` instead stays alive for as long as its caller
keeps its stdin open, answering one "<oid> <size> <out-path>" request
at a time with a JSON response line, reusing the same unwrapped
signer across all of them -- paired with radicle-lfs-transfer's new
FetchWorker, which spawns this once per transfer session instead of
once per object.

Also factors `fetch::run`'s core logic out into `fetch_object`,
taking a `&mut Option<MemorySigner>` the same way `store::store_object`
already does, so both the single-shot `rad lfs fetch` and the new
batch worker share one implementation.
 2026-07-17 23:42:30 +02:00
Maciek "mab122" Bator 4ab6fc2499 docs: document rad lfs backfill and the unseed refcounting fix 2026-07-17 15:22:31 +02:00
Maciek "mab122" Bator c059c7967b lfs: cross-repo pin refcounting on unseed, add `rad lfs backfill` 
Two known gaps, both flagged as follow-up work in LFS-IPFS.md:

`rad unseed` unconditionally unpinned every CID a repository's LFS
notes referenced. Two repositories that happen to track
byte-identical file content get the same content-addressed CID, so
unseeding one could silently evict content another still-seeded
repository needed -- that repo's next `git lfs fetch` would then fail
with no obvious cause. Fixed by checking every other currently-seeded
repository's own LFS notes before unpinning, and skipping any CID
still referenced elsewhere. No persistent refcount is kept anywhere;
this recomputes the still-needed set from git-notes each time,
consistent with the rest of this design treating notes as the single
source of truth rather than maintaining separate bookkeeping that
could drift out of sync.

`rad lfs backfill` retroactively pins any LFS-tracked file at HEAD
that doesn't yet have a note -- the situation a `--no-verify` commit
or a commit made while `ipfs`/`rad` weren't on `PATH` leaves behind
(the pre-commit hook soft-skips pinning in the latter case rather
than blocking the commit entirely). Content is read from Git LFS's
own local object cache (populated by the clean filter on `git add`
regardless of whether the hook ran), falling back to the working-tree
file. Reports what it backfilled, what already had a note, and what
it couldn't find content for locally (meaning some other peer that
does have it needs to run this instead).
 2026-07-17 15:21:40 +02:00
Maciek "mab122" Bator 581843fa92 build: rerun-if-changed on .git/logs/HEAD, fix stale version on rebuild 
Without an explicit rerun-if-changed directive, cargo has no way to
know it needs to re-run this build script just because the current
commit changed -- by default it only reruns based on files inside
the crate's own source tree, and .git is entirely outside that. This
let GIT_HEAD/RADICLE_VERSION silently go stale: several `cargo
build`s in a row during this session kept reporting an old commit's
version despite the working tree being clean at a newer commit,
because cargo decided nothing needed rebuilding at all.

`.git/logs/HEAD` (the reflog) is touched on every commit, checkout,
and merge, unlike `.git/HEAD` itself (which only changes when
switching branches) -- watching it catches the case that actually
matters here.
 2026-07-17 09:42:30 +02:00
Maciek "mab122" Bator 2306aabc4a docs: add fork banner to .github/README.md (GitHub profile-README convention) 
Matches the banner already on the root README.md -- this file is
otherwise upstream heartwood's own GitHub-specific profile readme,
unrelated to and predating the LFS-over-IPFS work, so someone landing
on it directly (or via GitHub's own rendering convention) had no
indication this repo is a fork with different setup instructions.
 2026-07-17 09:33:39 +02:00
Maciek "mab122" Bator 3ad836577a Bump radicle-lfs-transfer submodule: doc update for push refspec fix 2026-07-16 15:23:22 +02:00
Maciek "mab122" Bator a84aeb23f7 lfs: configure a branch push refspec too, single `git push rad` now works 
`rad lfs init` previously only configured a push refspec for the
notes ref (`+refs/notes/rad-lfs/local:refs/notes/rad-lfs`) -- once
any explicit push refspec exists on a remote, git stops falling back
to push.default-driven behavior for a bare `git push rad`, so it only
ever pushed the notes mapping, never the branch. That's the exact
footgun that just caused a real "blog did not update" report: `git
push rad` (bare) after a commit only sent the notes ref, silently
leaving the commit itself unpushed.

Add a second push refspec, `+refs/heads/*:refs/heads/*`, mirroring
the existing `+refs/heads/*:refs/remotes/rad/*` fetch refspec already
configured. A bare `git push rad` now pushes every local branch and
the notes mapping together in one command.

Only takes effect once `rad lfs init` has been (re-)run with this
build -- it's an additive, idempotent config change via the existing
`ensure_refspec` helper, no migration/cleanup needed since there's no
stale conflicting entry to remove this time. Updated `rad lfs init`'s
own success/info messages plus README.md/LFS-IPFS.md/
radicle-lfs-transfer's README to match -- the "push gotcha" workaround
they documented is now the *old*-repository fallback path, not the
primary instructions.
 2026-07-16 15:23:00 +02:00
Maciek "mab122" Bator 245fdd3fdc Bump radicle-lfs-transfer submodule: fix stale pre-batching README claim 2026-07-16 14:39:30 +02:00
Maciek "mab122" Bator 5227783cac docs: fix nonexistent rad push/pull commands, stale pre-batching claims 
There is no `rad push`/`rad pull` subcommand -- `rad`'s actual
subcommand set is auth/checkout/clone/.../sync/watch (see `rad
--help`); pushing/pulling git refs (branch, notes) is done via plain
`git push rad`/`git pull rad`, same as any git remote. LFS-IPFS.md's
"Use" section and `rad lfs init`'s own printed success message both
said `rad push`/`rad pull`, which doesn't exist and doesn't even
gesture at the real push gotcha (branch and notes ref need two
separate pushes).

Also updates LFS-IPFS.md's "How it works" section, which still
described the pre-batching design (a pre-commit hook calling `rad lfs
store` once per file, and the bare `refs/notes/rad-lfs` as if it were
the local write target) -- both stale since the LOCAL_NOTES_REF fix
and the `rad lfs precommit` batching added this session. `rad lfs
init`'s own success message now proactively states the push gotcha,
rather than only documenting it after the fact in Troubleshooting.
 2026-07-16 14:39:03 +02:00
Maciek "mab122" Bator 891383bf7b scripts: skip already-published versions, attach a changelog 
publish-package.sh now checks the Forgejo package listing first and
exits early if the current version is already published, instead of
failing with a 409 on the first upload attempt. Also generates a
changelog (git log since whichever previous package's embedded
short-sha we can find) and attaches it both inside the tarball
(MANIFEST.txt, CHANGELOG.txt) and as its own uploaded file, so anyone
browsing the package page can see what changed without downloading
the archive.
 2026-07-16 14:07:14 +02:00
Maciek "mab122" Bator 68202ce58d build: tag rad --version with a +lfs-ipfs suffix, fix newline bug 
Appends a semver build-metadata suffix to RADICLE_VERSION so `rad
--version` (and radicle-node/git-remote-rad, which share this build
script via symlink) unambiguously identify a binary as this fork
rather than upstream heartwood -- useful for checking what a deploy
target is actually running.

While wiring this up, found and fixed a real pre-existing bug: none
of the three `git` subprocess outputs captured here (GIT_HEAD,
RADICLE_VERSION, SOURCE_DATE_EPOCH) were trimmed, so each carried a
trailing newline into its `cargo::rustc-env=KEY=VALUE` directive.
Harmless on its own since nothing followed it, but appending the
+lfs-ipfs suffix landed after that embedded newline, splitting the
directive across two lines -- cargo silently discards the second
line as unrecognized, so the suffix never actually took effect until
this trim was added.

Also adds scripts/publish-package.sh: builds all four release
binaries (rad, radicle-node, git-remote-rad, rad-lfs-transfer) and
publishes them as a Forgejo generic package, so a deploy target
without a Rust toolchain can download instead of rebuilding from
source (see NOTES-lfs-store-note-write-bug.md's VPS deployment story
for why that matters).
 2026-07-16 13:49:13 +02:00
Maciek "mab122" Bator ae7aed5db5 docs: add quickstart cheatsheet to README, clarify push gotcha 
Adds a copy-pasteable day-to-day cheatsheet (init, track/commit/push,
clone-elsewhere, private-repo rekey) right after the existing "Zero to
usable" build section. Also corrects/expands the existing
git-push-rad troubleshooting row in LFS-IPFS.md based on today's live
round-trip test: `rad lfs init` only configures a push refspec for the
notes ref, not the branch, so pushing a commit needs both
`git push rad <branch>` and a separate bare `git push rad` -- missing
the second step is what produces "no CID recorded for oid" on a
collaborator's clone.
 2026-07-16 12:54:42 +02:00
Maciek "mab122" Bator fa4fdf5470 lfs: fix pre-commit hook silently skipping non-ASCII filenames 
`git diff --cached --name-only` C-quotes filenames with non-ASCII
bytes by default (e.g. a literal "\305\202" for a Polish "ł"), which
doesn't match any real path on disk -- the hook's `[ -f "$file" ]`
check silently skipped those files instead of erroring, so they were
committed as valid LFS pointers but never pinned to IPFS. Use `-z`
(NUL-separated, unquoted) and convert to newlines for the read loop.

Also add an idempotency fast-path to `rad lfs store`/`store_object`:
if a note already exists for an object (e.g. written moments earlier
by `rad lfs precommit`), return its CID directly instead of
re-encrypting. This avoids redundant work, and matters more than that
for private repos specifically: `git push`'s custom-transfer-agent
subprocess chain has no TTY, so without this fast path every push
would fail outright on the passphrase prompt even for objects already
stored interactively at commit time.
 2026-07-16 12:41:01 +02:00
Maciek "mab122" Bator c070aac6fe lfs: batch pre-commit hook into one `rad lfs precommit` process 
Previously the pre-commit hook shelled out to `rad lfs store` once per
staged LFS file, each a separate process. For private repos, each of
those needs a keystore passphrase for an ECDH key-agreement operation,
so a multi-file commit meant a separate interactive passphrase prompt
per file -- observed live against the blog repo (9+ prompts in a row).

Add `rad lfs precommit`, which reads every staged file's oid/size/path
from stdin and processes them all in one process, loading the signer
at most once and reusing it across the batch. `store::run` now
delegates to a shared `store_object` helper that both it and
`precommit::run` call.
 2026-07-15 19:23:21 +02:00
Maciek "mab122" Bator a9428a9ef9 Fix: rad lfs store/rekey fail with git ref D/F conflict on notes ref 
Found live while migrating the blog's food-gallery images into LFS,
diagnosed in NOTES-lfs-store-note-write-bug.md, confirmed and fixed here.

Root cause: store.rs/rekey.rs wrote local notes to the bare
refs/notes/rad-lfs, but the earlier fetch-bug fix (1c65ee62) has the
fetch refspec populate refs/notes/rad-lfs/<peer> siblings locally --
including your own peer's copy, fetched back after your own push. A bare
refs/notes/rad-lfs ref can't coexist with refs/notes/rad-lfs/<peer> in
the same git ref namespace (a leaf ref vs. a directory prefix at the same
path) -- so the very first commit+push+fetch cycle on a repo left it
permanently unable to `rad lfs store` again, failing with "failed to
write LFS note" and no further detail (git itself hits the same conflict
from the fetch direction and just refuses that one ref cleanly; it's
specifically libgit2's ref-write path used here that fails hard instead).

Fix: stop writing to the bare ref locally at all. New
lfs_crypto::LOCAL_NOTES_REF (refs/notes/rad-lfs/local) is what
store.rs/rekey.rs write to now -- living under the same
refs/notes/rad-lfs/ prefix as every fetched peer ref makes it just
another sibling leaf ref, structurally incapable of conflicting with
them ("local" can never collide with an actual peer ID). NOTES_REF
(bare) stays exactly as it was for the remote-side/push-destination
role, unchanged. rad lfs init's push refspec becomes asymmetric
(+refs/notes/rad-lfs/local:refs/notes/rad-lfs) instead of symmetric;
fetch refspec is untouched.

Added a migration step (migrate_local_notes_ref, run by rad lfs init)
for already-affected repos: reads any existing bare-ref notes out,
deletes the bare ref, then re-writes them under the new local ref --
also removes the old symmetric push refspec config entry the same way
the earlier fetch-refspec migration did, so re-running rad lfs init
fully repairs an already-broken repo with no manual git surgery needed.
(First version of the migration itself had the identical bug -- tried to
write the new ref before deleting the old one, hit the same conflict
self-inflicted -- fixed by reading all notes into memory before deleting
the bare ref.)

Verified locally: reproduced the exact failure in the precise order that
triggers it, confirmed the fix resolves that exact same failing call in
the same repo, verified the migration path preserves note content
exactly, and confirmed a full commit->push->fetch->decrypt round trip
plus `rad lfs rekey` all still work correctly afterward.
 2026-07-15 19:02:53 +02:00
Maciek "mab122" Bator 945da6f7f8 docs: record root cause and live verification of Bug #2 
Bug #2 (ls-remote vs fetch disagreeing on the VPS) turned out to be two
separate things, neither a bug in the fetch fix itself: a stale,
unrelated system-wide git-remote-rad binary being silently used in
ad-hoc manual testing (PATH had no ~/.radicle/bin), plus a genuine
cross-device rename bug in rad-lfs-transfer's download path (fixed
separately, ac4a27c) that only surfaced once the real pipeline actually
ran with the fetch fix for the first time. Verified live: triggered
blog-deploy.service for real, full success end-to-end, SHA-256-verified.
 2026-07-14 18:53:45 +02:00
Maciek "mab122" Bator a97484c92f Bump radicle-lfs-transfer submodule: fix cross-device rename on LFS download 2026-07-14 18:44:06 +02:00
Maciek "mab122" Bator ff762ef8f2 Prompt interactively for passphrase instead of requiring RAD_PASSPHRASE 
Private-repo LFS operations need a signer capable of ECDH, which
ssh-agent (the normal day-to-day flow after `rad auth`) can't provide --
that's a hard protocol limitation (standard ssh-agent only implements
signing, there's no key-agreement request type), not a gap in our code,
so there's no way to make it work "through the agent" the way signing
does.

Previously the only alternative to an unencrypted keystore was setting
RAD_PASSPHRASE, which breaks the normal ssh-agent-based workflow and
means putting a password in an environment variable -- not ergonomic,
and not how any other `rad` command behaves.

`rad` already has an established fallback for exactly this situation:
`crate::terminal::io::signer` tries ssh-agent first, and if that's
unavailable/unregistered and we're connected to a TTY, prompts
interactively for the passphrase via `inquire` (reusing the existing
`PassphraseValidator`). `lfs_crypto::load_signer` was missing this last
step entirely -- it only tried "unencrypted keystore" then
"RAD_PASSPHRASE" then gave up. Added the same interactive-prompt
fallback, reusing the exact same validator/prompt helpers already used
elsewhere, before falling back to a hard failure.

This covers the common case (an interactive `git commit` triggering the
pre-commit hook, which inherits the terminal's TTY) with no environment
variable needed at all. Only genuinely non-interactive contexts (CI, a
script with no TTY) still need RAD_PASSPHRASE -- and get a fast, clear
failure rather than hanging, since `inquire` returns `Ok(None)` on
`NotTTY` rather than blocking (verified: piping /dev/null as stdin still
fails immediately, not a timeout).

Verified against a real encrypted keystore through an actual pty (via
`script`, since a plain non-interactive shell doesn't have a TTY to
prompt on): the masked "Passphrase:" prompt appears, accepts the typed
passphrase, and produces a correctly encrypted note -- with
RAD_PASSPHRASE unset the entire time.
 2026-07-14 17:59:04 +02:00
Maciek "mab122" Bator 1c65ee6230 Fix: git fetch rad can never fetch refs/notes/rad-lfs 
Found in a real deployment (mbator.pl blog pipeline): rad lfs init
configures a client-side fetch refspec for refs/notes/rad-lfs, but
git-remote-rad's for_fetch() never advertised any such ref -- notes only
exist per-peer, under refs/namespaces/<peer>/refs/notes/rad-lfs, with no
canonicalization step the way refs/heads/refs/tags get one. Requesting an
unadvertised refspec fails the *entire* git fetch, not just that one ref,
so any repo with `rad lfs init` run could no longer git fetch/pull at
all, LFS or not.

Fix: expose each peer's refs/notes/rad-lfs individually, as
refs/notes/rad-lfs/<peer-id> (list.rs's for_fetch), matching how patch
refs already work rather than trying to canonicalize -- notes are
per-peer contributions (any peer can commit an LFS-tracked file, not
just delegates), so unlike heads there's no single "correct" value to
resolve to; canonicalizing would silently drop non-delegate contributors'
objects. rad lfs init's fetch refspec becomes a wildcard
(+refs/notes/rad-lfs/*:refs/notes/rad-lfs/*); push is unchanged (already
worked -- for_push() turns out to be purely informational for git's
list-for-push handshake, not an enforcement gate, so the actual send-pack
call was never restricted to heads/tags despite what for_push() lists).

rad lfs store/fetch/rekey and seed/unseed's pinning (ipfs.rs::lfs_cids)
now read across every notes ref instead of assuming one canonical ref,
merging recipient lists for notes that share a CID (e.g. after a rekey
performed by a different peer than the original committer) while keeping
genuinely different ciphertexts (from two peers independently encrypting
byte-identical content) as separate candidates to try in turn.

Also added a migration step so re-running `rad lfs init` on an
already-configured repo actually fixes it, by removing the old
non-wildcard fetch refspec rather than just adding the wildcard
alongside a still-broken entry.

Verified through the real git-remote-rad transport (not the storage-copy
shortcut used for earlier testing, since this bug lives specifically in
that transport): fresh identities, a real rad:// remote matching exactly
what `rad init` configures, a genuine `git fetch rad` that used to fail
with "fatal: couldn't find remote ref refs/notes/rad-lfs" now succeeds
and pulls the notes ref, and `git lfs pull` afterward correctly decrypts
private content end-to-end (SHA-256 verified). Re-verified the rekey
flow through the same real path too.
 2026-07-14 17:47:15 +02:00
Maciek "mab122" Bator 33ffa12286 Exclude radicle-lfs-transfer submodule from the outer cargo workspace 
Cargo's workspace auto-detection was pulling the submodule into the
parent workspace since it's nested under the repo root, breaking
`cargo install --path radicle-lfs-transfer` (the exact command in the
README/LFS-IPFS.md setup instructions) with "current package believes
it's in a workspace when it's not". Discovered while installing the
built binaries onto a real machine.
 2026-07-14 16:18:55 +02:00
Maciek "mab122" Bator e060442017 Add client-side encryption for private-repo LFS objects 
Radicle's Visibility::Private is enforced entirely by access control at
the replication layer -- git objects for a private repo are plain,
unencrypted objects, protected only by "unauthorized peers are never
given them" (identity/doc.rs). That protection has no jurisdiction over
IPFS's own block-exchange layer: once a legitimate collaborator's IPFS
daemon pins content and joins the network, anyone who obtains the CID by
any means can fetch the plaintext directly.

Adds envelope encryption for private repos: a random per-object content
key encrypts the file (XChaCha20-Poly1305), wrapped once per authorized
recipient (delegates + private allow-list) via ECDH between Radicle
identity keys (already Ed25519, no separate keypair needed) plus HKDF.
The wrapped keys travel in the same refs/notes/rad-lfs note as the CID.

New:
- crates/radicle-cli/src/lfs_crypto.rs -- encryption core (envelope
  schema, encrypt/decrypt, key wrapping, recipient resolution, signer
  loading).
- `rad lfs store`/`rad lfs fetch` -- plumbing commands that now own the
  entire IPFS add/cat + git-notes read/write step (encrypting/decrypting
  as needed), replacing what the pre-commit hook and rad-lfs-transfer
  used to do directly. This collapses what would've become a third copy
  of the note-format logic down to one canonical implementation.
- `rad lfs rekey` -- since Radicle's access control is retroactive (a
  newly-authorized DID can replicate full history) but encrypted objects
  aren't automatically re-wrapped for new recipients, this command lets
  an already-authorized collaborator grant a newly-authorized one access
  to previously-committed objects, without re-encrypting or re-uploading
  content. Revocation remains unsolved, matching Radicle's own model
  (already-replicated/decrypted content can't be un-known).

Operational requirement: private-repo LFS operations need a signer
capable of ECDH, which only an unencrypted keystore or RAD_PASSPHRASE
provides (an ssh-agent-only signer can sign but not do key agreement).
Fails fast with an actionable error rather than hanging on a passphrase
prompt inside a non-interactive git hook.

Breaking change: the refs/notes/rad-lfs note format moves from bare
`cid=<cid>` text to JSON (`{"v":1,"cid":"...","enc":...}`). No migration
path -- prototype, no external users yet.

Verified end-to-end with real Radicle profiles/identities (not just the
transfer-agent protocol): a private repo's IPFS-stored content is
confirmed ciphertext (byte-different from plaintext, +16 bytes AEAD tag);
an authorized peer decrypts correctly; an unauthorized peer is denied
with a clear error and no leaked content; RAD_PASSPHRASE is required and
enforced with a fast, clear failure; and rekey correctly grants a
newly-authorized peer access to a pre-existing object while leaving other
recipients' entries untouched.
 2026-07-14 16:01:49 +02:00
Maciek "mab122" Bator 23264b9b52 docs: drop reference to the rejected seed-hosted-server design attempt 2026-07-14 12:53:20 +02:00
Maciek "mab122" Bator 5d53fab673 docs: zero-to-install quick start, Arch dependency lists, fix cross-repo links 
Cross-repo markdown links (radicle-lfs-transfer) now use full https URLs
instead of relative paths, since Forgejo can't resolve ../other-repo across
repositories the way a plain filesystem or the git submodule mechanism can.
 2026-07-14 12:51:30 +02:00
Maciek "mab122" Bator 005d708059 readme: note this is a fork adding LFS-over-IPFS support 2026-07-14 12:34:49 +02:00
Maciek "mab122" Bator bb8608e984 Add rad lfs init and IPFS-backed pinning for seed/unseed 
Adds Git LFS support for repositories, with large file content stored on
each contributor's own local IPFS node instead of a seed-hosted server:

- `rad lfs init`: one-time setup (transfer-agent config, pre-commit hook,
  notes-ref push/fetch refspecs, IPFS daemon reachability check).
- `refs/notes/rad-lfs` carries the oid -> CID mapping, replicating with
  the repository via Radicle's existing ref sync (verified against
  `references_of` in crates/radicle/src/storage/git.rs: every ref is
  replicated except refs/tmp/heads/*, so no protocol changes were needed).
- `rad seed`/`rad unseed` now pin/unpin a repo's known LFS objects in IPFS,
  mirroring Radicle's existing "seeding = keep a full copy" model, and
  warning (not failing) if no local IPFS daemon is reachable.

The actual byte transfer is handled by a separate binary, rad-lfs-transfer
(added here as a submodule), which implements the Git LFS custom-transfer-
agent protocol against a local IPFS daemon. See LFS-IPFS.md for the full
design and setup instructions.

Building and using `rad` for anything other than `rad lfs` requires no
IPFS dependency at all; `rad lfs init` checks for a local daemon upfront
and fails with an actionable message rather than proceeding silently.
 2026-07-14 12:32:12 +02:00
30 changed files with 3204 additions and 4 deletions
Show Stats Expand all files Collapse all files

11
.github/README.md vendored
View File

@ -1,5 +1,16 @@
# ❤️🪵
> ## Fork: Git LFS support, backed by IPFS
>
> This is a fork of upstream [`radicle-dev/heartwood`](https://github.com/radicle-dev/heartwood)
> adding Git LFS (large file) support, with large file content stored on each contributor's own
> local IPFS node rather than a central server. Everything below is upstream's own text,
> unchanged, and describes upstream `heartwood` in general, not this fork specifically.
>
> This repository is developed on a self-hosted Forgejo instance, not GitHub — see
> [the main `README.md`](../README.md) for the fork's own quickstart, and
> [`LFS-IPFS.md`](../LFS-IPFS.md) for the full design, encryption details, and troubleshooting.
*Radicle Heartwood Protocol & Stack*
Heartwood is the third iteration of the Radicle Protocol, a powerful

3
.gitmodules vendored Normal file
View File

@ -0,0 +1,3 @@
[submodule "radicle-lfs-transfer"]
path = radicle-lfs-transfer
url = ../radicle-lfs-transfer.git

87
Cargo.lock generated
View File

@ -1837,6 +1837,15 @@ version = "0.5.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "2304e00983f87ffb38b55b444b5e3b60a884b5d30c0fca7d82fe33449bbe55ea"
[[package]]
name = "hkdf"
version = "0.12.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "7b5f8eb2ad728638ea2c7d47a21db23b7b58a72ed6a38256b8a1849f15fbbdf7"
dependencies = [
"hmac",
]
[[package]]
name = "hmac"
version = "0.12.1"
@ -1846,6 +1855,22 @@ dependencies = [
"digest",
]
[[package]]
name = "http"
version = "1.4.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "6970f50e31d6fc17d3fa27329444bfa74e196cf62e95052a3f6fee181dba6425"
dependencies = [
"bytes",
"itoa",
]
[[package]]
name = "httparse"
version = "1.10.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "6dbf3de79e51f3d586ab4cb9d5c3e2c14aa28ed23d180cf89b4df0454a69cc87"
[[package]]
name = "human-panic"
version = "2.0.6"
@ -2327,6 +2352,22 @@ dependencies = [
"autocfg",
]
[[package]]
name = "mime"
version = "0.3.17"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "6877bb514081ee2a7ff5ef9de3281f14a4dd4bceac4c09388074a6b5df8a139a"
[[package]]
name = "mime_guess"
version = "2.0.5"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "f7c44f8e672c00fe5308fa235f821cb4198414e1c77935c1ab6948d3fd78550e"
dependencies = [
"mime",
"unicase",
]
[[package]]
name = "miniz_oxide"
version = "0.8.9"
@ -2952,10 +2993,14 @@ version = "0.21.0"
dependencies = [
"anyhow",
"backtrace",
"base64 0.21.7",
"chacha20poly1305",
"chrono",
"clap",
"clap_complete",
"cyphernet",
"dunce",
"hkdf",
"human-panic",
"humantime",
"itertools",
@ -2974,6 +3019,7 @@ dependencies = [
"schemars",
"serde",
"serde_json",
"sha2",
"shlex",
"tempfile",
"thiserror 2.0.18",
@ -2992,7 +3038,9 @@ dependencies = [
"tree-sitter-rust",
"tree-sitter-toml-ng",
"tree-sitter-typescript",
"ureq",
"winsplit",
"zeroize",
]
[[package]]
@ -4631,6 +4679,12 @@ version = "0.1.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "eaea85b334db583fe3274d12b4cd1880032beab409c0d774be044d4480ab9a94"
[[package]]
name = "unicase"
version = "2.9.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "dbc4bc3a9f746d862c45cb89d705aa10f187bb96c76001afab07a0d35ce60142"
[[package]]
name = "unicode-display-width"
version = "0.3.0"
@ -4689,6 +4743,33 @@ dependencies = [
"subtle",
]
[[package]]
name = "ureq"
version = "3.3.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "dea7109cdcd5864d4eeb1b58a1648dc9bf520360d7af16ec26d0a9354bafcfc0"
dependencies = [
"base64 0.22.1",
"getrandom 0.4.2",
"log",
"mime_guess",
"percent-encoding",
"ureq-proto",
"utf8-zero",
]
[[package]]
name = "ureq-proto"
version = "0.6.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "e994ba84b0bd1b1b0cf92878b7ef898a5c1760108fe7b6010327e274917a808c"
dependencies = [
"base64 0.22.1",
"http",
"httparse",
"log",
]
[[package]]
name = "url"
version = "2.5.8"
@ -4701,6 +4782,12 @@ dependencies = [
"serde",
]
[[package]]
name = "utf8-zero"
version = "0.8.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "b8c0a043c9540bae7c578c88f91dda8bd82e59ae27c21baca69c8b191aaf5a6e"
[[package]]
name = "utf8_iter"
version = "1.0.4"

6
Cargo.toml
View File

@ -1,5 +1,6 @@
[workspace]
members = ["crates/*"]
exclude = ["radicle-lfs-transfer"]
resolver = "2"
[workspace.package]
@ -21,8 +22,10 @@ rust-version = "1.88.0"
[workspace.dependencies]
amplify = { version = "4.0.0", default-features = false }
backtrace = "0.3.75"
base64 = "0.21.3"
bstr = "1.3"
bytes = "1.11.1"
chacha20poly1305 = "0.10"
chrono = { version = "0.4.26", default-features = false }
colored = "2.1.0"
crossbeam-channel = "0.5.6"
@ -33,6 +36,7 @@ fastrand = { version = "2.0.0", default-features = false }
git2 = { version = "0.20.4", default-features = false, features = ["vendored-libgit2"] }
gix-hash = { version = "0.25.0", default-features = false, features = ["sha1"] }
gix-packetline = { version = "0.21.1", default-features = false }
hkdf = "0.12"
human-panic = "2.0.6"
humantime = "2.3"
indexmap = { version = "2", default-features = false }
@ -68,6 +72,7 @@ radicle-windows = { version = "0.1", path = "crates/radicle-windows" }
schemars = { version = "1.0.4", default-features = false }
serde = { version = "1.0", default-features = false }
serde_json = "1.0"
sha2 = "0.10.9"
shlex = "1.1.0"
signature = "2.2"
snapbox = "1.2"
@ -75,6 +80,7 @@ sqlite = "0.37.0"
tempfile = "3.3.0"
thiserror = { version = "2", default-features = false }
uds_windows = "1.1.0"
ureq = { version = "3", default-features = false }
windows = "0.62"
winpipe = "0.1.1"
winsplit = "0.1.0"

261
LFS-IPFS.md Normal file
View File

@ -0,0 +1,261 @@
# Git LFS support, backed by IPFS
This is a fork of [Radicle's `heartwood`][heartwood] adding Git LFS (large file) support,
with large file content stored on each contributor's own local IPFS node rather than on a
central server.
[heartwood]: https://github.com/radicle-dev/heartwood
## Why not a server?
Radicle's own FAQ notes that its first-generation protocol was built on IPFS and moved away
from it — IPFS is a content-addressed store, and repositories are mutable, so it wasn't a fit
for Radicle's core storage. That objection doesn't apply here: Git LFS's own pattern already
separates a small, mutable pointer file (which Radicle replicates exactly as it replicates any
other git object) from immutable large-file content addressed by hash — which is precisely
what IPFS is good at. This fork only uses IPFS for that second part.
There's no server anywhere in this design: every peer uses their own local IPFS node, and the
file-CID mapping travels with the repository itself, avoiding a single point of failure.
## How it works
- **The mapping**: a JSON git note, attached to each LFS pointer file's own git blob hash (not
the LFS oid — a different value), e.g. `{"v":1,"cid":"Qm..."}` for a public-repo object.
Written locally to `refs/notes/rad-lfs/local` rather than the bare `refs/notes/rad-lfs`
(writing to the bare ref directly causes a git ref D/F conflict once a peer's note has been
fetched — see the `rad lfs init` push/fetch refspecs below); `rad lfs init`'s push refspec
maps that local ref to the bare `refs/notes/rad-lfs` name on the remote, which is what other
peers then fetch as `refs/notes/rad-lfs/<your-peer-id>`. Verified against this repo's actual
replication code (`references_of` in `crates/radicle/src/storage/git.rs`) that Radicle
replicates every ref under a peer's namespace except `refs/tmp/heads/*` — there's no fixed
allowlist, so these notes refs replicate the same way `refs/heads`/`refs/tags`/`refs/cobs` do,
once the push/fetch refspecs below are configured.
- **On commit**: a pre-commit hook (installed by `rad lfs init`) computes every staged
LFS-tracked file's oid/size and passes them all to a single `rad lfs precommit` call, which
adds each one to your local IPFS daemon, pins it, and writes the note above — encrypting the
content first if the repository is private (see below). Batched into one call (rather than one
`rad lfs store` call per file) specifically so a private repo's passphrase is only prompted
for once per commit, not once per file.
- **On push**: `rad lfs init` configures `remote.rad.push` with *two* refspecs -- one for your
branches (`+refs/heads/*:refs/heads/*`), one for the notes mapping
(`+refs/notes/rad-lfs/local:refs/notes/rad-lfs`) -- so a single bare `git push rad` pushes
your commit and its notes mapping together. This wasn't always true: earlier builds only configured the notes
refspec, so a bare `git push rad` silently pushed *only* the mapping, leaving the commit
itself unpushed -- the single most common way to end up with content that looks committed but
that a collaborator can't fetch (`no CID recorded for oid ...`). Fixed, but only takes effect
once `rad lfs init` has been (re-)run with a build that includes the fix -- see
[Troubleshooting](#troubleshooting) below if you're still seeing the old behavior.
- **The actual bytes** move through [`rad-lfs-transfer`][rad-lfs-transfer], a Git LFS custom
transfer agent invoked automatically by `git`/`git-lfs` during a push or checkout — see that
repo for details. It's a separate binary/repository since it has nothing Radicle-specific in
it (no identity, no keys, no encryption). Uploads shell out to `rad lfs store` per object
(a no-op re-upload if the object already has a note, e.g. one your own pre-commit hook already
wrote). Downloads go through `rad lfs fetch-batch` instead of `rad lfs fetch` directly: since
git-lfs's custom-transfer protocol requests objects one at a time rather than all upfront,
`rad-lfs-transfer` spawns `fetch-batch` once per `git lfs pull`/checkout session and keeps it
alive for every object in that session, so a private repo's passphrase is prompted for once
per session rather than once per file.
- **Seeding**: `rad seed` now also pins a repository's known LFS objects in your local IPFS
node (mirroring "seeding = keep a full copy"); `rad unseed` unpins them, except any CID also
referenced by another currently-seeded repository's own LFS notes (e.g. two repositories that
happen to track byte-identical content, which hashes to the same CID) -- unpinning those would
silently break the other repository. There's no persistent pin refcount kept anywhere; this is
recomputed from every other seeded repository's notes each time `rad unseed` runs.
- **Recovering from a skipped pin**: if the pre-commit hook didn't run (`git commit --no-verify`)
or soft-skipped because `ipfs`/`rad` weren't on `PATH` yet, a file can end up committed as a
valid LFS pointer with no corresponding note -- nothing pins it anywhere. Run `rad lfs
backfill` to retroactively pin every LFS-tracked file at HEAD that's missing one, using
content from Git LFS's own local object cache (or the working-tree file, as a fallback).
## Encryption for private repositories
Radicle's `Visibility::Private` is enforced entirely by access control at the replication
layer — git objects for a private repo are plain, unencrypted objects, protected only by
"unauthorized peers are never given them" (see `crates/radicle/src/identity/doc.rs`). That
protection has no jurisdiction over IPFS's own block-exchange layer: once a legitimate
collaborator's IPFS daemon pins content and joins the network, anyone who obtains the CID by
any means can fetch the plaintext directly. So for private repositories, `rad lfs store`
encrypts content before it ever reaches IPFS:
- A random per-object content key (CEK) encrypts the file (XChaCha20-Poly1305).
- The CEK is wrapped once per authorized recipient (repo delegates + the private allow-list),
using ECDH between the committer's and each recipient's Radicle identity key (Ed25519,
directly on the curve — no separate encryption keypair to manage) plus HKDF, so only
identities Radicle already considers authorized for the repo can decrypt.
- The wrapped keys travel in the same git note as the CID — no separate key-distribution
mechanism, no server.
**This requires a signer capable of key agreement (ECDH), not just signing.** ssh-agent — the
normal day-to-day flow after `rad auth` — can sign but can't expose the key material ECDH
needs; that's a hard limitation of the standard ssh-agent protocol (it only implements
signing, there's no request type for general key agreement), not something to configure
around. `rad lfs store`/`rad lfs fetch`/`rad lfs rekey` on a *private* repository fall back to
whichever of these applies, in order:
1. An unencrypted keystore, if that's how you've set things up.
2. `RAD_PASSPHRASE`, if set in the environment.
3. **An interactive passphrase prompt**, if connected to a terminal — the same fallback
`rad`'s other commands already use when ssh-agent isn't available or the key isn't
registered with it. This covers the common case (an interactive `git commit` triggering
the pre-commit hook, which inherits the terminal) with no env var needed at all.
Only a genuinely non-interactive context (CI, a script with no TTY, `RAD_PASSPHRASE` unset)
hits a hard failure — and it fails fast with a clear message, rather than hanging waiting on a
prompt that can't be answered. Public repositories are entirely unaffected — no keys, no ECDH,
same plaintext behavior as before.
### Granting access to existing objects: `rad lfs rekey`
Radicle's access control is retroactive: once a DID is authorized, it can replicate a
repository's entire history, including everything from before it was added. Without extra
handling, this encryption design would *not* match that: a newly-authorized collaborator could
see a private repo's full git history but be permanently unable to decrypt any LFS object
committed before they were granted access, since those objects' wrapped-key lists only include
whoever was authorized at the time.
`rad lfs rekey` closes that gap. Run by anyone *already* authorized (it can't be self-served by
the new collaborator — they have nothing to decrypt yet), it walks every encrypted object,
recovers each one's content key using the runner's own existing access, and adds a wrapped-key
entry for any currently-authorized DID that's missing one. It never re-encrypts content or
touches IPFS — it's a pure git-notes operation, so it's fast and doesn't need the original
file. Run it after adding a new delegate or allow-listing a new DID on a private repository, if
you want them to have access to previously-committed LFS objects (not automatic — see
[Prerequisites](#prerequisites)/[Use](#use) below for the full flow).
**Still explicitly not solved**: revocation. A formerly-authorized peer who already decrypted
an object can't be made to un-know its content key — the same fundamental limitation as
Radicle's own replication-based access control (already-replicated git objects can't be
un-seen either), not a regression introduced by this design.
[rad-lfs-transfer]: https://git.hswro.org/mab122/radicle-lfs-transfer
## Prerequisites
- A [Rust toolchain](https://rustup.rs) (stable; see `rust-toolchain.toml` for the exact
version this workspace pins), Git, and OpenSSH.
- [Git LFS](https://git-lfs.com) (`git-lfs` on your `PATH`) — only needed to *use* `rad lfs`,
not to build or install anything.
- A local IPFS (Kubo) daemon — also only needed to *use* `rad lfs`, not to build/install.
Install [Kubo](https://docs.ipfs.tech/install/) and make sure `ipfs` is on your `PATH`.
On Arch Linux:
```sh
sudo pacman -S --needed rust git openssh base-devel # build
sudo pacman -S --needed git-lfs kubo # use `rad lfs`
```
## Build & install
Follows upstream `heartwood`'s own install convention (`cargo install --root ~/.radicle`, one
shared install root for the whole stack), with `radicle-lfs-transfer` added the same way:
```sh
git clone --branch rad-lfs-ipfs --recurse-submodules \
ssh://git@git.hswro.org:9022/mab122/radicle-heartwood-lfs.git
cd radicle-heartwood-lfs
cargo install --path crates/radicle-cli --force --locked --root ~/.radicle
cargo install --path crates/radicle-node --force --locked --root ~/.radicle
cargo install --path crates/radicle-remote-helper --force --locked --root ~/.radicle
cargo install --path radicle-lfs-transfer --force --locked --root ~/.radicle
```
Add `~/.radicle/bin` to your `PATH` (e.g. in `~/.bashrc`/`~/.zshrc`):
```sh
export PATH="$HOME/.radicle/bin:$PATH"
```
Confirm everything landed correctly:
```sh
rad --version
command -v rad-lfs-transfer # just needs to resolve; it only speaks git-lfs's stdin/stdout protocol, no --help output
```
If you already cloned without `--recurse-submodules`, run `git submodule update --init` before
building — otherwise `radicle-lfs-transfer/` will be an empty directory and that last
`cargo install` step will fail.
## Run
Nothing extra to run for `rad`/`radicle-node` themselves — use them exactly as upstream
(`rad auth`, `radicle-node`, etc.; see the main [README](README.md) and
[`docs/`](https://app.radicle.xyz/nodes/seed.radicle.xyz/rad:zzz)). The only new moving part is
the IPFS daemon, needed only when you use `rad lfs`:
```sh
ipfs init # first time only, creates ~/.ipfs
ipfs daemon # leave running in the background while you use `rad lfs`
```
## Use
Inside a Radicle repository you're a contributor on (i.e. it already has a `rad` remote from
`rad init`/`rad clone`):
```sh
# One-time per repository (with the IPFS daemon above already running)
rad lfs init
# Then it's just Git LFS, as normal
git lfs track "*.psd" # or whatever large-file patterns you need
git add .gitattributes
git add my-large-file.psd
git commit -m "Add asset" # pre-commit hook pins it to IPFS and records the CID here
git push rad # pushes your commit(s) and the refs/notes/rad-lfs mapping together
```
Someone else cloning the same repository:
```sh
rad clone rad:<repo-id>
cd <repo>
rad lfs init # one-time, same as above (needs their own IPFS daemon running)
git lfs pull # fetches large files via IPFS instead of git
```
`rad seed`/`rad unseed` additionally pin/unpin a repository's known LFS objects in your local
IPFS node, so seeding a repository keeps a full copy of its large files too, not just its git
history.
**On a private repository**, committing from a terminal you'll be prompted for your passphrase
if needed (see [Encryption for private repositories](#encryption-for-private-repositories)
above) — no extra setup required. After granting a new collaborator access
(`rad id update --allow <did>` or adding them as a delegate), have an already-authorized
collaborator run `rad lfs rekey` and push, so the new collaborator can decrypt
previously-committed LFS objects too, not just ones committed after they were added.
### What happens without an IPFS daemon running
- `rad lfs init` checks upfront and refuses with a clear message
(`no IPFS (Kubo) daemon reachable at http://127.0.0.1:5001 — start one with 'ipfs daemon'`)
rather than silently continuing.
- `rad seed`/`rad unseed` print a warning and skip pinning/unpinning — seeding/unseeding itself
still succeeds.
- **Nothing else in this fork needs IPFS at all.** Building, installing, and using `rad` for
anything other than `rad lfs`/pinning works exactly like upstream `heartwood`, with zero IPFS
dependency.
## Troubleshooting
| Symptom | Cause / fix |
|---|---|
| `` `rad lfs init` must be run inside a Radicle repository working copy `` | Run it inside a directory that already has a `rad` remote (from `rad init` or `rad clone`), not a plain git repo. |
| `Git LFS is not installed (the 'git-lfs' binary was not found on your PATH)` | Install [Git LFS](https://git-lfs.com). |
| `no IPFS (Kubo) daemon reachable at ...` | Start one: `ipfs daemon` (see [Run](#run) above). |
| `rad-lfs-transfer: command not found` during `git lfs push`/`pull` | `cargo install --path radicle-lfs-transfer` didn't complete, or `~/.radicle/bin` isn't on `PATH`. |
| Large file didn't get pinned (commit went through, but no `rad-lfs` note) | The pre-commit hook silently skips pinning if the `ipfs`/`rad` CLIs aren't on `PATH` — check `command -v ipfs` / `command -v rad`. Once available, run `rad lfs backfill` to pin it retroactively rather than needing to re-commit. |
| Committed with `git commit --no-verify` (or the pre-commit hook was skipped some other way) and now nothing's pinned | Run `rad lfs backfill` — it walks every LFS-tracked file at HEAD and pins whichever ones are missing a note, without needing a new commit. |
| `private-repo LFS operations need to perform a key-agreement (ECDH) operation ...` | Your keystore is encrypted, ssh-agent can't do key agreement (protocol limitation), and no passphrase is available — normally you'd just be prompted interactively; this only happens in a non-interactive context (no TTY, e.g. CI). Set `RAD_PASSPHRASE`, or use an unencrypted keystore, or run it from a terminal. |
| `you are not an authorized recipient of this object` | Either genuinely unauthorized (not a delegate/allow-listed DID for this private repo), or authorized *after* this specific object was committed and nobody has run `rad lfs rekey` since — ask an already-authorized collaborator to run it and push. |
| `fatal: couldn't find remote ref refs/notes/rad-lfs` on `git fetch`/`git pull` (breaks *all* fetching, not just LFS) | Fixed — but if this repo had `rad lfs init` run against an older build, re-run `rad lfs init` once to replace the stale, now-invalid fetch refspec it configured (see `NOTES-lfs-notes-fetch-bug.md` for the full story). |
| Plain `git push rad` (no branch name) doesn't push your commits, only the notes ref | Your repository was set up with an older build: `rad lfs init` used to only configure a push refspec for `refs/notes/rad-lfs`, not the branch, so a bare push covered only that. Fixed — re-run `rad lfs init` once (harmless, idempotent) to add the missing `+refs/heads/*:refs/heads/*` push refspec, then a bare `git push rad` pushes both together. Until then, push both explicitly: `git push rad <branch>` for the commit, then `git push rad` for the notes mapping. |
| Collaborator gets `no CID recorded for oid ...; the peer that has this object needs to push its refs/notes/rad-lfs ref` even though they can see your commit | You (or whoever committed the LFS file) pushed from a repository still missing the branch push refspec — see the row above. Re-run `rad lfs init`, push again, then have them `git fetch rad` (or re-clone/re-pull). |
## Status
This is a prototype, not upstream Radicle functionality. See the commits on the
`rad-lfs-ipfs` branch for the full change set relative to upstream `heartwood`.

362
NOTES-lfs-notes-fetch-bug.md Normal file
View File

@ -0,0 +1,362 @@
# Bug: `git fetch rad` can never fetch `refs/notes/rad-lfs`
**RESOLVED** (2026-07-14). Fix: `refs/notes/rad-lfs` is now exposed per-peer
(`refs/notes/rad-lfs/<peer-id>`) instead of expecting one canonical bare ref, matching how
patch refs already work — see "Fix actually shipped" for the full writeup and verification.
**Follow-up "Bug #2" also RESOLVED** (2026-07-14, same day) — turned out to be two separate
things, neither a bug in the fix above: a stale, unrelated system-wide `git-remote-rad`
install being silently used in manual ad-hoc testing (not a real pipeline issue), plus one
genuinely new bug (a cross-device `rename()` failure on LFS download) found only once the
real pipeline actually ran with the fix. See "Root cause of Bug #2, actually found" at the
very bottom for the full story — verified live against the production deploy pipeline on
mbator.pl, which completed successfully end-to-end.
Kept the original report and the mid-investigation "Follow-up" section below unedited, for
context on how the debugging actually went.
Found while deploying `rad-lfs-ipfs` to mbator.pl for the blog pipeline (2026-07-14).
Blocks the whole LFS-over-IPFS flow end-to-end, not just a cold-start edge case.
## Symptom
After `rad lfs init` in any repo, `git fetch rad` (and therefore `git pull`, and anything
that does the equivalent, e.g. `deploy.sh`'s `git fetch rad && git reset --hard rad/master`)
fails every time with:
```
fatal: couldn't find remote ref refs/notes/rad-lfs
```
This isn't specific to a fresh repo where nobody has pushed an LFS object yet — it
reproduces even after a real push has created the note and it's fully present in the
receiving node's storage.
## Root cause
Two places are involved:
1. **`crates/radicle-cli/src/commands/lfs/init.rs:134`** — `rad lfs init` configures the
client-side fetch/push refspec as a bare, non-namespaced ref:
```rust
let notes_refspec = format!("+{NOTES_REF}:{NOTES_REF}"); // +refs/notes/rad-lfs:refs/notes/rad-lfs
```
2. **`crates/radicle-remote-helper/src/list.rs`, `for_fetch()`** (lines 38-79) — this is
what actually determines which refs `git-remote-rad` advertises as fetchable for a
plain (non-namespaced) `git fetch rad`. It only ever lists:
- `HEAD`
- `refs/heads/*` and `refs/tags/*` (canonicalized across delegates)
- patch refs (via `patch_refs()`)
It has **no knowledge of `refs/notes/rad-lfs` at all**. Notes only exist per-peer,
under `refs/namespaces/<peer-id>/refs/notes/rad-lfs` (confirmed directly on a live
node: `git for-each-ref` in the node's storage shows
`refs/namespaces/z6Mkkph.../refs/notes/rad-lfs`, never a bare `refs/notes/rad-lfs`).
`refs/heads/*` gets a canonical, non-namespaced view via delegate-threshold
resolution; `refs/notes/rad-lfs` has no equivalent resolution step, so it's simply
never in the list the remote helper offers — regardless of what refspec the client
has configured in `.git/config`.
So the client asks for a ref the server-side listing never advertises, and `git fetch`
(which validates requested refspecs against the advertised ref list) fails hard for the
*entire* fetch, not just that one ref — meaning a repo with `rad lfs init` run can't
`git fetch`/`git pull` *anything* anymore, heads included, until this is fixed.
## Suggested direction for a fix
`for_fetch()` needs to expose some resolved form of the LFS notes ref for the
non-namespaced case, analogous to how `refs/heads/*` gets canonicalized. Options,
roughly in order of how much they change the design:
- **Simplest**: expose *all* peers' notes refs individually under some fetchable name
(e.g. `refs/notes/rad-lfs/<peer-id>`), and change the LFS tooling (`rad lfs
store`/`fetch`/`rekey`) to read/merge across all of them instead of expecting one
canonical ref. Matches how git notes are designed to be merged anyway.
- **Closer to current design**: pick one canonical note per commit the same way heads
get a canonical value (e.g. delegate-threshold agreement, or "the delegate's own
namespace" the way `for_push` already restricts pushes to `profile.id()`'s own
namespace) and expose that as the bare `refs/notes/rad-lfs`.
- Either way, `init.rs`'s refspec (`+refs/notes/rad-lfs:refs/notes/rad-lfs`) is probably
fine as-is *once* the server side actually advertises something at that path — the
bug is really in `list.rs`, not `init.rs`.
## Confirmed working around this bug (verified live on mbator.pl, 2026-07-14)
Everything upstream and downstream of this one step works:
- Fork built clean (`cargo install` for `radicle-cli`, `radicle-node`,
`radicle-remote-helper`, `radicle-lfs-transfer`) at commit `33ffa122`.
- `rad lfs init` succeeds and configures the pre-commit hook, transfer agent, and (broken)
refspec correctly per its own logic.
- A real private-repo LFS push (`git push rad master`, encrypted via ECDH, `RAD_PASSPHRASE`
from an ssh-agent-backed key) succeeded and replicated to a seed node.
- Node-to-node replication of the note itself works fine at the storage layer — the
receiving node's `refs/namespaces/<pusher>/refs/notes/rad-lfs` is populated correctly.
- The failure is *only* in the working-copy-level `git fetch rad` being unable to see it.
## Also found along the way (separate, minor)
Two `radicle-cli` crashes (not yet investigated, crash reports written to `/tmp/report-*.toml`
on the VPS at the time):
- `rad node status` crashed once (worked on retry).
- `rad seed` (no args, to check a repo's local seeding policy) crashed consistently.
## Deployment state left in place (mbator.pl)
- `blog` user has a working IPFS (Kubo) container, the fork built at `~/.radicle/bin`,
`radicle-node-blog.service`/`blog-radicle-watch.service`/`blog-deploy.service` all
pointed at the new binaries with `PATH`/`KUBO_API_URL` set correctly.
- Added an explicit `connect` entry for the local `seed` node
(`z6MkmUF9KWFBQzYXBeNJk1bXXQGzaATnEsWxmbUYt54fyEC6@127.0.0.1:8776`) to
`/home/blog/.radicle/config.json` — the private `blog` node had no route to the local
public seed by default (`connect: []`, generic `preferredSeeds`), so `rad sync` timed
out until this was added. Worth keeping regardless of the notes bug.
- Public `seed` node/`/usr/bin/radicle-node`/`radicle-httpd` untouched throughout.
- A harmless test commit (`3f650d2`, "infra: LFS-over-IPFS smoke test (safe to remove)")
is sitting on top of the blog repo's real history in `~/Blog/Blog` and pushed to the
network — adds `assets/_infra-test/lfs-ipfs-smoketest.jpg` (LFS-tracked, not referenced
from any post, won't appear anywhere on the live site). Safe to leave or revert
whenever convenient.
## Next step once fixed
Re-run `git fetch rad` in `/home/blog/repo` on the VPS (as `blog`) — if the fix resolves
the notes ref, that fetch (and the LFS smudge it triggers) is the last remaining piece to
confirm the whole IPFS content-fetch path end-to-end.
## Follow-up: fix deployed, uncovered a second, more specific bug (2026-07-14, same day)
Rebuilt both the local (`~/Blog/Blog`) and VPS (`/home/blog/repo`) checkouts against
`ff762ef8` (the fix + the interactive-passphrase commit) and re-ran `rad lfs init` to pick
up the migrated wildcard refspec (`+refs/notes/rad-lfs/*:refs/notes/rad-lfs/*`; the old
exact-match entry needed manual removal once — `git update-ref -d refs/notes/rad-lfs` — on
the author's machine where a stale *local* plain note ref pre-existed and D/F-conflicted
with the incoming namespaced one; the `remove_refspec_if_present` migration in `init.rs`
correctly stripped the *refspec config*, but doesn't (and structurally can't, from
`init.rs`) also clean up a ref that already exists on disk from before).
**On the author's machine**: `git fetch rad` now genuinely works — pulls
`refs/notes/rad-lfs/<peer>` for real, exactly the repro case from the original bug.
Confirmed fixed there.
**On the VPS (`blog` user, second peer/node)**: same `rad lfs init` migration, but the
actual fetch behaves inconsistently:
- `git ls-remote rad` correctly lists the note:
`c4538c2d... refs/notes/rad-lfs/z6MkkphWkrBU3Buhh4ZN8Gtt9KLzn2DUsx3hZhhXirK7XAyn`
- `git fetch rad` (relying on the configured wildcard refspec) silently skips it — only
reports `master` as `[up to date]`, no attempt at the notes ref, no error.
- An **explicit** fetch of that exact literal ref name (copy-pasted straight from the
`ls-remote` output above) still fails outright:
```
$ git fetch rad refs/notes/rad-lfs/z6MkkphWkrBU3Buhh4ZN8Gtt9KLzn2DUsx3hZhhXirK7XAyn:refs/notes/rad-lfs/z6MkkphWkrBU3Buhh4ZN8Gtt9KLzn2DUsx3hZhhXirK7XAyn
fatal: couldn't find remote ref refs/notes/rad-lfs/z6MkkphWkrBU3Buhh4ZN8Gtt9KLzn2DUsx3hZhhXirK7XAyn
```
So `git-remote-rad`'s ref listing is inconsistent *between its own `list` output and what
`fetch` actually resolves against*, for the identical ref, in the identical process
invocation style — `ls-remote` and `fetch` are supposed to both go through the remote
helper's `list` verb (that's the whole point of `for_fetch()` being one shared function),
so seeing them disagree points at something conditional in how `list` is invoked (fetch
vs. ls-remote may pass different capabilities/args to the helper, e.g. a `list for-push`
vs. plain `list`, or some caching), rather than `for_fetch()`'s ref-selection logic itself
(which is presumably fine, given `ls-remote` gets it right).
Confirmed the underlying data is genuinely present and correct the whole time (verified
directly in `blog`'s node storage, bypassing git-remote-rad entirely):
```
$ git -C /home/blog/.radicle/storage/z2Z16c9C5BaqSQPfbVrGPGG68wAx3 for-each-ref | grep lfs
c4538c2d... commit refs/namespaces/z6MkkphW.../refs/notes/rad-lfs
```
and the note content itself has the right CID and correctly lists `blog`'s own node DID
(`z6Mkpqi56U6...`) as an authorized decrypt recipient. So this is purely a
`git-remote-rad` transport/listing bug, not a storage/replication/encryption issue.
**Not yet root-caused** — flagging with the concrete repro above rather than guessing
further at the Rust internals blind. Two machines, two outcomes, same binary version, same
refspec, same underlying data: something about the *fetch* invocation path differs from
the *ls-remote* invocation path in a way that changes what the helper's `list` returns or
how the client matches against it. Worth checking: does `list.rs` (or whatever calls
`for_fetch()`) get invoked with different arguments for `list` vs. `list for-push` vs. a
fetch-triggered `list`, and does the *local* `remote.<rad>.fetch` refspec (not just the
server-side listing) factor into which lines the helper actually returns — i.e. is there a
second, client-driven filtering step somewhere that behaves differently between the two
call sites?
## Fix actually shipped
Went with the "simplest" option from the list above, since it's also the semantically
correct one: `refs/heads`/`refs/tags` get a canonical, non-namespaced value because only
*delegates'* view matters for canonical history — a whole separate subsystem (canonical-ref
computation on push/sync) maintains that. Notes have no equivalent: any peer, not just
delegates, can commit an LFS-tracked file and write a note for it, so there's no single
"correct" value to resolve to. Building delegate-consensus canonicalization for notes would
also have been semantically wrong, not just more work — it would silently drop non-delegate
contributors' LFS objects from ever being discoverable by anyone else.
- `crates/radicle-remote-helper/src/list.rs`, `for_fetch()`: now also lists every peer's
`refs/notes/rad-lfs`, individually, as `refs/notes/rad-lfs/<peer-id>` (new `lfs_notes_refs`
helper, using `stored.remotes()` + `stored.reference_oid()`). `for_push()` needed no change
— turns out it's purely informational (used for the `list for-push` git protocol handshake,
i.e. fast-forward computation), not an enforcement gate; the actual `push_ref`/`send-pack`
call already accepts any refspec regardless of what `for_push()` lists, which is why notes
pushing already worked despite `for_push()` only ever listing `refs/heads`/`refs/tags`.
- `crates/radicle-cli/src/commands/lfs/init.rs`: fetch refspec becomes a wildcard
(`+refs/notes/rad-lfs/*:refs/notes/rad-lfs/*`); push refspec is unchanged (`git push`
lands under the pusher's own namespace server-side regardless of the local ref name, same
as `refs/heads/*`, so no wildcard needed there). Added a migration step
(`remove_refspec_if_present`) that strips the old broken non-wildcard fetch refspec from
already-configured repos — otherwise re-running `rad lfs init` would've just added the
correct wildcard *alongside* the still-broken exact-match entry, which would still break
the whole fetch (git fails the entire fetch if *any* requested refspec isn't advertised).
- `crates/radicle-cli/src/lfs_crypto.rs`: new `find_envelopes`/`all_note_targets`/`find_cids`
read across every notes ref (local + every fetched peer), not just one. Notes sharing the
same `cid` (e.g. an original commit plus a later `rekey` that added recipients under a
*different* peer's namespace) have their recipient lists safely unioned. Notes with
*different* `cid`s for the same object (only possible if two peers independently encrypted
byte-identical content — same oid/size — producing different ciphertext each time, since
encryption uses a fresh random key/nonce) are kept as separate candidates; `rad lfs
fetch`/`rekey` try each in turn rather than assuming there's only ever one.
- `crates/radicle-cli/src/commands/lfs/fetch.rs`, `rekey.rs`, `ipfs.rs` (`lfs_cids`, used by
`seed`/`unseed` pinning): all switched from single-ref `repo.notes(Some(NOTES_REF))` /
`find_note` calls to the new multi-ref helpers above. `store.rs` needed no read-side change
(write-only, always writes to the local plain ref, which is correct as-is).
### Verified for real (not the storage-copy shortcut used for the base feature's original
tests — this bug lives specifically in the `git-remote-rad` transport, so the fix had to be
exercised through the actual thing)
Built fresh `rad`/`git-remote-rad`/`radicle-node`/`rad-lfs-transfer`, created two-then-three
real Radicle identities/profiles, a real private repo with an allow-list, and used the
**real** `rad://` remote (`git remote add rad rad://<rid>` for fetch, matching exactly what
`rad init` itself configures — an earlier test attempt using a peer-scoped URL for both fetch
and push accidentally bypassed the bug entirely by routing into `for_fetch()`'s *namespaced*
branch, which was never broken; had to redo it with the correct canonical URL to actually
exercise the bug/fix).
- `git ls-remote rad` on the canonical URL now advertises `refs/notes/rad-lfs/<peer>` instead
of nothing.
- A genuine `git fetch rad` (no explicit refspec workaround) succeeds and actually pulls the
peer's notes ref down locally — this is the exact command from the bug report that used to
fail with `fatal: couldn't find remote ref refs/notes/rad-lfs`.
- `git lfs pull` after that fetch correctly decrypts the private object end-to-end
(SHA-256-verified).
- Re-tested the rekey scenario (grant a new peer access after the object was already
committed) through the same real fetch path: denied before rekey, an authorized peer runs
`rad lfs rekey` + pushes, the new peer re-fetches and successfully decrypts.
### Also discovered along the way (separate, not fixed here — flagging, not scope-creeping)
Once `rad lfs init` adds *any* explicit `remote.<rad>.push` refspec (the notes one), a plain
`git push rad` (no branch argument) stops falling back to `push.default=simple` for the
current branch — git only pushes whatever's explicitly configured once any explicit refspec
exists. So after running `rad lfs init`, `git push rad` alone only pushes the notes ref;
`git push rad master` (explicit) is needed to also push the branch. This isn't new — it was
already true before this fix, and both the original bug report's own reproduction and this
fix's testing always used an explicit branch name when pushing, so it never surfaced as a
problem in practice. Worth a real fix at some point (e.g. `rad lfs init` also ensuring an
explicit `+refs/heads/*:refs/remotes/rad/*`-equivalent push default is present, or documenting
"always `git push rad <branch>` explicitly" as a hard requirement), but out of scope for this
specific fetch-bug fix.
## Post-verification cleanup (2026-07-14, later still)
VPS only needs the compiled binaries, not a persistent build environment. After confirming
`/home/blog/.radicle/bin/{rad,radicle-node,git-remote-rad,rad-lfs-transfer}` are correct and
working (live deploy already succeeded, see above), removed the source checkout and Rust
toolchain to reclaim disk: `rm -rf /home/blog/radicle-heartwood-lfs /home/blog/.rustup
/home/blog/.cargo`, plus dropped the now-dangling `. "$HOME/.cargo/env"` lines from
`blog`'s `.bashrc`/`.profile`. Reclaimed ~3.2GB (9.7G → 6.5G used on `/`). Confirmed no
systemd unit referenced the source dir or cargo/rustup, and the binaries are dynamically
linked only against standard system libs (`ldd` shows just `libgcc_s`/`libm`/`libc`) — fine
to run standalone.
**Implication for next time a rebuild is needed**: the VPS has no source checkout or Rust
toolchain anymore. Re-deploying a new fix means redoing the copy-source-and-build steps
from scratch (or, better, sorting out `blog`'s lack of Forgejo SSH credentials so it can
`git clone`/`pull` properly instead of the source being hand-copied in — see Part 1 of Bug
#2 below for why that copy-instead-of-clone approach already caused one bit of git-history
drift).
## Root cause of Bug #2, actually found (2026-07-14, same day, later)
Got shell access to the VPS (`ssh mbator.pl`, passwordless sudo in a real tty) and reproduced
directly rather than guessing further at the Rust internals. It was **two separate things**,
not one — neither of them a bug in the fetch fix above.
### Part 1: not a code bug at all — a stale, unrelated binary
`sudo -u blog bash -c '...'` (a bare, non-login shell — the same style of ad-hoc command used
for all the manual reproduction so far) resolves `PATH` to the plain system default
(`/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin`), which has **no**
`~/.radicle/bin` in it. `which rad` in that context returns `/usr/local/bin/rad` — version
`1.8.0`, a completely unrelated, root-owned, March-2026 system-wide install, nothing to do
with this fork. Confirmed directly:
- With `PATH=/home/blog/.radicle/bin:$PATH` explicitly set: `git ls-remote rad` *and*
`git fetch rad` both correctly see and fetch `refs/notes/rad-lfs/<peer>` — full
`GIT_TRANSPORT_HELPER_DEBUG=1` trace shows `list` correctly advertising it and git correctly
issuing `fetch <oid> <ref>` for it, ending in `[new ref] refs/notes/rad-lfs/<peer>`.
- With the bare/default `PATH` (resolving to `/usr/local/bin/git-remote-rad`, the unrelated old
binary): `git fetch rad` produces **zero output at all** — exactly the silent-skip symptom
originally reported, byte for byte.
So `list`/`for_fetch()` never actually disagreed between `ls-remote` and `fetch` for the same
binary — the two commands were, at some point during manual testing, silently invoking two
*different* binaries. The real automated pipeline was never actually at risk from this: all
three blog-owned systemd units (`radicle-node-blog`, `blog-radicle-watch`, `blog-deploy`)
already have `Environment=...PATH=/home/blog/.radicle/bin:...` correctly set. Confirmed via
`systemctl cat` on all three. This was purely a manual-testing artifact.
### Part 2: a real, new bug — but only surfaces once you get past Part 1
`blog-deploy.service`'s last run before the fork rebuild had failed with the *original*
Bug #1 error (timestamped *before* the 16:10–16:14 UTC rebuild/restart) — meaning the real
pipeline, with its correct `PATH`, hadn't actually run even once against the fetch fix yet.
Triggering it manually (`sudo systemctl start blog-deploy.service`) surfaced a genuinely new
failure:
```
Downloading assets/_infra-test/lfs-ipfs-smoketest.jpg (692 B)
Error downloading object: ...: failed to copy downloaded file: cannot replace
".../ .git/lfs/objects/c5/e3/c5e3b0c1..." with "/tmp/rad-lfs-c5e3b0c1...":
rename /tmp/rad-lfs-c5e3b0c1... .../.git/lfs/objects/c5/e3/c5e3b0c1...: invalid cross-device link
```
The download and decryption had already succeeded by this point (692 B, matching exactly) —
this was purely the final handoff to git-lfs failing. Root cause:
`radicle-lfs-transfer/src/main.rs`'s `download_inner` reported a path under
`std::env::temp_dir()` (`/tmp`) back to git-lfs as the "complete" event's path. Git-lfs then
does an *atomic* `rename()` from that path into `.git/lfs/objects/...` — it does not fall back
to a copy if that fails. On this VPS, `/tmp` and the repository are on different
filesystems/mounts, so the rename hits `EXDEV` ("invalid cross-device link") and git-lfs just
errors out rather than retrying with a copy.
Fixed in `radicle-lfs-transfer` (commit `ac4a27c`): resolve `.git/lfs/tmp` (via
`git rev-parse --git-dir`, so this works from a subdirectory too, not just the repo root) and
download there instead, mirroring git-lfs's own existing convention for its own internal temp
downloads — guaranteed same-filesystem by construction, not by hoping `/tmp` happens to line
up. This does reintroduce a `git` binary runtime dependency for `rad-lfs-transfer` (previously
dropped once it stopped needing to read/write notes directly) — an acceptable tradeoff, since
anything using `rad`/git-lfs at all already requires git to be installed anyway. Submodule
pointer bumped in this repo at `a97484c9`.
### Verified live, for real, against production
Deployed the fix to the VPS (`blog` has no Forgejo SSH credentials, so copied the fixed
`main.rs` directly rather than provisioning new access; rebuilt in place with
`~/.cargo/bin`/`~/.radicle/bin` on `PATH`), then actually triggered `blog-deploy.service`:
```
● blog-deploy.service ... Active: inactive (dead) ...
Process: ExecStart=/home/blog/repo/deploy/deploy.sh (code=exited, status=0/SUCCESS)
[blog-deploy] deploying to /srv/mbator.pl...
[blog-deploy] deploy complete: 3f650d2d7e9accfba3468a885fc63984e18ceadb
```
Full success, end to end: `git fetch rad` pulled the notes ref for real, `git lfs pull`
downloaded and decrypted the private object, the site built, and it deployed. Confirmed the
checked-out asset's SHA-256 matches the original exactly
(`c5e3b0c1eb2d6e2f3ef98fe74e70fe431986dc2a401b1eae961e175b5a2164ca`). It doesn't appear under
`/srv/mbator.pl` because it's deliberately unreferenced by any post — expected, by the smoke
test's own design, not a bug.

198
NOTES-lfs-store-note-write-bug.md Normal file
View File

@ -0,0 +1,198 @@
# Bug: `rad lfs store` fails with "failed to write LFS note" — likely fallout from the `refs/notes/rad-lfs/*` migration
**RESOLVED** (2026-07-15). The leading hypothesis in this report — a git ref D/F conflict
between the bare `refs/notes/rad-lfs` (what `store.rs`/`rekey.rs` wrote to) and the namespaced
`refs/notes/rad-lfs/<peer>` (what fetch populates, including your own peer's copy fetched back
after a push) — was confirmed exactly, reproduced locally in the identical order that
triggers it (fetch a namespaced sibling in *first*, then attempt a local write). See
"Fix actually shipped" at the bottom for the full writeup, the fix, and local verification
(including the migration path for already-affected repos like this one).
`~/Blog/Blog` and the VPS checkout both need `rad lfs init` re-run once the fix is installed,
to run the migration and pick up the corrected push refspec — see the deployment steps at the
bottom before relying on `rad lfs store` there again.
## Repro
```
$ rad lfs store --oid <sha256 of a food jpg> --size <n> -- assets/food/boczekwziemniaczkach.jpg
✗ Error: failed to write LFS note
```
Exit code 1. No further detail is printed — `store.rs`'s only `.context("failed to write LFS
note")?` wraps the `repo.note(...)` call at the very end of `run()`
(`crates/radicle-cli/src/commands/lfs/store.rs:74`), and the CLI's top-level error printer only
shows the outermost context, not the full `anyhow` chain, so the actual libgit2/`radicle`
error underneath is not visible from the CLI alone.
Confirmed via `RUST_LOG=trace` that everything *before* the note write succeeds:
- the LFS pointer blob is written (`repo.blob(...)`, no error)
- `ipfs::check_daemon()` succeeds (local Kubo at `127.0.0.1:5001`, confirmed reachable)
- `ipfs::add()` succeeds — HTTP 200 from `/api/v0/add`, got back a CID
(`QmXshEtcDYxmXvpaF13otbysCeqzvhNS2U8GF1hr6qFxrc`)
- `ipfs::pin_all()` succeeds — HTTP 200 from `/api/v0/pin/add`
So the failure is isolated to the final `repo.note(&signature, &signature,
Some(ipfs::LFS_NOTES_REF), blob_oid, &message, true)` call. No network I/O happens after the
pin step (confirmed in the trace log — the error is printed immediately after the pin response,
with no further HTTP/socket activity), so this is a local storage-write failure, not a
network/IPFS-daemon issue. This also rules out "rad node isn't running" as the cause — it was
freshly restarted and confirmed connected to peers (`rad node status`) before this repro.
## Suspected root cause: `NOTES_REF` was never updated to match the per-peer ref migration
`crates/radicle-cli/src/lfs_crypto.rs:42`:
```rust
pub const NOTES_REF: &str = "refs/notes/rad-lfs";
```
This is the *unnamespaced* ref name, and it's what `store.rs` passes as the notes ref to write
into (`ipfs::LFS_NOTES_REF` re-exports this constant, see `ipfs.rs:28`).
But the fetch-side fix in `NOTES-lfs-notes-fetch-bug.md` (commit `1c65ee62`, "Fix: git fetch rad
can never fetch refs/notes/rad-lfs") deliberately moved away from one canonical bare ref to a
**per-peer** ref shape, `refs/notes/rad-lfs/<peer-id>`, and `rad lfs init` now configures the
remote fetch refspec accordingly:
```
+refs/notes/rad-lfs/*:refs/notes/rad-lfs/*
```
Confirmed on this machine, in the blog repo's working copy, right now:
```
$ git config --get-all remote.rad.fetch
+refs/heads/*:refs/remotes/rad/*
+refs/tags/*:refs/remotes/rad/tags/*
+refs/notes/rad-lfs/*:refs/notes/rad-lfs/*
$ git for-each-ref refs/notes/
c4538c2d... commit refs/notes/rad-lfs/z6MkkphWkrBU3Buhh4ZN8Gtt9KLzn2DUsx3hZhhXirK7XAyn
```
i.e. the **working copy** now has the note under the new namespaced shape (fetched down from
the remote, post-fix). But in the **storage repo** (`~/.radicle/storage/<rid>`, where
`store.rs`'s `repo.note()` call actually writes, via `NOTES_REF` = the old bare name):
```
$ git -C ~/.radicle/storage/z2Z16c9C5BaqSQPfbVrGPGG68wAx3 for-each-ref | grep notes
c4538c2d... commit refs/namespaces/z6MkkphW.../refs/notes/rad-lfs
```
— still the **old, unnamespaced** ref (`refs/notes/rad-lfs`, no peer suffix), holding the one
note that was successfully written earlier today (17:08:38, for the now-reverted
`_infra-test/lfs-ipfs-smoketest.jpg` smoke test — that first write succeeded because at that
point nothing had fetched/created the namespaced sibling ref yet).
So the repo now has **two differently-shaped note refs pointing at overlapping data**: the
legacy bare `refs/notes/rad-lfs` (what `store.rs` still writes to) in storage, and the new
namespaced `refs/notes/rad-lfs/<peer>` (what fetch/init now expect and manage) in the working
copy and presumably also fetchable from storage once something writes it there. This is the
same shape of problem already diagnosed once in `NOTES-lfs-notes-fetch-bug.md` — "a stale local
plain note ref pre-existed and D/F-conflicted with the incoming namespaced one" — just hit from
the **write** side (`store.rs`) this time instead of the fetch side. Given git's notes trees use
fanout-by-target-oid internally, and notes refs are just regular refs, having both a bare
`refs/notes/rad-lfs` and a `refs/notes/rad-lfs/<peer>` **coexist** is a classic D/F (file vs.
directory) ref conflict: `refs/notes/rad-lfs` can't simultaneously be a leaf ref and a directory
prefix for `refs/notes/rad-lfs/<peer>` in the same ref namespace. That would explain why the
very first note write (before any namespaced sibling existed) succeeded, and why the second one
(now that `rad lfs init` has created the namespaced ref shape) fails outright, silently, at
exactly the ref-write step.
**Not yet confirmed against the actual libgit2 error text** (the CLI swallows it — see above),
so treat the D/F-conflict theory as the leading hypothesis, not a confirmed fix target. Worth
checking next:
1. Add temporary `eprintln!("{:#}", e)` (or similar) around the `repo.note(...)` call in
`store.rs` to print the full `anyhow` chain instead of just the top context, to get the real
libgit2 error text.
2. If it is a D/F conflict: `NOTES_REF` in `lfs_crypto.rs:42` needs to move to the same
per-peer-namespaced shape `store.rs` writes with (probably `refs/notes/rad-lfs/<own-peer-id>`,
matching what `init.rs`'s migrated refspec expects), not just the fetch/init side.
3. Given the write already touched storage in a bad way, may need a one-off manual cleanup on
this machine (`git update-ref -d refs/namespaces/<peer>/refs/notes/rad-lfs` inside
`~/.radicle/storage/<rid>`, analogous to the working-copy cleanup already documented for the
fetch bug) once the write-path fix lands, to avoid the same ref existing in both old and new
shape indefinitely.
## Impact
Blocks committing any new file into an LFS-tracked path (`assets/food/*.jpg`,
`assets/_infra-test/*.jpg`) in the blog repo via the `rad-lfs-init`-managed pre-commit hook,
which calls the equivalent of this same `rad lfs store` command per staged LFS file. Worked
around for the immediate food-gallery migration by committing with `--no-verify` to skip the
hook (plain `git-lfs` still produces correct LFS pointers; only the IPFS pinning step is
skipped for those files until this is fixed).
## Fix actually shipped
The D/F-conflict hypothesis was exactly right, confirmed by reproducing it locally in the
precise order that triggers it: `rad lfs init` → commit+push a file (writes bare
`refs/notes/rad-lfs` locally, lands namespaced in storage) → `git fetch rad` (pulls that same
note back down as `refs/notes/rad-lfs/<own-peer-id>`, since the fetch fix advertises *every*
peer including yourself) → a second `rad lfs store` call now fails outright with the exact
reported "failed to write LFS note", because the working copy now has both a bare
`refs/notes/rad-lfs` (a leaf ref) and `refs/notes/rad-lfs/<peer>` (implying `refs/notes/rad-lfs`
is a directory) — git can't reconcile that. (Interesting side note: plain `git fetch` itself
hits the *same* conflict from the other direction and just refuses that one ref update cleanly
with "unable to update local ref" — it's specifically `repo.note()`/libgit2's ref-write path
in `store.rs`/`rekey.rs` that fails hard instead, which is why the CLI's error was so much less
informative.)
### The fix
Stop ever writing to the bare `refs/notes/rad-lfs` locally. `crates/radicle-cli/src/lfs_crypto.rs`
now has two constants instead of one:
- `NOTES_REF` (`refs/notes/rad-lfs`, unchanged) — the *remote-side* canonical name: push
destination, and what `list.rs`'s `lfs_notes_refs` looks for within each peer's namespace on
the server. Still correct and unchanged.
- `LOCAL_NOTES_REF` (new: `refs/notes/rad-lfs/local`) — what `rad lfs store`/`rad lfs rekey`
actually write to now. Living under the same `refs/notes/rad-lfs/` prefix as every fetched
peer ref means it's just another sibling leaf ref, structurally incapable of D/F-conflicting
with them — `local` isn't a valid peer ID string, so it can never collide with an actual
fetched peer namespace either.
`rad lfs init`'s push refspec becomes asymmetric: `+refs/notes/rad-lfs/local:refs/notes/rad-lfs`
(local source → bare remote destination — the destination staying bare is correct and
unchanged, since push already lands under the pusher's own namespace server-side regardless of
the local ref name, same as `refs/heads/*`). The fetch refspec is untouched
(`+refs/notes/rad-lfs/*:refs/notes/rad-lfs/*`), already correct from the previous fix.
**Migration for already-affected repos** (this one included): `rad lfs init` now also runs
`migrate_local_notes_ref()` — if a bare `refs/notes/rad-lfs` ref still exists locally, it reads
every note out, deletes the bare ref, then re-writes them all under `refs/notes/rad-lfs/local`.
Nothing is lost. (First implementation of this had its own bug, caught during local
verification: it tried to *create* `refs/notes/rad-lfs/local` while the bare ref still existed
— the exact same D/F conflict, self-inflicted, inside the migration meant to fix it. Fixed by
reading all notes into memory first, deleting the bare ref, *then* writing them to the new
location.) Also removes the old symmetric push refspec entry
(`+refs/notes/rad-lfs:refs/notes/rad-lfs`) the same way the earlier fetch-refspec migration
did, so re-running `rad lfs init` is enough to fully repair an already-configured repo — no
manual `git update-ref`/config surgery needed.
### Verified locally (reproduced the exact bug, then the exact fix, same session)
- Reproduced the failure precisely: fresh repo, commit+push (bare ref written), fetch (creates
the namespaced sibling), second `rad lfs store` call → `✗ Error: failed to write LFS note`,
byte-for-byte the reported symptom.
- Rebuilt with the fix, retried the *exact same failing call* in the *same repo* → succeeds,
both `refs/notes/rad-lfs/local` and `refs/notes/rad-lfs/<peer>` coexist with no conflict.
- Tested the migration path directly: a repo with a pre-existing bare `refs/notes/rad-lfs`
(content: `{"v":1,"cid":"Qmb..."}`) → `rad lfs init` → note correctly moved to
`refs/notes/rad-lfs/local` with identical content, bare ref gone.
- Full round-trip after the fix: committed a new file (pre-commit hook → `rad lfs store` →
writes `refs/notes/rad-lfs/local`), pushed (asymmetric refspec correctly maps it to the
bare remote ref), a second checkout fetched it and `rad lfs fetch`'d the content —
SHA-256 matched exactly.
- `rad lfs rekey` smoke-tested post-fix (it writes notes too, via the same `LOCAL_NOTES_REF`
now) — ran cleanly, no crash.
### Deployment needed on already-affected machines
Both `~/Blog/Blog` (author's machine) and the VPS blog repo hit this bug and need the fix
installed, followed by one `rad lfs init` re-run each (to migrate the existing bare ref and
pick up the corrected push refspec) before `rad lfs store`/committing new LFS files will work
cleanly again there. The files committed via the `--no-verify` workaround are fine as-is
(valid LFS pointers, just not yet IPFS-pinned) — re-adding/re-committing them normally (or
running `rad lfs store` on them directly) after the fix is installed will pin them retroactively.

60
NOTES-package-publish-todo.md Normal file
View File

@ -0,0 +1,60 @@
# Status: 10MB IPFS `cat` fix — code done and pushed; package build/publish still needs your input
## Done (verified, no further action needed)
- Root-caused and fixed the bug: `crates/radicle-cli/src/ipfs.rs`'s `cat()` used ureq's
`Body::read_to_vec()`, which silently caps reads at 10MB (a default meant for typical JSON
API responses, not LFS content). Any LFS object over 10MB failed with "the response body is
larger than request limit: 10485760" — this is exactly what broke `blog-deploy.service` on
the VPS today, on `assets/posts/hashcash/thumbnail.xcf` (29MB).
- Fix: raised the limit to 1GiB via ureq's `.with_config().limit(...)`. Commit `35f1f972`,
branch `rad-lfs-ipfs`.
- Verification performed:
1. Added a `#[cfg(test)]` regression test (`ipfs::tests::cat_round_trips_content_larger_than_ureqs_default_10mb_limit`,
`#[ignore]`d since it needs a live Kubo daemon) — a 15MB add+cat round-trip.
2. Confirmed it **fails** with the exact production error when the fix is reverted, and
**passes** with the fix in place.
3. Ran the full `radicle-cli` test suite (123 tests) — one pre-existing, unrelated failure
(`rad_help`, a stale `--help`-output snapshot missing the `lfs` subcommand — confirmed to
fail identically on the unmodified base commit, nothing to do with this change).
4. `cargo clippy` clean (one pre-existing warning in an unrelated file/crate).
5. **Live reproduction against the real file**: moved aside the cached LFS blob for
`thumbnail.xcf` in `~/Blog/Blog`, ran `rad lfs fetch` with the *old* installed binary —
reproduced the identical production error byte-for-byte. Ran it again with the *newly
built* binary — the 10MB error is completely gone; it now correctly proceeds to the next
step (prompting for a passphrase to decrypt the private-repo content, which is expected
behavior and unrelated to this bug — I did not attempt to supply a passphrase). Restored
the blog repo's `.git/lfs/objects` cache back to its original state afterward; no changes
left in `~/Blog/Blog`.
- Pushed to `origin` (the Forgejo remote, `ssh://git.hswro.org:9022/mab122/radicle-heartwood-lfs.git`),
branch `rad-lfs-ipfs`, commit `35f1f972..4ae9756a`.
## Not done — needs your call
I did **not** build or publish a package. Stopping here rather than guessing, because:
1. **No existing Forgejo-package publishing path in this repo.** There's a `build/Dockerfile`
that cross-compiles reproducible `.tar.xz` releases (4 targets: linux/macOS ×
x86_64/aarch64) via `cargo-zigbuild`, but nothing that uploads anywhere — no CI workflow,
no `curl`-to-package-registry script, nothing referencing Forgejo's package API. I don't
know what package type you want registered (generic package? a distro-specific one?), what
name/version scheme you use there, or the auth (token?) needed to publish to
`git.hswro.org`'s package registry. I deliberately did not go hunting through credential
stores (`~/.netrc`, etc.) to find one myself — that's exactly the kind of thing I should ask
about rather than dig for.
2. **The existing Dockerfile's package doesn't include `rad-lfs-transfer`** — only `rad`,
`git-remote-rad`, and `radicle-node` get copied into the release tarball. `rad-lfs-transfer`
is a separate crate/binary (`radicle-lfs-transfer/`) and is required for this fork's whole
LFS-over-IPFS feature to work on a fresh install — worth fixing in the Dockerfile too, but
I didn't want to change the packaging script's scope without confirming that's wanted.
3. **Platform scope is unclear.** The existing Dockerfile builds all 4 targets (needed for a
real public release); the VPS only needs `x86_64-unknown-linux-musl`. Building all 4 takes
meaningfully longer and needs the macOS SDK blob (`build/macos-sdk-11.3.tar.xz`) and Zig
cross-toolchain set up — I can do it, but wanted to confirm whether you want the full
release matrix or just what's needed to update the VPS.
If you want me to keep going on this (rather than picking it up in the session that already
knows this fork's packaging conventions), I'd need: the Forgejo package type/target and an
auth token (however you'd like to provide it — e.g. via an env var you set, not me searching
for it), confirmation on whether to add `rad-lfs-transfer` to the Dockerfile's package list,
and whether to build the full 4-target matrix or just linux/x86_64.

109
README.md
View File

@ -1,5 +1,114 @@
# ❤️🪵
> ## Fork: Git LFS support, backed by IPFS
>
> This is a fork of upstream [`radicle-dev/heartwood`](https://github.com/radicle-dev/heartwood)
> adding Git LFS (large file) support, with large file content stored on each contributor's own
> local IPFS node rather than a central server. Everything below the horizontal rule is
> upstream's own README, unchanged. See [`LFS-IPFS.md`](LFS-IPFS.md) for the full design,
> troubleshooting, and background — this section is just the quick start.
>
> The LFS byte-transfer logic lives in a separate, small companion repository:
> [**radicle-lfs-transfer**](https://git.hswro.org/mab122/radicle-lfs-transfer), included here
> as a git submodule.
>
> ### Dependencies (Arch Linux)
>
> ```sh
> # To build and install rad/radicle-node/radicle-lfs-transfer
> sudo pacman -S --needed rust git openssh base-devel
>
> # To actually use Git LFS (not needed to build/install anything)
> sudo pacman -S --needed git-lfs kubo
> ```
>
> On other distributions: a Rust toolchain (e.g. via [rustup](https://rustup.rs)), Git, OpenSSH,
> a C toolchain — and, only for using `rad lfs`, [Git LFS](https://git-lfs.com) and
> [Kubo](https://docs.ipfs.tech/install/).
>
> ### Zero to usable
>
> ```sh
> # Clone with the submodule
> git clone --branch rad-lfs-ipfs --recurse-submodules \
> ssh://git@git.hswro.org:9022/mab122/radicle-heartwood-lfs.git
> cd radicle-heartwood-lfs
>
> # Build & install rad, radicle-node, git-remote-rad, and rad-lfs-transfer to one place
> cargo install --path crates/radicle-cli --force --locked --root ~/.radicle
> cargo install --path crates/radicle-node --force --locked --root ~/.radicle
> cargo install --path crates/radicle-remote-helper --force --locked --root ~/.radicle
> cargo install --path radicle-lfs-transfer --force --locked --root ~/.radicle
>
> # Add the install root to your PATH (e.g. in ~/.bashrc / ~/.zshrc)
> export PATH="$HOME/.radicle/bin:$PATH"
>
> # Verify
> rad --version
> ```
>
> From here, use `rad` exactly as upstream describes below (`rad auth`, `rad init`, etc). The
> only new commands are `rad lfs init` (run once per repository you want large-file support in),
> `rad lfs rekey` (run after granting a new collaborator access to a **private** repository, so
> they can decrypt previously-committed LFS objects too), and `rad lfs backfill` (retroactively
> pins any LFS-tracked file that got committed without going through the pre-commit hook, e.g.
> `--no-verify`) — see [`LFS-IPFS.md`](LFS-IPFS.md) for those workflows and how private-repo
> content gets encrypted before it reaches IPFS.
> **Nothing above requires IPFS**; Git LFS support specifically needs a running `ipfs daemon`,
> and `rad lfs init` will tell you plainly if one isn't reachable rather than failing confusingly
> later.
>
> ### Quickstart cheatsheet
>
> One-time, per repository (with `ipfs daemon` already running in the background):
>
> ```sh
> rad lfs init
> git lfs track "*.psd" # or whatever large-file patterns you need
> git add .gitattributes
> ```
>
> Day to day — same as plain Git LFS:
>
> ```sh
> git add my-large-file.psd
> git commit -m "Add asset" # pre-commit hook pins it to IPFS, records the CID as a git note
> git push rad # pushes your commit(s) and the refs/notes/rad-lfs mapping together
> ```
>
> Someone else, cloning the repository for the first time:
>
> ```sh
> rad clone rad:<repo-id>
> cd <repo>
> rad lfs init # one-time, needs their own ipfs daemon running
> git lfs pull # fetches large files via IPFS instead of git
> ```
>
> **Private repository?** You'll just be prompted for your keystore passphrase on `commit`/
> `push`/`pull` when needed (ssh-agent alone can't do the key-agreement encryption requires —
> see [`LFS-IPFS.md`](LFS-IPFS.md#encryption-for-private-repositories)). After granting a new
> collaborator access, have an *already-authorized* collaborator run `rad lfs rekey` and push,
> so the newcomer can decrypt files committed before they were added:
>
> ```sh
> rad id update --allow <did> # or add them as a delegate
> rad lfs rekey # run by someone already authorized
> git push rad # publish the updated wrapped keys
> ```
>
> **Repository set up with an older build?** A bare `git push rad` used to push *only* the
> notes mapping, silently leaving your commit unpushed (`rad lfs init` only configured a push
> refspec for the notes ref, not the branch). Fixed — but only for repositories where
> `rad lfs init` has been (re-)run with this build; re-run it once (harmless, idempotent) to
> pick up the fix on a repo set up before this. Until then, push both explicitly:
> `git push rad <branch>` then `git push rad`.
>
> Full design, encryption details, and a longer troubleshooting table:
> [`LFS-IPFS.md`](LFS-IPFS.md).
---
*Radicle Heartwood Protocol & Stack*
Heartwood is the third iteration of the Radicle Protocol, a powerful

28
build.rs
View File

@ -2,6 +2,19 @@ use std::env;
use std::process::Command;
fn main() -> Result<(), Box<dyn std::error::Error>> {
// Without this, cargo has no way to know it needs to re-run this
// script just because the current commit changed -- by default it
// only reruns a build script when a file inside the crate's own
// source tree changes, and `.git` is outside that entirely. That
// silently let GIT_HEAD/RADICLE_VERSION go stale on ordinary
// incremental rebuilds across commits that didn't happen to touch
// this crate's own sources (discovered live: several `cargo build`s
// in a row kept reporting an old commit's version despite the
// working tree being clean at a newer commit). `.git/logs/HEAD` (the
// reflog) is touched on every commit/checkout/merge, unlike
// `.git/HEAD` itself, which only changes when you switch branches.
println!("cargo::rerun-if-changed=../../.git/logs/HEAD");
// Set a build-time `GIT_HEAD` env var which includes the commit id;
// such that we can tell which code is running.
let hash = env::var("GIT_HEAD").unwrap_or_else(|_| {
@ -13,7 +26,7 @@ fn main() -> Result<(), Box<dyn std::error::Error>> {
.ok()
.and_then(|output| {
if output.status.success() {
String::from_utf8(output.stdout).ok()
String::from_utf8(output.stdout).ok().map(|s| s.trim().to_string())
} else {
None
}
@ -44,7 +57,7 @@ fn main() -> Result<(), Box<dyn std::error::Error>> {
.ok()
.and_then(|output| {
if output.status.success() {
String::from_utf8(output.stdout).ok()
String::from_utf8(output.stdout).ok().map(|s| s.trim().to_string())
} else {
None
}
@ -61,6 +74,15 @@ fn main() -> Result<(), Box<dyn std::error::Error>> {
version
};
// This is the `rad-lfs-ipfs` fork (Git LFS support backed by IPFS, see
// README.md/LFS-IPFS.md) -- append a semver build-metadata suffix (the
// `+...` part a `git describe`-derived version doesn't otherwise have
// room for) so `--version` unambiguously identifies a binary as this
// fork rather than upstream heartwood, e.g. when checking what a
// deploy target is actually running. Applies to every binary in this
// workspace, since they all share this build script via symlink.
let version = format!("{version}+lfs-ipfs");
// Set a build-time `SOURCE_DATE_EPOCH` env var which includes the commit time.
let commit_time = env::var("SOURCE_DATE_EPOCH").unwrap_or_else(|_| {
Command::new("git")
@ -72,7 +94,7 @@ fn main() -> Result<(), Box<dyn std::error::Error>> {
.ok()
.and_then(|output| {
if output.status.success() {
String::from_utf8(output.stdout).ok()
String::from_utf8(output.stdout).ok().map(|s| s.trim().to_string())
} else {
None
}

9
crates/radicle-cli/Cargo.toml
View File

@ -20,10 +20,14 @@ tor = ["radicle/tor"]
[dependencies]
anyhow = "1"
base64 = { workspace = true }
chacha20poly1305 = { workspace = true }
chrono = { workspace = true, features = ["clock", "std"] }
clap = { version = "4.5.44", features = ["derive"] }
clap_complete = "4.5"
cyphernet = { workspace = true, features = ["ed25519"] }
dunce = { workspace = true }
hkdf = { workspace = true }
human-panic.workspace = true
humantime.workspace = true
itertools.workspace = true
@ -31,7 +35,7 @@ log = { workspace = true, features = ["std"] }
nonempty = { workspace = true }
radicle = { workspace = true, features = ["schemars"] }
radicle-cob = { workspace = true }
radicle-crypto = { workspace = true }
radicle-crypto = { workspace = true, features = ["cyphernet"] }
radicle-localtime = { workspace = true }
radicle-surf = { workspace = true }
radicle-term = { workspace = true }
@ -39,9 +43,11 @@ radicle-log = { workspace = true }
schemars = { workspace = true }
serde = { workspace = true }
serde_json = { workspace = true }
sha2 = { workspace = true }
tempfile = { workspace = true }
thiserror = { workspace = true, default-features = true }
timeago = { version = "0.4.2", default-features = false }
zeroize = { workspace = true }
tree-sitter = "0.24.4"
tree-sitter-bash = "0.23.3"
tree-sitter-c = "0.23.2"
@ -56,6 +62,7 @@ tree-sitter-ruby = "0.23.1"
tree-sitter-rust = "0.23.2"
tree-sitter-toml-ng = "0.6.0"
tree-sitter-typescript = "0.23.2"
ureq = { workspace = true, features = ["multipart"] }
[target.'cfg(unix)'.dependencies]
backtrace = { workspace = true, optional = true }

1
crates/radicle-cli/src/commands.rs
View File

@ -14,6 +14,7 @@ pub mod inbox;
pub mod init;
pub mod inspect;
pub mod issue;
pub mod lfs;
pub mod ls;
pub mod node;
pub mod patch;

28
crates/radicle-cli/src/commands/lfs.rs Normal file
View File

@ -0,0 +1,28 @@
//! `rad lfs` command implementation.
pub mod backfill;
pub mod fetch;
pub mod fetch_batch;
pub mod init;
pub mod precommit;
pub mod rekey;
pub mod store;
mod args;
use crate::terminal as term;
pub use args::Args;
use args::Command;
pub fn run(args: Args, ctx: impl term::Context) -> anyhow::Result<()> {
match args.command {
Command::Init => self::init::run(),
Command::Store { oid, size, path } => self::store::run(oid, size, path, ctx),
Command::Fetch { oid, size, out } => self::fetch::run(oid, size, out, ctx),
Command::Rekey => self::rekey::run(ctx),
Command::Precommit => self::precommit::run(ctx),
Command::Backfill => self::backfill::run(ctx),
Command::FetchBatch => self::fetch_batch::run(ctx),
}
}

64
crates/radicle-cli/src/commands/lfs/args.rs Normal file
View File

@ -0,0 +1,64 @@
use clap::{Parser, Subcommand};
const ABOUT: &str = "Manage Git LFS (large file) support, backed by IPFS";
const LONG_ABOUT: &str = r#"
The `lfs` command manages Git LFS (Large File Storage) support for a
Radicle repository. Large file content is stored on the contributor's
local IPFS (Kubo) node rather than on a seed-hosted HTTP server: there
is no additional server-side infrastructure to run.
See `rad lfs init --help` for the one-time setup command.
"#;
#[derive(Parser, Debug)]
#[command(about = ABOUT, long_about = LONG_ABOUT, disable_version_flag = true)]
pub struct Args {
#[command(subcommand)]
pub command: Command,
}
#[derive(Subcommand, Debug)]
pub enum Command {
/// Set up Git LFS support for this repository, backed by IPFS
Init,
/// Store an LFS object's content in IPFS (invoked by the custom
/// transfer agent; not for interactive use)
#[command(hide = true)]
Store {
#[arg(long)]
oid: String,
#[arg(long)]
size: i64,
path: std::path::PathBuf,
},
/// Fetch an LFS object's content from IPFS (invoked by the custom
/// transfer agent; not for interactive use)
#[command(hide = true)]
Fetch {
#[arg(long)]
oid: String,
#[arg(long)]
size: i64,
#[arg(long)]
out: std::path::PathBuf,
},
/// Grant newly-authorized collaborators access to previously-encrypted
/// LFS objects
Rekey,
/// Store every staged LFS file's content in IPFS in one batch (invoked
/// by the pre-commit hook; not for interactive use). Reads
/// "<oid> <size> <path>" lines from stdin.
#[command(hide = true)]
Precommit,
/// Retroactively pin any LFS-tracked file at HEAD that was committed
/// without going through the pre-commit hook (e.g. `--no-verify`, or
/// the IPFS daemon/`ipfs`/`rad` weren't available at commit time)
Backfill,
/// Long-lived worker that fetches many LFS objects across one process
/// (invoked by the custom transfer agent; not for interactive use).
/// Reads "<oid> <size> <out-path>" lines from stdin, one at a time,
/// writing a JSON response line for each.
#[command(hide = true)]
FetchBatch,
}

171
crates/radicle-cli/src/commands/lfs/backfill.rs Normal file
View File

@ -0,0 +1,171 @@
//! `rad lfs backfill` — retroactively stores any already-committed
//! LFS-tracked file that doesn't yet have a `refs/notes/rad-lfs` note (so
//! it was never pinned to IPFS). This happens whenever the pre-commit hook
//! didn't run or didn't have what it needed at commit time: a commit made
//! with `--no-verify`, or one made while the `ipfs`/`rad` binaries weren't
//! on `PATH` yet (the hook soft-skips pinning in that case rather than
//! blocking the commit -- see `HOOK_BODY`'s `command -v` checks).
//!
//! Content actually needs to be available locally to backfill it from:
//! this reads from Git LFS's own local object cache
//! (`.git/lfs/objects/<oid prefix>/<oid>`, populated by the `git-lfs`
//! clean filter on `git add`, which -- unlike the pre-commit *hook* --
//! isn't skipped by `--no-verify`), falling back to the working-tree file
//! if that's missing. A file whose content isn't available through either
//! path can't be backfilled from this machine; some other peer that
//! already has it needs to run this instead.
use std::path::{Path, PathBuf};
use anyhow::Context as _;
use radicle::storage::{ReadRepository as _, ReadStorage as _};
use crate::ipfs;
use crate::terminal as term;
use super::store::store_object;
pub fn run(ctx: impl term::Context) -> anyhow::Result<()> {
let profile = ctx.profile()?;
let (repo, rid) = radicle::rad::cwd()
.context("`rad lfs backfill` must be run inside a Radicle repository working copy")?;
let workdir = repo
.workdir()
.ok_or_else(|| anyhow::anyhow!("`rad lfs backfill` needs a git working copy, not a bare repository"))?;
let doc = profile.storage.repository(rid)?.identity_doc()?.doc;
ipfs::check_daemon()?;
// Every path git currently tracks at HEAD (not the working tree or
// index, which may have unrelated local modifications) -- mirrors
// HOOK_BODY's own approach of shelling out to `git` for this rather
// than reimplementing tree/attribute logic against git2 directly.
let ls_files = radicle::git::run(Some(workdir), ["ls-tree", "-r", "-z", "--name-only", "HEAD"])
.context("failed to list files at HEAD")?;
let paths: Vec<String> = String::from_utf8_lossy(&ls_files.stdout)
.split('\0')
.filter(|s| !s.is_empty())
.map(str::to_string)
.collect();
let mut candidates = Vec::new();
for path in paths {
// Matches HOOK_BODY's own check: only files the *current*
// `.gitattributes` marks as `filter=lfs` are our concern.
let is_lfs = radicle::git::run(Some(workdir), ["check-attr", "filter", "--", &path])
.ok()
.map(|out| String::from_utf8_lossy(&out.stdout).contains(": filter: lfs"))
.unwrap_or(false);
if !is_lfs {
continue;
}
let blob_oid_out = radicle::git::run(Some(workdir), ["rev-parse", &format!("HEAD:{path}")])
.with_context(|| format!("failed to resolve blob oid for `{path}`"))?;
let blob_oid_str = String::from_utf8_lossy(&blob_oid_out.stdout).trim().to_string();
let Ok(blob_oid) = radicle::git::raw::Oid::from_str(&blob_oid_str) else {
continue;
};
candidates.push((path, blob_oid));
}
if candidates.is_empty() {
term::success!("No LFS-tracked files found in HEAD; nothing to backfill");
return Ok(());
}
let mut signer = None;
let mut backfilled = 0;
let mut already_pinned = 0;
let mut unavailable = Vec::new();
for (path, blob_oid) in candidates {
if !crate::lfs_crypto::find_cids(&repo, blob_oid)?.is_empty() {
already_pinned += 1;
continue;
}
let blob = repo
.find_blob(blob_oid)
.with_context(|| format!("failed to read blob for `{path}`"))?;
let Ok(pointer_text) = std::str::from_utf8(blob.content()) else {
continue;
};
let Some(oid) = pointer_text
.lines()
.find_map(|line| line.strip_prefix("oid sha256:"))
else {
continue;
};
let Some(size) = pointer_text
.lines()
.find_map(|line| line.strip_prefix("size "))
.and_then(|s| s.parse::<i64>().ok())
else {
continue;
};
let Some(content_path) = locate_content(workdir, oid, size, &path) else {
unavailable.push(path);
continue;
};
store_object(&profile, &repo, rid, &doc, oid, size, &content_path, &mut signer)
.with_context(|| format!("failed to backfill `{path}`"))?;
term::success!("Backfilled `{path}`");
backfilled += 1;
}
term::blank();
if backfilled > 0 {
term::success!("Backfilled {backfilled} object(s)");
}
if already_pinned > 0 {
term::info!("{already_pinned} object(s) already had a note; left untouched");
}
if !unavailable.is_empty() {
term::warning(format!(
"{} object(s) have no note and no content available locally to backfill from -- \
ask a peer that already has them to run `rad lfs backfill`:",
unavailable.len()
));
for path in &unavailable {
term::warning(format!(" {path}"));
}
}
Ok(())
}
/// Git LFS's own local object cache is the reliable source: the
/// `git-lfs` clean filter populates it on `git add` regardless of
/// whether the pre-commit *hook* ran (`--no-verify` skips hooks, not
/// filters). Falls back to the working-tree file, in case content is
/// present there but the LFS cache was pruned -- rejected if the size on
/// disk doesn't match what the pointer recorded, since that means it's
/// just the unsmudged pointer text, not the real content.
fn locate_content(workdir: &Path, oid: &str, size: i64, path: &str) -> Option<PathBuf> {
if oid.len() >= 4 {
let cached = workdir
.join(".git/lfs/objects")
.join(&oid[0..2])
.join(&oid[2..4])
.join(oid);
if matches_size(&cached, size) {
return Some(cached);
}
}
let working_copy = workdir.join(path);
if matches_size(&working_copy, size) {
return Some(working_copy);
}
None
}
fn matches_size(path: &Path, expected: i64) -> bool {
std::fs::metadata(path)
.map(|meta| meta.len() == expected as u64)
.unwrap_or(false)
}

88
crates/radicle-cli/src/commands/lfs/fetch.rs Normal file
View File

@ -0,0 +1,88 @@
//! `rad lfs fetch` — plumbing command invoked by the Git LFS custom
//! transfer agent (`rad-lfs-transfer`) to download one LFS object's
//! content from the local IPFS (Kubo) node, decrypting it first if the
//! repository is private.
use std::path::{Path, PathBuf};
use anyhow::{Context as _, anyhow};
use radicle::Profile;
use radicle::git::raw::Repository;
use radicle::identity::RepoId;
use radicle_crypto::ssh::keystore::MemorySigner;
use crate::ipfs;
use crate::lfs_crypto;
use crate::terminal as term;
pub fn run(oid: String, size: i64, out: PathBuf, ctx: impl term::Context) -> anyhow::Result<()> {
let profile = ctx.profile()?;
let (repo, rid) = radicle::rad::cwd()
.context("`rad lfs fetch` must be run inside a Radicle repository working copy")?;
let mut signer = None;
fetch_object(&profile, &repo, rid, &oid, size, &out, &mut signer)
}
/// Fetches one LFS object's content from IPFS (decrypting it first for
/// private repos) and writes it to `out`. Factored out of [`run`] so
/// `rad lfs fetch-batch` can process many objects across a single
/// process, loading (and prompting for) the signer at most once across
/// however many objects it ends up handling, instead of once per object.
#[allow(clippy::too_many_arguments)]
pub fn fetch_object(
profile: &Profile,
repo: &Repository,
rid: RepoId,
oid: &str,
size: i64,
out: &Path,
signer: &mut Option<MemorySigner>,
) -> anyhow::Result<()> {
let pointer_text =
format!("version https://git-lfs.github.com/spec/v1\noid sha256:{oid}\nsize {size}\n");
let blob_oid = repo
.blob(pointer_text.as_bytes())
.context("failed to write LFS pointer blob")?;
let candidates = lfs_crypto::find_envelopes(repo, blob_oid)
.with_context(|| format!("failed to read LFS notes for oid {oid}"))?;
if candidates.is_empty() {
anyhow::bail!(
"no CID recorded for oid {oid}; the peer that has this object needs to push its \
{} ref",
lfs_crypto::NOTES_REF
);
}
ipfs::check_daemon()?;
// Almost always exactly one candidate. More than one only happens if
// two peers independently encrypted byte-identical content (same
// oid/size) producing different ciphertext each time -- try each
// until one actually decrypts for us.
let mut last_err = None;
for envelope in candidates {
let result = match &envelope.enc {
None => ipfs::cat(&envelope.cid),
Some(enc) => ipfs::cat(&envelope.cid).and_then(|ciphertext| {
if signer.is_none() {
*signer = Some(lfs_crypto::load_signer(profile)?);
}
let signer = signer.as_ref().expect("just populated above");
lfs_crypto::decrypt_with(&ciphertext, enc, signer, profile.did(), &rid, oid)
}),
};
match result {
Ok(bytes) => {
std::fs::write(out, bytes)
.with_context(|| format!("failed to write `{}`", out.display()))?;
return Ok(());
}
Err(err) => last_err = Some(err),
}
}
Err(last_err.unwrap_or_else(|| anyhow!("no CID recorded for oid {oid}")))
}

67
crates/radicle-cli/src/commands/lfs/fetch_batch.rs Normal file
View File

@ -0,0 +1,67 @@
//! `rad lfs fetch-batch` — long-lived plumbing worker invoked by the Git
//! LFS custom transfer agent (`rad-lfs-transfer`) to download multiple LFS
//! objects across a single process, kept alive for the lifetime of one
//! `git lfs pull`/checkout session -- specifically so a private repo's
//! passphrase is only prompted for once per session, not once per file.
//!
//! Unlike `rad lfs precommit`/`backfill` (which know every file upfront
//! and can just read stdin until EOF), this can't collect a batch before
//! starting: git-lfs's custom-transfer protocol requests objects one at a
//! time by default (`rad-lfs-transfer` doesn't negotiate the "concurrent"
//! capability), so the caller only knows about the next object once the
//! previous one has been answered. This process instead stays alive and
//! answers requests one at a time as they arrive, reusing the same
//! unwrapped signer across all of them.
//!
//! Protocol (line-delimited, UTF-8, on stdin/stdout):
//! request: "<oid> <size> <out-path>\n"
//! response: a JSON object, one line per request, in the same order:
//! `{"ok":true}` or `{"ok":false,"error":"<message>"}`
//! The process exits cleanly on stdin EOF (the parent closing its write
//! end signals the transfer session is over).
use std::io::{BufRead as _, Write as _};
use std::path::PathBuf;
use anyhow::Context as _;
use serde_json::json;
use crate::terminal as term;
use super::fetch::fetch_object;
pub fn run(ctx: impl term::Context) -> anyhow::Result<()> {
let profile = ctx.profile()?;
let (repo, rid) = radicle::rad::cwd()
.context("`rad lfs fetch-batch` must be run inside a Radicle repository working copy")?;
let mut signer = None;
let stdin = std::io::stdin();
let mut stdout = std::io::stdout();
for line in stdin.lock().lines() {
let line = line.context("failed to read stdin")?;
// Each line is "<oid> <size> <out-path>" -- oid is a fixed-length
// hex string and size is decimal, so splitting on the first two
// spaces leaves the path (which may itself contain spaces) intact
// as the remainder. Mirrors `rad lfs precommit`'s input format.
let mut parts = line.splitn(3, ' ');
let (Some(oid), Some(size), Some(out)) = (parts.next(), parts.next(), parts.next())
else {
anyhow::bail!("malformed `rad lfs fetch-batch` input line: {line:?}");
};
let Ok(size) = size.parse::<i64>() else {
anyhow::bail!("invalid size in `rad lfs fetch-batch` input: {size:?}");
};
let out = PathBuf::from(out);
let response = match fetch_object(&profile, &repo, rid, oid, size, &out, &mut signer) {
Ok(()) => json!({"ok": true}),
Err(err) => json!({"ok": false, "error": format!("{err:#}")}),
};
writeln!(stdout, "{response}").context("failed to write response")?;
stdout.flush().context("failed to flush stdout")?;
}
Ok(())
}

400
crates/radicle-cli/src/commands/lfs/init.rs Normal file
View File

@ -0,0 +1,400 @@
//! `rad lfs init` — one-time, idempotent setup of Git LFS support for a
//! Radicle repository, backed by the contributor's local IPFS (Kubo) node.
//!
//! This deliberately does *not* talk to any seed-hosted HTTP LFS server:
//! large file content lives on each peer's own IPFS node, and the
//! oid -> CID mapping is carried in a git-notes ref (`refs/notes/rad-lfs`)
//! that replicates alongside the rest of the repository once the
//! push/fetch refspecs below are configured.
use std::fs;
#[cfg(unix)]
use std::os::unix::fs::PermissionsExt;
use std::path::Path;
use std::process::Command;
use anyhow::{Context as _, anyhow};
use radicle::git::raw::Repository;
use crate::git;
use crate::ipfs::{self, LFS_NOTES_REF as NOTES_REF};
use crate::lfs_crypto::{LOCAL_NOTES_REF, NOTE_AUTHOR_EMAIL, NOTE_AUTHOR_NAME};
use crate::terminal as term;
/// Name under which our custom transfer agent is registered with Git LFS.
const TRANSFER_AGENT_NAME: &str = "rad-ipfs";
/// Binary implementing the custom transfer agent. Must be on the user's
/// `PATH`; we intentionally don't assume a specific install location.
const TRANSFER_AGENT_BIN: &str = "rad-lfs-transfer";
/// Name of the remote that `rad init` sets up.
const RAD_REMOTE: &str = "rad";
/// Markers delimiting the block we manage inside `.git/hooks/pre-commit`,
/// so re-running `rad lfs init` updates our block in place instead of
/// duplicating it, and so we don't clobber any hook content that was
/// already there.
const HOOK_BEGIN: &str = "# >>> rad-lfs-init managed pre-commit hook >>>";
const HOOK_END: &str = "# <<< rad-lfs-init managed pre-commit hook <<<";
/// Body of the managed pre-commit hook block (POSIX `sh`).
///
/// For every file staged in the commit that's tracked by Git LFS (i.e.
/// matches a `filter=lfs` pattern in `.gitattributes`), this computes the
/// file's LFS oid (sha256) and size, then feeds all of them to a single
/// `rad lfs precommit` invocation as "<oid> <size> <path>" lines on stdin.
/// `rad lfs precommit` does the actual IPFS add/pin (encrypting first for
/// private repos) and writes the corresponding notes on `refs/notes/rad-lfs`
/// itself — see `commands/lfs/precommit.rs` and `commands/lfs/store.rs`.
///
/// Batching into one `rad` process (rather than one per file) matters for
/// private repos specifically: each file needs the keystore passphrase for
/// an ECDH key-agreement operation, and a separate process per file meant a
/// separate passphrase prompt per file. One process loads it once.
///
/// The hook only computes the cheap, dependency-free oid/size values in
/// shell; the CID/encryption/git-notes logic lives in `rad lfs precommit`
/// since it needs access to the repo's identity/visibility and (for private
/// repos) key material that a plain shell script has no business handling —
/// a git hook shelling out to our own `rad` binary is far simpler than
/// reimplementing that here.
///
/// Requires both `ipfs` and `rad` to be available on `PATH` at commit time.
const HOOK_BODY: &str = r#"rad_lfs_precommit() {
command -v ipfs >/dev/null 2>&1 || {
echo "rad-lfs: 'ipfs' CLI not found on PATH; skipping CID pinning for this commit" >&2
return 0
}
command -v rad >/dev/null 2>&1 || {
echo "rad-lfs: 'rad' CLI not found on PATH; skipping CID pinning for this commit" >&2
return 0
}
sha256() {
if command -v sha256sum >/dev/null 2>&1; then
sha256sum "$1" | awk '{ print $1 }'
else
shasum -a 256 "$1" | awk '{ print $1 }'
fi
}
staged=$(mktemp) || return 1
batch=$(mktemp) || { rm -f "$staged"; return 1; }
# `-z` avoids git's default C-style quoting of non-ASCII filenames (e.g.
# a literal "\305\202" for a Polish "ł") in `--name-only`'s normal
# output, which would otherwise not match any real file on disk and
# silently drop that file from LFS pinning. NUL-separated records are
# converted to newline-separated here for a plain `read` loop, which is
# safe since filenames practically never contain literal newlines.
git diff --cached --name-only --diff-filter=ACM -z | tr '\0' '\n' > "$staged"
while IFS= read -r file; do
[ -n "$file" ] || continue
[ -f "$file" ] || continue
attr=$(git check-attr filter -- "$file" | sed -n 's/.*: filter: //p')
[ "$attr" = "lfs" ] || continue
oid=$(sha256 "$file") || { rm -f "$staged" "$batch"; exit 1; }
size=$(wc -c < "$file" | tr -d ' ')
printf '%s %s %s\n' "$oid" "$size" "$file" >> "$batch"
done < "$staged"
rm -f "$staged"
if [ -s "$batch" ]; then
rad lfs precommit < "$batch" || {
echo "rad-lfs: 'rad lfs precommit' failed" >&2
rm -f "$batch"
exit 1
}
fi
rm -f "$batch"
}
rad_lfs_precommit"#;
pub fn run() -> anyhow::Result<()> {
let (repo, _rid) = radicle::rad::cwd()
.context("`rad lfs init` must be run inside a Radicle repository working copy")?;
let workdir = repo.workdir().ok_or_else(|| {
anyhow!("`rad lfs init` must be run inside a git working copy (not a bare repository)")
})?;
ipfs::check_daemon()?;
term::success!("Found a local IPFS (Kubo) daemon at {}", ipfs::kubo_api_url());
ensure_git_lfs_available()?;
git::git(workdir, ["lfs", "install", "--local"])
.context("failed to run `git lfs install --local`")?;
term::success!("Initialized Git LFS for this repository");
git::git(
workdir,
["config", "lfs.standalonetransferagent", TRANSFER_AGENT_NAME],
)
.context("failed to configure the LFS standalone transfer agent")?;
git::git(
workdir,
[
"config",
&format!("lfs.customtransfer.{TRANSFER_AGENT_NAME}.path"),
TRANSFER_AGENT_BIN,
],
)
.context("failed to configure the LFS custom transfer agent path")?;
term::success!(
"Configured Git LFS to transfer objects via `{TRANSFER_AGENT_BIN}` (backed by IPFS)"
);
// Push maps our *local-only* notes ref (`LOCAL_NOTES_REF`) to the bare
// `NOTES_REF` on the remote: `git push` lands whatever local ref we
// push under the pusher's own namespace on the server regardless of
// the local ref name, the same way `refs/heads/*` does, so the
// destination staying bare is correct and matches what
// `list.rs`'s `lfs_notes_refs` looks for server-side. The *source*
// is deliberately not the bare name too -- see `LOCAL_NOTES_REF`'s
// doc comment for why (a git ref D/F conflict with fetched peer
// refs).
//
// Fetch is different: `refs/notes/rad-lfs` is a per-peer ref (any
// peer can commit an LFS-tracked file, not just delegates), so
// there's no single canonical value the remote can advertise the way
// it does for `refs/heads`/`refs/tags`. The remote instead advertises
// every peer's note individually under `refs/notes/rad-lfs/<peer>`,
// so the fetch refspec needs a wildcard to pull them all; the LFS
// tooling merges across whatever's fetched (see
// `lfs_crypto::find_envelope`) rather than expecting one ref.
let notes_push_refspec = format!("+{LOCAL_NOTES_REF}:{NOTES_REF}");
let fetch_refspec = format!("+{NOTES_REF}/*:{NOTES_REF}/*");
let push_key = format!("remote.{RAD_REMOTE}.push");
let fetch_key = format!("remote.{RAD_REMOTE}.fetch");
// Migration: earlier versions of `rad lfs init` configured a plain,
// non-wildcard fetch refspec for the notes ref, which `git fetch` can
// never satisfy -- the remote never advertises a bare
// `refs/notes/rad-lfs` (only the wildcarded per-peer form below), so
// a stale entry here breaks the *entire* fetch, not just this one
// ref. Remove it if present so re-running `rad lfs init` actually
// fixes an already-configured repository, rather than leaving the
// broken entry alongside the corrected one.
remove_refspec_if_present(workdir, &fetch_key, &format!("+{NOTES_REF}:{NOTES_REF}"))?;
// Migration: earlier versions also pushed from the bare local ref
// (symmetric `+refs/notes/rad-lfs:refs/notes/rad-lfs`). Replace it
// with the corrected asymmetric refspec above.
remove_refspec_if_present(workdir, &push_key, &format!("+{NOTES_REF}:{NOTES_REF}"))?;
ensure_refspec(workdir, &push_key, &notes_push_refspec)?;
ensure_refspec(workdir, &fetch_key, &fetch_refspec)?;
// As soon as *any* explicit push refspec is configured on a remote
// (the notes one above), git stops falling back to its usual
// push.default-driven behavior (pushing whatever branch you're on) for
// a bare `git push rad` -- it only pushes what's explicitly listed.
// Without a branch refspec too, that means a bare `git push rad` after
// committing an LFS-tracked file would silently push *only* the notes
// mapping, not the commit itself -- exactly the gotcha that caused a
// collaborator to see "no CID recorded for oid ..." even though the
// committer had "pushed". Configuring this explicitly, mirroring the
// existing `+refs/heads/*:refs/remotes/rad/*` fetch refspec above,
// makes a single bare `git push rad` push every local branch *and*
// the notes mapping together.
let heads_push_refspec = "+refs/heads/*:refs/heads/*".to_string();
ensure_refspec(workdir, &push_key, &heads_push_refspec)?;
term::success!(
"Configured the `{RAD_REMOTE}` remote so a single `git push {RAD_REMOTE}` pushes your \
branches and the `{NOTES_REF}` notes mapping together"
);
migrate_local_notes_ref(&repo)?;
install_pre_commit_hook(&repo)?;
term::success!("Installed the `rad-lfs` pre-commit hook");
term::blank();
term::success!("Git LFS is now configured for this repository, backed by your local IPFS node.");
term::info!(
"File CIDs are recorded in the `{NOTES_REF}` git-notes ref, which now travels with a plain `git push {RAD_REMOTE}` / `git pull {RAD_REMOTE}` alongside your branches."
);
Ok(())
}
/// Make sure the `git-lfs` binary is installed before we shell out to it,
/// so we can give a clear, actionable error instead of a raw "unknown git
/// command" failure.
fn ensure_git_lfs_available() -> anyhow::Result<()> {
match Command::new("git-lfs").arg("version").output() {
Ok(output) if output.status.success() => Ok(()),
Ok(output) => Err(anyhow!(
"`git-lfs version` failed: {}",
String::from_utf8_lossy(&output.stderr)
)),
Err(err) if err.kind() == std::io::ErrorKind::NotFound => Err(anyhow!(
"Git LFS is not installed (the `git-lfs` binary was not found on your PATH). \
Install Git LFS — see https://git-lfs.com — and try again."
)),
Err(err) => Err(err).context("failed to check for `git-lfs`"),
}
}
/// Add `value` to the (potentially multi-valued) git config key `key`,
/// unless it's already present, so re-running `rad lfs init` doesn't
/// duplicate refspecs.
fn ensure_refspec(workdir: &Path, key: &str, value: &str) -> anyhow::Result<()> {
// `git config --get-all` exits non-zero when the key is unset, which
// isn't an error for us, so we use the lower-level `run` rather than
// `git::git` (which treats a non-zero exit as failure).
let output = radicle::git::run(Some(workdir), ["config", "--get-all", key])
.with_context(|| format!("failed to read git config `{key}`"))?;
let existing = String::from_utf8_lossy(&output.stdout);
if existing.lines().any(|line| line.trim() == value) {
return Ok(());
}
git::git(workdir, ["config", "--add", key, value])
.with_context(|| format!("failed to set git config `{key}`"))?;
Ok(())
}
/// Remove `value` from the (potentially multi-valued) git config key
/// `key`, if present, without disturbing any other values under the same
/// key (e.g. the default `+refs/heads/*:refs/remotes/rad/*` that plain
/// `git remote add` already configures on `remote.<rad>.fetch`).
fn remove_refspec_if_present(workdir: &Path, key: &str, value: &str) -> anyhow::Result<()> {
let output = radicle::git::run(Some(workdir), ["config", "--get-all", key])
.with_context(|| format!("failed to read git config `{key}`"))?;
let existing = String::from_utf8_lossy(&output.stdout);
if !existing.lines().any(|line| line.trim() == value) {
return Ok(());
}
// `git config --unset` matches its value argument as a regex against
// the whole existing value, not literally -- anchor and escape so it
// removes exactly this one entry and nothing else.
let escaped: String = value
.chars()
.map(|c| {
if "\\^$.|?*+()[]{}".contains(c) {
format!("\\{c}")
} else {
c.to_string()
}
})
.collect();
let pattern = format!("^{escaped}$");
git::git(workdir, ["config", "--unset", key, &pattern])
.with_context(|| format!("failed to remove stale git config `{key}` entry"))?;
Ok(())
}
/// Migration: earlier versions of this fork wrote local notes directly to
/// the bare `NOTES_REF` (`refs/notes/rad-lfs`). Since the fetch refspec
/// populates `refs/notes/rad-lfs/<peer>` siblings locally -- including our
/// own peer's copy, fetched back after a push -- a leftover bare ref
/// causes a git ref D/F (file-vs-directory) conflict: `refs/notes/rad-lfs`
/// can't simultaneously be a leaf ref and a directory prefix for
/// `refs/notes/rad-lfs/<peer>` in the same namespace. That made every
/// `rad lfs store`/`rad lfs rekey` call after the first fetch fail
/// outright with "failed to write LFS note".
///
/// If a bare `refs/notes/rad-lfs` ref exists, copy its notes into
/// `LOCAL_NOTES_REF` (so nothing already recorded there is lost) and
/// delete the bare ref, eliminating the conflict. A no-op if the bare ref
/// doesn't exist (fresh setups, or a repo that's already been migrated).
fn migrate_local_notes_ref(repo: &Repository) -> anyhow::Result<()> {
let Ok(mut reference) = repo.find_reference(NOTES_REF) else {
return Ok(());
};
// Read every note out *before* touching anything -- writing to
// `LOCAL_NOTES_REF` while the bare `NOTES_REF` still exists is the
// exact same D/F conflict this migration exists to fix, just
// self-inflicted. The bare ref has to be gone first.
let mut migrated = Vec::new();
if let Ok(notes) = repo.notes(Some(NOTES_REF)) {
for (_, target_oid) in notes.flatten() {
if let Ok(note) = repo.find_note(Some(NOTES_REF), target_oid)
&& let Some(message) = note.message()
{
migrated.push((target_oid, message.to_string()));
}
}
}
reference
.delete()
.context("failed to remove the legacy bare `refs/notes/rad-lfs` ref")?;
if migrated.is_empty() {
return Ok(());
}
let signature = radicle::git::raw::Signature::now(NOTE_AUTHOR_NAME, NOTE_AUTHOR_EMAIL)
.context("failed to construct note signature")?;
for (target_oid, message) in migrated {
repo.note(&signature, &signature, Some(LOCAL_NOTES_REF), target_oid, &message, true)
.with_context(|| {
format!("failed to migrate note for {target_oid} to `{LOCAL_NOTES_REF}`")
})?;
}
term::success!("Migrated local LFS notes from the legacy `{NOTES_REF}` ref");
Ok(())
}
/// Install (or update, or chain onto) the `.git/hooks/pre-commit` hook that
/// pins staged LFS objects to IPFS and records their CID as a git note.
fn install_pre_commit_hook(repo: &Repository) -> anyhow::Result<()> {
let hooks_dir = repo.path().join("hooks");
fs::create_dir_all(&hooks_dir)
.with_context(|| format!("failed to create hooks directory `{}`", hooks_dir.display()))?;
let hook_path = hooks_dir.join("pre-commit");
let existing = fs::read_to_string(&hook_path).unwrap_or_default();
let new_content = if let Some((before, after_begin)) = existing.split_once(HOOK_BEGIN) {
// We've already installed our block before: replace it in place so
// re-running `rad lfs init` doesn't duplicate it, while preserving
// anything the user added before/after it.
let after = after_begin
.split_once(HOOK_END)
.map_or("", |(_, after)| after);
format!("{before}{HOOK_BEGIN}\n{HOOK_BODY}\n{HOOK_END}{after}")
} else if existing.trim().is_empty() {
format!("#!/bin/sh\n\n{HOOK_BEGIN}\n{HOOK_BODY}\n{HOOK_END}\n")
} else {
// Some other hook (not ours) is already installed (`git lfs
// install` itself only installs a pre-push hook, so this is
// usually a hook the user or another tool set up). Chain onto it
// rather than clobbering it.
let mut content = existing;
if !content.ends_with('\n') {
content.push('\n');
}
content.push('\n');
content.push_str(HOOK_BEGIN);
content.push('\n');
content.push_str(HOOK_BODY);
content.push('\n');
content.push_str(HOOK_END);
content.push('\n');
content
};
fs::write(&hook_path, new_content)
.with_context(|| format!("failed to write hook `{}`", hook_path.display()))?;
#[cfg(unix)]
{
let mut perms = fs::metadata(&hook_path)?.permissions();
perms.set_mode(perms.mode() | 0o111);
fs::set_permissions(&hook_path, perms)?;
}
Ok(())
}

55
crates/radicle-cli/src/commands/lfs/precommit.rs Normal file
View File

@ -0,0 +1,55 @@
//! `rad lfs precommit` — plumbing command invoked once per commit by the
//! `rad-lfs` pre-commit hook. Reads every staged LFS file's oid/size/path
//! from stdin and stores them all in a single process, so a private repo's
//! passphrase (needed for the ECDH key-agreement `rad lfs store` performs)
//! is prompted for at most once per commit, not once per file.
use std::io::BufRead as _;
use std::path::PathBuf;
use anyhow::Context as _;
use radicle::storage::{ReadRepository as _, ReadStorage as _};
use crate::terminal as term;
use super::store::store_object;
pub fn run(ctx: impl term::Context) -> anyhow::Result<()> {
let profile = ctx.profile()?;
let (repo, rid) = radicle::rad::cwd()
.context("`rad lfs precommit` must be run inside a Radicle repository working copy")?;
let doc = profile.storage.repository(rid)?.identity_doc()?.doc;
let mut signer = None;
let stdin = std::io::stdin();
for line in stdin.lock().lines() {
let line = line.context("failed to read stdin")?;
// Each line is "<oid> <size> <path>" -- oid is a fixed-length hex
// string and size is decimal, so splitting on the first two spaces
// leaves the path (which may itself contain spaces) intact as the
// remainder.
let mut parts = line.splitn(3, ' ');
let (Some(oid), Some(size), Some(path)) = (parts.next(), parts.next(), parts.next())
else {
anyhow::bail!("malformed `rad lfs precommit` input line: {line:?}");
};
let size: i64 = size
.parse()
.with_context(|| format!("invalid size in `rad lfs precommit` input: {size:?}"))?;
store_object(
&profile,
&repo,
rid,
&doc,
oid,
size,
&PathBuf::from(path),
&mut signer,
)
.with_context(|| format!("failed to store LFS object for `{path}`"))?;
}
Ok(())
}

152
crates/radicle-cli/src/commands/lfs/rekey.rs Normal file
View File

@ -0,0 +1,152 @@
//! `rad lfs rekey` — grants newly-authorized collaborators (new delegates,
//! or newly-added entries in a private repo's allow-list) access to
//! previously-encrypted LFS objects, without re-encrypting or re-uploading
//! any content: it only adds new wrapped-key entries to each object's
//! note for recipients who are missing one.
use std::collections::BTreeSet;
use anyhow::Context as _;
use radicle::git::raw::Signature;
use radicle::identity::Did;
use radicle::storage::{ReadRepository as _, ReadStorage as _};
use crate::lfs_crypto::{self, LOCAL_NOTES_REF, NOTE_AUTHOR_EMAIL, NOTE_AUTHOR_NAME};
use crate::terminal as term;
pub fn run(ctx: impl term::Context) -> anyhow::Result<()> {
let profile = ctx.profile()?;
let (repo, rid) = radicle::rad::cwd()
.context("`rad lfs rekey` must be run inside a Radicle repository working copy")?;
let doc = profile.storage.repository(rid)?.identity_doc()?.doc;
let signer = lfs_crypto::load_signer(&profile)?;
let my_did = profile.did();
// The full set of currently-authorized DIDs: delegates ∪ the private
// allow-list ∪ `my_did` itself (the `sender` parameter of
// `recipients_for` is only meaningful for `encrypt_for`'s "make sure
// the committer can read their own object back" guarantee; here we
// just want the full authorized set, and `my_did` is necessarily
// already a member of it since running `rekey` requires being
// authorized).
let current = lfs_crypto::recipients_for(&doc, my_did);
// Walk every note across every peer's notes ref -- not just the local
// one -- since an object's only note may live under a peer we've
// fetched from rather than one we wrote ourselves.
let targets = lfs_crypto::all_note_targets(&repo)
.context("failed to enumerate LFS notes")?;
if targets.is_empty() {
term::success!("Nothing to rekey: no LFS objects are tracked in this repository yet");
return Ok(());
}
let signature = Signature::now(NOTE_AUTHOR_NAME, NOTE_AUTHOR_EMAIL)
.context("failed to construct note signature")?;
let mut rekeyed_objects = 0;
let mut added_recipients = 0;
for target_oid in targets {
// The note only stores the CID and encryption metadata, not the
// oid/size that were used as HKDF context when wrapping the
// content key. Recover them losslessly from the pointer blob
// itself: `target_oid` *is* the git blob hash of the exact
// deterministic pointer text `store`/`fetch` compute from
// oid/size, so we can parse it back out.
let Ok(blob) = repo.find_blob(target_oid) else {
continue;
};
let Ok(pointer_text) = std::str::from_utf8(blob.content()) else {
continue;
};
let Some(oid) = pointer_text
.lines()
.find_map(|line| line.strip_prefix("oid sha256:"))
else {
continue;
};
let candidates = match lfs_crypto::find_envelopes(&repo, target_oid) {
Ok(candidates) => candidates,
Err(err) => {
term::warning(format!("Skipping object with oid {oid}: {err}"));
continue;
}
};
// Almost always exactly one candidate; more than one only if two
// peers independently encrypted byte-identical content (same
// oid/size) producing different ciphertext each time. Rekey
// whichever ones we're actually authorized to unwrap -- skip the
// rest quietly, they're not our concern.
for mut envelope in candidates {
let Some(enc) = envelope.enc.as_mut() else {
// Public (unencrypted) object: nothing to rekey.
continue;
};
let cek = match lfs_crypto::unwrap_cek(&enc.recipients, &signer, my_did, &rid, oid) {
Ok(cek) => cek,
Err(_) => {
// Not authorized for *this particular* candidate --
// expected and not worth warning about, since another
// candidate for the same object (or none) may apply.
continue;
}
};
let existing: BTreeSet<Did> = enc.recipients.iter().map(|w| w.did).collect();
let missing: Vec<Did> = current.difference(&existing).copied().collect();
if missing.is_empty() {
continue;
}
let cek_key: chacha20poly1305::Key = (*cek).into();
let mut any_added = false;
for recipient in &missing {
match lfs_crypto::wrap_cek_for(&cek_key, &signer, my_did, recipient, &rid, oid) {
Ok(wrapped) => {
enc.recipients.push(wrapped);
added_recipients += 1;
any_added = true;
}
Err(err) => {
term::warning(format!(
"Failed to wrap the content key for {recipient} on object with oid \
{oid}: {err}"
));
}
}
}
if !any_added {
continue;
}
// Always write to our own local ref, regardless of which
// peer's ref the original note came from -- once pushed, this
// lands under our own namespace and is picked up by others
// via the fetch refspec, alongside (not replacing) the
// original note.
let message = serde_json::to_string(&envelope)
.context("failed to serialize LFS note envelope")?;
repo.note(&signature, &signature, Some(LOCAL_NOTES_REF), target_oid, &message, true)
.context("failed to write LFS note")?;
rekeyed_objects += 1;
}
}
if rekeyed_objects == 0 {
term::success!("Nothing to rekey: all recorded recipients already have access");
} else {
term::success!(
"Rekeyed {rekeyed_objects} object(s), added {added_recipients} new recipient(s)"
);
}
Ok(())
}

107
crates/radicle-cli/src/commands/lfs/store.rs Normal file
View File

@ -0,0 +1,107 @@
//! `rad lfs store` — plumbing command invoked by the Git LFS custom
//! transfer agent (`rad-lfs-transfer`) to upload one LFS object's content
//! to the local IPFS (Kubo) node, encrypting it first if the repository
//! is private.
use std::path::Path;
use std::path::PathBuf;
use anyhow::Context as _;
use radicle::Profile;
use radicle::git::raw::{Repository, Signature};
use radicle::identity::{Doc, RepoId};
use radicle::storage::{ReadRepository as _, ReadStorage as _};
use radicle_crypto::ssh::keystore::MemorySigner;
use crate::ipfs;
use crate::lfs_crypto::{self, Envelope, LOCAL_NOTES_REF, NOTE_AUTHOR_EMAIL, NOTE_AUTHOR_NAME};
use crate::terminal as term;
pub fn run(oid: String, size: i64, path: PathBuf, ctx: impl term::Context) -> anyhow::Result<()> {
let profile = ctx.profile()?;
let (repo, rid) = radicle::rad::cwd()
.context("`rad lfs store` must be run inside a Radicle repository working copy")?;
let doc = profile.storage.repository(rid)?.identity_doc()?.doc;
let mut signer = None;
let cid = store_object(&profile, &repo, rid, &doc, &oid, size, &path, &mut signer)?;
term::println(cid);
Ok(())
}
/// Stores one LFS object's content in IPFS (encrypting first for private
/// repos) and records the resulting CID as a git note. Factored out of
/// [`run`] so `rad lfs precommit` can process every staged LFS file in a
/// single process, loading (and prompting for) the signer at most once
/// across the whole batch instead of once per file.
#[allow(clippy::too_many_arguments)]
pub fn store_object(
profile: &Profile,
repo: &Repository,
rid: RepoId,
doc: &Doc,
oid: &str,
size: i64,
path: &Path,
signer: &mut Option<MemorySigner>,
) -> anyhow::Result<String> {
let pointer_text =
format!("version https://git-lfs.github.com/spec/v1\noid sha256:{oid}\nsize {size}\n");
let blob_oid = repo
.blob(pointer_text.as_bytes())
.context("failed to write LFS pointer blob")?;
// Fast path: a note already recorded for this object (typically written
// moments ago by `rad lfs precommit`, or fetched from a peer) means the
// content is already pinned in IPFS -- nothing left to do. This matters
// beyond just avoiding duplicate work: `git push`'s custom-transfer-agent
// subprocess chain has no TTY (its stdio is consumed by the transfer
// protocol, not a terminal), so without this fast path, a private repo's
// `git push` would call back into `store_object` per file and fail
// outright on the passphrase prompt every time, even though the actual
// encryption work was already done (and the passphrase already
// supplied) once, interactively, at commit time.
if let Some(cid) = lfs_crypto::find_cids(repo, blob_oid)?.into_iter().next() {
return Ok(cid);
}
ipfs::check_daemon()?;
let envelope = if doc.visibility().is_public() {
let bytes = std::fs::read(path)
.with_context(|| format!("failed to read `{}`", path.display()))?;
let cid = ipfs::add(&bytes)?;
let _ = ipfs::pin_all(std::slice::from_ref(&cid));
Envelope::plain(cid)
} else {
if signer.is_none() {
*signer = Some(lfs_crypto::load_signer(profile)?);
}
let signer = signer.as_ref().expect("just populated above");
let sender_did = profile.did();
let recipients = lfs_crypto::recipients_for(doc, sender_did);
let plaintext = std::fs::read(path)
.with_context(|| format!("failed to read `{}`", path.display()))?;
let (ciphertext, enc) =
lfs_crypto::encrypt_for(&plaintext, signer, sender_did, &rid, oid, &recipients)?;
let cid = ipfs::add(&ciphertext)?;
let _ = ipfs::pin_all(std::slice::from_ref(&cid));
Envelope {
v: 1,
cid,
enc: Some(enc),
}
};
let cid = envelope.cid.clone();
let message =
serde_json::to_string(&envelope).context("failed to serialize LFS note envelope")?;
let signature = Signature::now(NOTE_AUTHOR_NAME, NOTE_AUTHOR_EMAIL)
.context("failed to construct note signature")?;
repo.note(&signature, &signature, Some(LOCAL_NOTES_REF), blob_oid, &message, true)
.context("failed to write LFS note")?;
Ok(cid)
}

13
crates/radicle-cli/src/commands/seed.rs
View File

@ -70,6 +70,19 @@ pub fn update(
term::format::tertiary(rid),
);
// Since seeding a repository means keeping a full local copy of it,
// also pin any of its Git-LFS objects that are stored in IPFS, so
// that they follow the same seed/unseed lifecycle.
if let Ok(repo) = profile.storage.repository(rid) {
let cids = crate::ipfs::lfs_cids(&repo.backend);
if !cids.is_empty() {
let pinned = crate::ipfs::pin_all(&cids);
if pinned > 0 {
term::success!("Pinned {pinned} LFS object(s) in IPFS");
}
}
}
Ok(())
}

65
crates/radicle-cli/src/commands/unseed.rs
View File

@ -1,5 +1,8 @@
pub mod args;
use std::collections::BTreeSet;
use radicle::node::policy;
use radicle::{Node, prelude::*};
use crate::terminal as term;
@ -18,8 +21,70 @@ pub fn run(args: Args, ctx: impl term::Context) -> anyhow::Result<()> {
}
pub fn delete(rid: RepoId, node: &mut Node, profile: &Profile) -> anyhow::Result<()> {
// Gather the repository's known LFS objects *before* unseeding, since
// we still need access to its git notes at that point.
let cids = profile
.storage
.repository(rid)
.map(|repo| crate::ipfs::lfs_cids(&repo.backend))
.unwrap_or_default();
if profile.unseed(rid, node)? {
term::success!("Seeding policy for {} removed", term::format::tertiary(rid));
}
// Mirror the seed/unseed lifecycle for IPFS pins of this repository's
// Git-LFS objects -- except any CID also referenced by an LFS note in
// some *other* still-seeded repository (e.g. two repositories that
// happen to both track a byte-identical file, giving it the same
// content-addressed CID). Unpinning unconditionally would evict
// content the other repository still needs, breaking it silently the
// next time someone there runs `git lfs fetch`. There's no
// cross-repository pin refcount kept anywhere -- this recomputes the
// still-needed set from every other seeded repository's own notes each
// time, which costs a pass over every other seeded repository but
// requires no extra persistent state, consistent with this design's
// git-notes-as-source-of-truth approach everywhere else.
if !cids.is_empty() {
let still_needed = cids_still_needed_elsewhere(profile, rid)?;
let skipped = cids.iter().filter(|cid| still_needed.contains(*cid)).count();
let safe_to_unpin: Vec<String> =
cids.into_iter().filter(|cid| !still_needed.contains(cid)).collect();
if !safe_to_unpin.is_empty() {
crate::ipfs::unpin_all(&safe_to_unpin);
}
if skipped > 0 {
term::info!(
"Kept {skipped} IPFS object(s) pinned -- still referenced by another seeded repository"
);
}
}
Ok(())
}
/// The set of CIDs (among `excluding`'s own) that are also referenced by an
/// LFS note in some other repository this node is currently seeding.
fn cids_still_needed_elsewhere(
profile: &Profile,
excluding: RepoId,
) -> anyhow::Result<BTreeSet<String>> {
let mut needed = BTreeSet::new();
let store = profile.policies()?;
for entry in store.seed_policies()? {
let policy::SeedPolicy { rid, policy } = entry?;
if rid == excluding {
continue;
}
if !matches!(policy, policy::SeedingPolicy::Allow { .. }) {
continue;
}
if let Ok(repo) = profile.storage.repository(rid) {
needed.extend(crate::ipfs::lfs_cids(&repo.backend));
}
}
Ok(needed)
}

246
crates/radicle-cli/src/ipfs.rs Normal file
View File

@ -0,0 +1,246 @@
//! Pinning of Git-LFS-backed content in a local IPFS (Kubo) node.
//!
//! This is a prototype of "Git-LFS over IPFS": large files tracked via
//! Git LFS are stored in IPFS rather than as git blobs, and the mapping
//! from an LFS pointer file's git blob oid to the corresponding IPFS CID
//! is recorded as a git note on `refs/notes/rad-lfs` (note content is a
//! JSON [`crate::lfs_crypto::Envelope`], e.g. `{"v":1,"cid":"Qm...",
//! "enc":null}` for a public repo, or with populated `enc` metadata for
//! a private one).
//!
//! Since seeding a repository already means "keep a full local copy of
//! it", we extend that lifecycle to IPFS: seeding a repository pins all
//! of its known LFS objects locally, and unseeding unpins them.
use std::env;
use anyhow::anyhow;
use radicle::git::raw::Repository;
use crate::terminal as term;
/// Git notes ref under which oid -> CID mappings for LFS objects are stored.
///
/// This is the *local* ref name; readers need to merge across every
/// fetched peer's copy too (`refs/notes/rad-lfs/<peer>`) -- see
/// `crate::lfs_crypto::find_envelopes`/`all_note_targets`, which
/// `lfs_cids` below delegates to.
pub use crate::lfs_crypto::NOTES_REF as LFS_NOTES_REF;
/// Default HTTP API address of a local Kubo (IPFS) daemon.
const DEFAULT_KUBO_API_URL: &str = "http://127.0.0.1:5001";
/// Cap on how large an LFS object's content we'll read back from IPFS in
/// one go. `ureq`'s `Body::read_to_vec()` otherwise defaults to a 10MB
/// limit meant for typical HTTP API responses, which is far too small for
/// LFS content (that's the whole point of storing it out-of-band). 1GiB
/// comfortably covers any file worth tracking in this repo; raise if a
/// larger LFS object legitimately needs to be fetched.
const MAX_CAT_RESPONSE_BYTES: u64 = 1024 * 1024 * 1024;
/// Returns the Kubo HTTP API base URL, respecting the `KUBO_API_URL`
/// environment variable if set.
pub(crate) fn kubo_api_url() -> String {
env::var("KUBO_API_URL").unwrap_or_else(|_| DEFAULT_KUBO_API_URL.to_string())
}
/// Check that a local IPFS (Kubo) daemon is reachable, failing with a
/// clear, actionable error otherwise. Used by `rad lfs init` to refuse to
/// proceed silently when there's no daemon to back LFS content with.
pub fn check_daemon() -> anyhow::Result<()> {
let base = kubo_api_url();
let url = format!("{base}/api/v0/id");
match ureq::post(&url).send_empty() {
Ok(_) => Ok(()),
Err(err) if is_daemon_unreachable(&err) => Err(anyhow!(
"no IPFS (Kubo) daemon reachable at {base} — start one with `ipfs daemon`.\n\
This repo's large-file (LFS) support is backed by your local IPFS node."
)),
Err(err) => Err(anyhow!("IPFS daemon at {base} responded unexpectedly: {err}")),
}
}
/// Collect the set of CIDs referenced by LFS notes, across every peer's
/// notes ref (not just our own — a repo we're seeding may hold objects
/// whose only note came from someone else's `refs/notes/rad-lfs/<peer>`).
///
/// Returns an empty vector if no notes exist, i.e. the repository has no
/// LFS objects tracked yet. This is not an error.
#[must_use]
pub fn lfs_cids(repo: &Repository) -> Vec<String> {
let Ok(targets) = crate::lfs_crypto::all_note_targets(repo) else {
return Vec::new();
};
let mut cids = Vec::new();
for target in targets {
if let Ok(found) = crate::lfs_crypto::find_cids(repo, target) {
cids.extend(found);
}
}
cids
}
/// Add `bytes` to the local IPFS node as a single blob, returning its CID.
pub fn add(bytes: &[u8]) -> anyhow::Result<String> {
use ureq::unversioned::multipart::{Form, Part};
let base = kubo_api_url();
let url = format!("{base}/api/v0/add");
let form = Form::new().part("file", Part::bytes(bytes).file_name("blob"));
let mut response = ureq::post(&url).send(form).map_err(|err| {
if is_daemon_unreachable(&err) {
anyhow!(
"no IPFS (Kubo) daemon reachable at {base} — start one with `ipfs daemon`.\n\
This repo's large-file (LFS) support is backed by your local IPFS node."
)
} else {
anyhow!("failed to add object to IPFS: {err}")
}
})?;
let body = response
.body_mut()
.read_to_string()
.map_err(|err| anyhow!("failed to read IPFS `add` response: {err}"))?;
// The response is newline-delimited JSON, one object per added file;
// we only ever add a single blob per call, so the first line is all
// we need.
let line = body
.lines()
.next()
.ok_or_else(|| anyhow!("empty response from IPFS `add`"))?;
let value: serde_json::Value = serde_json::from_str(line)
.map_err(|err| anyhow!("failed to parse IPFS `add` response: {err}"))?;
let hash = value
.get("Hash")
.and_then(|h| h.as_str())
.ok_or_else(|| anyhow!("IPFS `add` response missing a `Hash` field"))?;
Ok(hash.to_string())
}
/// Fetch the raw bytes of the object identified by `cid` from the local
/// IPFS node.
pub fn cat(cid: &str) -> anyhow::Result<Vec<u8>> {
let base = kubo_api_url();
let url = format!("{base}/api/v0/cat?arg={cid}");
let mut response = ureq::post(&url).send_empty().map_err(|err| {
if is_daemon_unreachable(&err) {
anyhow!(
"no IPFS (Kubo) daemon reachable at {base} — start one with `ipfs daemon`.\n\
This repo's large-file (LFS) support is backed by your local IPFS node."
)
} else {
anyhow!("failed to fetch object {cid} from IPFS: {err}")
}
})?;
response
.body_mut()
.with_config()
.limit(MAX_CAT_RESPONSE_BYTES)
.read_to_vec()
.map_err(|err| anyhow!("failed to read IPFS `cat` response for {cid}: {err}"))
}
/// Whether a `ureq` error indicates that the local IPFS daemon could
/// not be reached at all, as opposed to an HTTP-level error response.
fn is_daemon_unreachable(err: &ureq::Error) -> bool {
matches!(
err,
ureq::Error::ConnectionFailed
| ureq::Error::Io(_)
| ureq::Error::HostNotFound
| ureq::Error::Timeout(_)
)
}
/// Pin the given CIDs recursively in the local IPFS daemon.
///
/// Returns the number of CIDs successfully pinned. If the local daemon
/// is not reachable, a warning is printed and `0` is returned; this is
/// not treated as a hard failure, since seeding itself already succeeded.
#[must_use]
pub fn pin_all(cids: &[String]) -> usize {
let base = kubo_api_url();
let mut pinned = 0;
for cid in cids {
let url = format!("{base}/api/v0/pin/add?arg={cid}&recursive=true");
match ureq::post(&url).send_empty() {
Ok(_) => pinned += 1,
Err(err) if is_daemon_unreachable(&err) => {
term::warning(format!(
"Could not reach local IPFS daemon at {base}; LFS objects were not pinned"
));
return pinned;
}
Err(err) => {
term::warning(format!("Failed to pin LFS object {cid} in IPFS: {err}"));
}
}
}
pinned
}
/// Unpin the given CIDs from the local IPFS daemon.
///
/// If the local daemon is not reachable, a warning is printed and the
/// function returns without failing; this is not treated as a hard
/// failure, since unseeding itself already succeeded.
pub fn unpin_all(cids: &[String]) {
let base = kubo_api_url();
for cid in cids {
let url = format!("{base}/api/v0/pin/rm?arg={cid}");
// NOTE: this unconditionally unpins `cid`, even if it's still
// referenced by LFS objects of another repository that remains
// seeded. This prototype has no cross-repo pin refcounting, so
// a CID shared between repositories will be unpinned here
// regardless of whether other seeded repos still need it.
match ureq::post(&url).send_empty() {
Ok(_) => {}
Err(err) if is_daemon_unreachable(&err) => {
term::warning(format!(
"Could not reach local IPFS daemon at {base}; LFS objects were not unpinned"
));
return;
}
Err(err) => {
term::warning(format!("Failed to unpin LFS object {cid} in IPFS: {err}"));
}
}
}
}
#[cfg(test)]
mod tests {
use super::*;
/// Regression test for the `cat()` response being truncated at
/// `ureq`'s default 10MB read limit for any LFS object larger than
/// that (e.g. the ~29MB `assets/posts/hashcash/thumbnail.xcf` in the
/// blog repo failed to smudge with exactly this error before the
/// `MAX_CAT_RESPONSE_BYTES` fix). Requires a local Kubo daemon, so
/// it's `#[ignore]`d by default: `cargo test -- --ignored`.
#[test]
#[ignore = "requires a local IPFS (Kubo) daemon at 127.0.0.1:5001"]
fn cat_round_trips_content_larger_than_ureqs_default_10mb_limit() {
check_daemon().expect("local Kubo daemon must be reachable for this test");
let bytes = vec![0x42u8; 15 * 1024 * 1024]; // 15MB, past the old 10MB cap
let cid = add(&bytes).expect("add should succeed");
let fetched = cat(&cid).expect("cat should succeed for content past the old 10MB cap");
assert_eq!(fetched.len(), bytes.len());
assert_eq!(fetched, bytes);
unpin_all(std::slice::from_ref(&cid));
}
}

418
crates/radicle-cli/src/lfs_crypto.rs Normal file
View File

@ -0,0 +1,418 @@
//! Client-side encryption for Git-LFS-over-IPFS objects on private
//! repositories.
//!
//! Radicle's `Visibility::Private` is enforced entirely by access control
//! at the replication layer -- git objects for a private repo are plain,
//! unencrypted objects, protected only by "unauthorized peers are never
//! given them". That protection has no jurisdiction over IPFS's own
//! block-exchange layer: once a legitimate collaborator's IPFS daemon
//! pins LFS content and joins the network, anyone who obtains the CID by
//! any means can fetch the plaintext directly. This module makes sure
//! IPFS only ever stores ciphertext for private repos, using Radicle's
//! own identity keys via envelope encryption (a random per-object content
//! key, wrapped once per authorized recipient).
use std::collections::BTreeSet;
use anyhow::{Context as _, anyhow, bail};
use base64::Engine as _;
use base64::engine::general_purpose::STANDARD as BASE64;
use chacha20poly1305::aead::rand_core::RngCore;
use chacha20poly1305::aead::{Aead, AeadCore, KeyInit, OsRng};
use chacha20poly1305::{Key, XChaCha20Poly1305, XNonce};
use cyphernet::Ecdh;
use hkdf::Hkdf;
use radicle::Profile;
use radicle::git::raw::{Oid, Repository};
use radicle::identity::{Did, Doc, RepoId, Visibility};
use radicle_crypto::ssh::keystore::MemorySigner;
use serde::{Deserialize, Serialize};
use sha2::Sha256;
use zeroize::Zeroizing;
pub const ENVELOPE_VERSION: u32 = 1;
const HKDF_DOMAIN: &[u8] = b"radicle-lfs-wrap-v1";
const ALG: &str = "xchacha20poly1305";
/// The canonical, bare notes ref name. This is what the *remote* uses --
/// it's the push destination (lands in the pusher's own namespace on the
/// server regardless of the local source ref, same as `refs/heads/*`) and
/// what `crates/radicle-remote-helper/src/list.rs`'s `lfs_notes_refs`
/// looks for within each peer's namespace server-side.
///
/// It is deliberately **not** used as a local write target anymore (see
/// [`LOCAL_NOTES_REF`]): `rad lfs init`'s fetch refspec populates
/// `refs/notes/rad-lfs/<peer>` siblings locally, including our own peer's
/// copy fetched back after a push. A bare `refs/notes/rad-lfs` ref
/// coexisting with `refs/notes/rad-lfs/<peer>` is a git ref D/F
/// (file-vs-directory) conflict -- `refs/notes/rad-lfs` can't
/// simultaneously be a leaf ref and a directory prefix in the same ref
/// namespace -- which made every `rad lfs store` call after the first
/// fetch fail outright with "failed to write LFS note".
pub const NOTES_REF: &str = "refs/notes/rad-lfs";
/// The local, not-yet-pushed note ref -- `rad lfs store`/`rekey` write
/// here instead of the bare [`NOTES_REF`], specifically to avoid the D/F
/// conflict above: living under the same `refs/notes/rad-lfs/` prefix as
/// every fetched peer ref means it's just another sibling leaf ref, never
/// a conflicting directory/file pair. `rad lfs init`'s push refspec maps
/// this local name to the bare [`NOTES_REF`] on the remote.
pub const LOCAL_NOTES_REF: &str = "refs/notes/rad-lfs/local";
/// Glob matching [`LOCAL_NOTES_REF`] and every fetched peer ref, plus (for
/// backward compatibility while any not-yet-migrated repo or peer might
/// still have one) the legacy bare [`NOTES_REF`] itself, purely for
/// reading -- nothing writes there anymore.
const NOTES_REF_GLOB: &str = "refs/notes/rad-lfs*";
/// Committer identity used for `refs/notes/rad-lfs/*` notes. There's no
/// natural "author" for a note that just records a CID, so a fixed
/// identity is used everywhere a note gets written.
pub const NOTE_AUTHOR_NAME: &str = "rad-lfs";
pub const NOTE_AUTHOR_EMAIL: &str = "rad-lfs@localhost";
/// Note content stored on `refs/notes/rad-lfs`, replacing the earlier
/// bare `cid=<cid>` text. `enc` is `None` for public repos (plain
/// passthrough, unchanged behavior).
#[derive(Debug, Serialize, Deserialize)]
pub struct Envelope {
pub v: u32,
pub cid: String,
#[serde(default, skip_serializing_if = "Option::is_none")]
pub enc: Option<Encryption>,
}
impl Envelope {
#[must_use]
pub fn plain(cid: String) -> Self {
Self {
v: ENVELOPE_VERSION,
cid,
enc: None,
}
}
}
#[derive(Debug, Serialize, Deserialize)]
pub struct Encryption {
pub alg: String,
/// Nonce for the content AEAD. Fixed for the object's life -- `rekey`
/// never re-encrypts content, only adds wrapped-key entries.
pub content_nonce: String,
pub recipients: Vec<WrappedKey>,
}
/// One recipient's wrapped copy of the object's content-encryption key.
/// `sender_did` is per-entry, not per-envelope: a `rekey` operation is
/// performed by a different identity (using their own key) than whoever
/// originally committed the object, so each entry must record who
/// actually produced it.
#[derive(Debug, Serialize, Deserialize)]
pub struct WrappedKey {
pub did: Did,
pub sender_did: Did,
pub salt: String,
pub wrapped_cek: String,
}
/// Recipients authorized for a repository: delegates, plus the private
/// allow-list (empty for public repos), plus the sender themselves --
/// don't rely on "delegates always includes the author", make it explicit
/// so the committer can always fetch their own commits back.
#[must_use]
pub fn recipients_for(doc: &Doc, sender: Did) -> BTreeSet<Did> {
let mut set: BTreeSet<Did> = doc.delegates().iter().copied().collect();
if let Visibility::Private { allow } = doc.visibility() {
set.extend(allow.iter().copied());
}
set.insert(sender);
set
}
/// Loads a signer capable of ECDH. Only `MemorySigner` implements the
/// `Ecdh` trait -- ssh-agent (the normal day-to-day flow after `rad
/// auth`) can sign but can't expose the raw key material a key-agreement
/// operation needs. This is a hard limitation of the standard ssh-agent
/// protocol, which only implements signing operations, not general key
/// agreement -- there's no ssh-agent request type for it, so this can't
/// be routed through the agent the way signing is.
///
/// The next best thing, matching the exact fallback
/// `crate::terminal::io::signer` already uses elsewhere in this CLI when
/// ssh-agent isn't available or the key isn't registered with it: if
/// connected to a TTY, prompt interactively for the passphrase instead of
/// requiring `RAD_PASSPHRASE` to be pre-set. This covers the common case
/// (an interactive `git commit` triggering the pre-commit hook, which
/// inherits the terminal's TTY) without needing an env var at all; only
/// genuinely non-interactive contexts (CI, a script with no TTY) still
/// need `RAD_PASSPHRASE`, and get a fast, clear failure rather than a
/// hang, since `inquire` (via `radicle_term::io::passphrase`) returns
/// `Ok(None)` on `NotTTY` rather than blocking.
pub fn load_signer(profile: &Profile) -> anyhow::Result<MemorySigner> {
if !profile.keystore.is_encrypted()? {
return Ok(MemorySigner::load(&profile.keystore, None)?);
}
if let Some(passphrase) = radicle::profile::env::passphrase() {
return Ok(MemorySigner::load(&profile.keystore, Some(passphrase))?);
}
if let Some(passphrase) = prompt_passphrase(profile)? {
return Ok(MemorySigner::load(&profile.keystore, Some(passphrase))?);
}
bail!(
"private-repo LFS operations need to perform a key-agreement (ECDH) operation, which \
requires direct access to your secret key -- ssh-agent can sign but can't do this \
(a protocol limitation, not something to configure around). Run this from a \
terminal to be prompted for your passphrase, or set RAD_PASSPHRASE, or use an \
unencrypted keystore."
)
}
/// Prompts for the keystore passphrase if connected to a TTY. Returns
/// `Ok(None)` (not an error) when there's no TTY to prompt on, so callers
/// can fall through to a clear error instead of hanging.
fn prompt_passphrase(profile: &Profile) -> anyhow::Result<Option<radicle_crypto::ssh::keystore::Passphrase>> {
let validator = crate::terminal::io::PassphraseValidator::new(profile.keystore.clone());
Ok(crate::terminal::io::passphrase(validator)?)
}
/// Encrypts `plaintext` with a fresh content key, then wraps that key once
/// per recipient. Returns the ciphertext to store in IPFS and the
/// encryption metadata to record in the note.
pub fn encrypt_for(
plaintext: &[u8],
sender: &MemorySigner,
sender_did: Did,
rid: &RepoId,
oid: &str,
recipients: &BTreeSet<Did>,
) -> anyhow::Result<(Vec<u8>, Encryption)> {
let cek = XChaCha20Poly1305::generate_key(&mut OsRng);
let content_nonce = XChaCha20Poly1305::generate_nonce(&mut OsRng);
let cipher = XChaCha20Poly1305::new(&cek);
let ciphertext = cipher
.encrypt(&content_nonce, plaintext)
.map_err(|_| anyhow!("content encryption failed"))?;
let mut wrapped = Vec::with_capacity(recipients.len());
for recipient in recipients {
wrapped.push(wrap_cek_for(&cek, sender, sender_did, recipient, rid, oid)?);
}
Ok((
ciphertext,
Encryption {
alg: ALG.to_string(),
content_nonce: BASE64.encode(content_nonce),
recipients: wrapped,
},
))
}
pub fn wrap_cek_for(
cek: &Key,
sender: &MemorySigner,
sender_did: Did,
recipient: &Did,
rid: &RepoId,
oid: &str,
) -> anyhow::Result<WrappedKey> {
let shared = sender
.ecdh(recipient.as_key())
.map_err(|e| anyhow!("ECDH with {recipient} failed: {e}"))?;
let mut salt = [0u8; 16];
OsRng.fill_bytes(&mut salt);
let wrap_key = derive_wrap_key(&shared, &salt, rid, oid, &sender_did, recipient);
// Fixed all-zero nonce is safe here: `wrap_key` is derived via HKDF
// from a salt that's freshly generated for this exact
// (rid, oid, sender, recipient) wrap operation, so it is never reused
// to encrypt more than this one 32-byte CEK -- key-once nonce reuse
// is not a vulnerability.
let cipher = XChaCha20Poly1305::new(&wrap_key);
let wrapped_cek = cipher
.encrypt(&XNonce::default(), cek.as_slice())
.map_err(|_| anyhow!("failed to wrap key for {recipient}"))?;
Ok(WrappedKey {
did: *recipient,
sender_did,
salt: BASE64.encode(salt),
wrapped_cek: BASE64.encode(wrapped_cek),
})
}
/// HKDF context mixes in more than the raw ECDH shared secret: the shared
/// secret between two fixed DIDs is *static*, reused across every object
/// they ever exchange. Without per-object context here, the "nonce fixed
/// because key-once" argument in `wrap_cek_for` would be false -- the same
/// key could end up wrapping multiple different CEKs. Mixing in `rid` +
/// `oid` (plus a fresh random `salt`) makes every wrap operation
/// cryptographically independent, even between the same two DIDs.
fn derive_wrap_key(
shared: &[u8; 32],
salt: &[u8],
rid: &RepoId,
oid: &str,
sender_did: &Did,
recipient_did: &Did,
) -> Key {
let hk = Hkdf::<Sha256>::new(Some(salt), shared);
let mut info = Vec::new();
info.extend_from_slice(HKDF_DOMAIN);
info.extend_from_slice(rid.to_string().as_bytes());
info.extend_from_slice(oid.as_bytes());
info.extend_from_slice(sender_did.to_string().as_bytes());
info.extend_from_slice(recipient_did.to_string().as_bytes());
let mut out = [0u8; 32];
hk.expand(&info, &mut out)
.expect("32 is a valid HKDF-SHA256 output length");
out.into()
}
/// Recovers the content-encryption key from the caller's own entry among a
/// set of wrapped keys, without needing the ciphertext at all -- this is
/// the exact primitive `rad lfs rekey` needs (it only adds new wrapped-key
/// entries for the same never-changing CEK, it never re-encrypts content).
pub fn unwrap_cek(
recipients: &[WrappedKey],
me: &MemorySigner,
my_did: Did,
rid: &RepoId,
oid: &str,
) -> anyhow::Result<Zeroizing<[u8; 32]>> {
let entry = recipients
.iter()
.find(|w| w.did == my_did)
.ok_or_else(|| anyhow!("you are not an authorized recipient of this object"))?;
let shared = me
.ecdh(entry.sender_did.as_key())
.map_err(|e| anyhow!("ECDH with {} failed: {e}", entry.sender_did))?;
let salt = BASE64
.decode(&entry.salt)
.context("invalid salt in wrapped key entry")?;
let wrap_key = derive_wrap_key(&shared, &salt, rid, oid, &entry.sender_did, &my_did);
let wrapped_cek = BASE64
.decode(&entry.wrapped_cek)
.context("invalid wrapped content key")?;
let cipher = XChaCha20Poly1305::new(&wrap_key);
let cek = cipher
.decrypt(&XNonce::default(), wrapped_cek.as_slice())
.map_err(|_| anyhow!("failed to unwrap key -- wrong key or corrupted data"))?;
let mut out = [0u8; 32];
out.copy_from_slice(&cek);
Ok(Zeroizing::new(out))
}
/// Decrypts object content using the caller's own wrapped-key entry.
pub fn decrypt_with(
ciphertext: &[u8],
enc: &Encryption,
me: &MemorySigner,
my_did: Did,
rid: &RepoId,
oid: &str,
) -> anyhow::Result<Vec<u8>> {
let cek = unwrap_cek(&enc.recipients, me, my_did, rid, oid)?;
let nonce_bytes = BASE64
.decode(&enc.content_nonce)
.context("invalid content nonce")?;
let nonce = XNonce::from_slice(&nonce_bytes);
let key: Key = (*cek).into();
let cipher = XChaCha20Poly1305::new(&key);
cipher
.decrypt(nonce, ciphertext)
.map_err(|_| anyhow!("content decryption failed -- wrong key or corrupted data"))
}
/// Finds every distinct envelope recorded for `blob_oid`, across the
/// local `refs/notes/rad-lfs` ref and every fetched peer ref under
/// `refs/notes/rad-lfs/<peer>`. Notes are per-peer contributions -- any
/// peer can commit an LFS-tracked file, not just delegates -- so more
/// than one peer's note can exist for the same object, e.g. after a
/// `rad lfs rekey` that added recipients under a different peer's
/// namespace than the original committer.
///
/// Notes sharing the same `cid` are the *same* encrypted object (rekey
/// never changes the ciphertext, only adds wrapped-key entries for it),
/// so their recipient lists are safely unioned together. Notes with
/// *different* `cid`s (which can only happen if two peers independently
/// encrypted byte-identical plaintext -- same oid/size, since that's
/// what determines the pointer blob -- producing different ciphertext
/// each time since encryption uses a fresh random key/nonce) are kept as
/// separate candidates, since a wrapped-key entry from one is only valid
/// against its own ciphertext, never the other's. Callers that need to
/// decrypt should try each returned candidate in turn.
pub fn find_envelopes(repo: &Repository, blob_oid: Oid) -> anyhow::Result<Vec<Envelope>> {
let mut by_cid: Vec<Envelope> = Vec::new();
for name in notes_ref_names(repo)? {
let Ok(note) = repo.find_note(Some(&name), blob_oid) else {
continue;
};
let Some(message) = note.message() else {
continue;
};
let Ok(envelope) = serde_json::from_str::<Envelope>(message) else {
continue;
};
match by_cid.iter_mut().find(|e| e.cid == envelope.cid) {
Some(existing) => merge_recipients(existing, envelope),
None => by_cid.push(envelope),
}
}
Ok(by_cid)
}
/// Convenience wrapper for callers that only need the set of distinct
/// CIDs recorded for `blob_oid` (e.g. seed/unseed pinning, which needs
/// no key material and doesn't care about recipients at all).
pub fn find_cids(repo: &Repository, blob_oid: Oid) -> anyhow::Result<Vec<String>> {
Ok(find_envelopes(repo, blob_oid)?
.into_iter()
.map(|e| e.cid)
.collect())
}
fn merge_recipients(into: &mut Envelope, other: Envelope) {
let (Some(into_enc), Some(other_enc)) = (into.enc.as_mut(), other.enc) else {
return;
};
let existing: BTreeSet<Did> = into_enc.recipients.iter().map(|w| w.did).collect();
into_enc
.recipients
.extend(other_enc.recipients.into_iter().filter(|w| !existing.contains(&w.did)));
}
/// Every distinct blob oid that has at least one note recorded against it,
/// across every notes ref (local plus every fetched peer). Used by
/// `rad lfs rekey` to walk every known LFS object, since a single
/// `notes()` call on one ref would now miss objects whose only note lives
/// under a different peer's ref.
pub fn all_note_targets(repo: &Repository) -> anyhow::Result<BTreeSet<Oid>> {
let mut targets = BTreeSet::new();
for name in notes_ref_names(repo)? {
let Ok(notes) = repo.notes(Some(&name)) else {
continue;
};
for (_, target_oid) in notes.flatten() {
targets.insert(target_oid);
}
}
Ok(targets)
}
fn notes_ref_names(repo: &Repository) -> anyhow::Result<Vec<String>> {
let mut names = Vec::new();
for reference in repo.references_glob(NOTES_REF_GLOB)? {
let reference = reference?;
if let Some(name) = reference.name() {
names.push(name.to_string());
}
}
Ok(names)
}

2
crates/radicle-cli/src/lib.rs
View File

@ -4,6 +4,8 @@
#![deny(clippy::print_stdout)]
pub mod commands;
pub mod git;
pub mod ipfs;
pub mod lfs_crypto;
pub mod node;
pub mod pager;
pub mod project;

2
crates/radicle-cli/src/main.rs
View File

@ -76,6 +76,7 @@ enum Command {
#[command(alias = ".")]
Inspect(inspect::Args),
Issue(issue::Args),
Lfs(lfs::Args),
Ls(ls::Args),
Node(node::Args),
Patch(patch::Args),
@ -271,6 +272,7 @@ fn run_command(command: Command, ctx: impl term::Context) -> Result<(), anyhow::
Command::Init(args) => init::run(args, ctx),
Command::Inspect(args) => inspect::run(args, ctx),
Command::Issue(args) => issue::run(args, ctx),
Command::Lfs(args) => lfs::run(args, ctx),
Command::Ls(args) => ls::run(args, ctx),
Command::Node(args) => node::run(args, ctx),
Command::Patch(args) => patch::run(args, ctx),

31
crates/radicle-remote-helper/src/list.rs
View File

@ -32,6 +32,9 @@ pub(super) enum Error {
/// General repository error.
#[error(transparent)]
Repository(#[from] radicle::storage::RepositoryError),
/// Refs error.
#[error(transparent)]
Refs(#[from] radicle::storage::refs::Error),
}
/// List refs for fetching (`git fetch` and `git ls-remote`).
@ -73,6 +76,34 @@ pub(super) fn for_fetch<R: ReadRepository + cob::Store<Namespace = NodeId> + 'st
Ok(mut refs) => lines.append(&mut refs),
Err(e) => eprintln!("remote: error listing patch refs: {e}"),
}
// List each peer's `refs/notes/rad-lfs`, individually, under
// `refs/notes/rad-lfs/<peer-id>`. Notes are per-peer
// contributions -- any peer (not just delegates) can commit an
// LFS-tracked file and write a note for it -- so unlike
// `refs/heads`/`refs/tags` there is no single canonical value to
// resolve here; the reading side merges across every peer's note
// for a given object instead (see `radicle-cli`'s
// `lfs_crypto::find_envelope`). Do not abort the whole fetch if
// this fails, same as patch refs above.
match lfs_notes_refs(stored) {
Ok(mut refs) => lines.append(&mut refs),
Err(e) => eprintln!("remote: error listing LFS notes refs: {e}"),
}
}
Ok(lines)
}
/// List every peer's `refs/notes/rad-lfs`, under `refs/notes/rad-lfs/<peer-id>`.
fn lfs_notes_refs<R: ReadRepository>(stored: &R) -> Result<Vec<String>, Error> {
let mut lines = Vec::new();
let notes_ref = git::fmt::qualified!("refs/notes/rad-lfs");
for (peer, _) in stored.remotes()? {
if let Ok(oid) = stored.reference_oid(&peer, &notes_ref) {
lines.push(format!("{oid} refs/notes/rad-lfs/{peer}"));
}
}
Ok(lines)

1
radicle-lfs-transfer Submodule

@ -0,0 +1 @@
Subproject commit 3215f231d80752266d2438f760c9dc244f1fc828

163
scripts/publish-package.sh Executable file
View File

@ -0,0 +1,163 @@
#!/usr/bin/env bash
#
# Builds release binaries for the LFS-over-IPFS stack (rad, radicle-node,
# git-remote-rad, rad-lfs-transfer) and publishes them as a Forgejo generic
# package, so a deploy target can `curl` a tarball instead of needing a full
# Rust toolchain (see NOTES-lfs-store-note-write-bug.md's VPS deployment
# story for why that matters -- rebuilding in place isn't always possible).
#
# Usage:
# FORGEJO_TOKEN=... ./scripts/publish-package.sh [version]
#
# `version` defaults to `git describe --tags --always --dirty`, sanitized
# for use in a URL path. The token needs the `write:package` scope; it's
# read from $FORGEJO_TOKEN, falling back to ~/.config/forgejo/token if that
# env var isn't set (chmod 600 that file if you use it).
#
# Config (override via env):
# FORGEJO_HOST default: git.hswro.org
# FORGEJO_OWNER default: mab122
# PACKAGE_NAME default: radicle-lfs-ipfs
set -euo pipefail
FORGEJO_HOST="${FORGEJO_HOST:-git.hswro.org}"
FORGEJO_OWNER="${FORGEJO_OWNER:-mab122}"
PACKAGE_NAME="${PACKAGE_NAME:-radicle-lfs-ipfs}"
repo_root="$(git rev-parse --show-toplevel)"
cd "$repo_root"
if [ -n "${FORGEJO_TOKEN:-}" ]; then
token="$FORGEJO_TOKEN"
elif [ -r "$HOME/.config/forgejo/token" ]; then
token="$(cat "$HOME/.config/forgejo/token")"
else
echo "error: no Forgejo API token found. Set \$FORGEJO_TOKEN, or put one" \
"(scope: write:package) in ~/.config/forgejo/token (chmod 600)." >&2
exit 1
fi
version_raw="${1:-$(git describe --tags --always --dirty)}"
# Forgejo generic package versions must be URL-path-safe; a `git describe`
# with a "-dirty" suffix or a leading "v" is otherwise fine, but sanitize
# defensively rather than let a stray character break the upload URL.
version="$(printf '%s' "$version_raw" | tr -c 'A-Za-z0-9._-' '-')"
if [ ! -f "$repo_root/radicle-lfs-transfer/Cargo.toml" ]; then
echo "error: radicle-lfs-transfer/ submodule is empty -- run" \
"'git submodule update --init' first." >&2
exit 1
fi
existing_versions="$(curl -sS "https://${FORGEJO_HOST}/api/v1/packages/${FORGEJO_OWNER}?type=generic" \
-H "Authorization: token ${token}" \
| grep -o "\"name\":\"${PACKAGE_NAME}\",\"version\":\"[^\"]*\"" \
| sed -E 's/.*"version":"([^"]*)"/\1/' || true)"
if printf '%s\n' "$existing_versions" | grep -qx "$version"; then
echo "==> $PACKAGE_NAME $version is already published:" \
"https://${FORGEJO_HOST}/${FORGEJO_OWNER}/-/packages/generic/${PACKAGE_NAME}/${version}"
exit 0
fi
# Changelog since whichever previously-published version's commit we can
# find (a `git describe`-style version ends in "g<short-sha>"). Falls back
# to the full log if there's no previous package, or its version string
# doesn't parse -- better a long changelog than a failed release.
prev_sha=""
if [ -n "$existing_versions" ]; then
prev_sha="$(printf '%s\n' "$existing_versions" \
| grep -oE 'g[0-9a-f]{7,40}$' \
| sed 's/^g//' \
| head -1)"
fi
if [ -n "$prev_sha" ] && git cat-file -e "$prev_sha" 2>/dev/null; then
changelog_range="${prev_sha}..HEAD"
else
prev_sha=""
changelog_range="HEAD"
fi
changelog="$(git log --oneline "$changelog_range")"
if [ -z "$changelog" ]; then
changelog="(no commits since the previously published version)"
fi
echo "==> Building release binaries (version: $version)"
cargo build --release --locked -p radicle-cli --bin rad
cargo build --release --locked -p radicle-node
cargo build --release --locked -p radicle-remote-helper
cargo build --release --locked --manifest-path radicle-lfs-transfer/Cargo.toml
target_triple="$(rustc -vV | sed -n 's/host: //p')"
staging="$(mktemp -d)"
trap 'rm -rf "$staging"' EXIT
cp target/release/rad "$staging/"
cp target/release/radicle-node "$staging/"
cp target/release/git-remote-rad "$staging/"
cp radicle-lfs-transfer/target/release/rad-lfs-transfer "$staging/"
# Recorded because a binary built on a newer glibc can fail to run on an
# older one (discovered live deploying this exact stack to a Debian VPS) --
# anyone consuming this package needs to be able to check compatibility
# before copying the binaries over, not find out by them failing to start.
cat > "$staging/MANIFEST.txt" << EOF
package: $PACKAGE_NAME
version: $version
git commit: $(git rev-parse HEAD)
git branch: $(git rev-parse --abbrev-ref HEAD)
built: $(date -u +%Y-%m-%dT%H:%M:%SZ)
target triple: $target_triple
rustc: $(rustc --version)
glibc: $(ldd --version | head -1)
contents:
rad $(target/release/rad --version)
radicle-node $(target/release/radicle-node --version)
git-remote-rad $(target/release/git-remote-rad --version)
rad-lfs-transfer (no --version; speaks git-lfs custom-transfer protocol on stdin/stdout)
changelog (since ${prev_sha:-the beginning}):
$changelog
EOF
printf 'Changes since %s:\n\n%s\n' "${prev_sha:-the beginning}" "$changelog" > "$staging/CHANGELOG.txt"
archive_name="${PACKAGE_NAME}-${version}-${target_triple}.tar.gz"
archive_path="$staging/../${archive_name}"
tar -C "$staging" -czf "$archive_path" .
sha256sum "$archive_path" | awk '{print $1}' > "${archive_path}.sha256"
echo "==> Built $(du -h "$archive_path" | cut -f1) archive: $archive_name"
upload() {
local file="$1" name="$2"
local url="https://${FORGEJO_HOST}/api/packages/${FORGEJO_OWNER}/generic/${PACKAGE_NAME}/${version}/${name}"
local status
status="$(curl -sS -o /tmp/forgejo-upload-response.$$ -w '%{http_code}' \
-H "Authorization: token ${token}" \
-X PUT "$url" \
-T "$file")"
if [ "$status" = "201" ]; then
echo " uploaded: $name"
elif [ "$status" = "409" ]; then
echo " error: $name already exists at version $version -- bump the version" \
"or delete the existing package file in Forgejo first." >&2
cat /tmp/forgejo-upload-response.$$ >&2
rm -f /tmp/forgejo-upload-response.$$
exit 1
else
echo " error: upload of $name failed (HTTP $status)" >&2
cat /tmp/forgejo-upload-response.$$ >&2
rm -f /tmp/forgejo-upload-response.$$
exit 1
fi
rm -f /tmp/forgejo-upload-response.$$
}
echo "==> Uploading to https://${FORGEJO_HOST}/${FORGEJO_OWNER}/-/packages/generic/${PACKAGE_NAME}/${version}"
upload "$archive_path" "$archive_name"
upload "${archive_path}.sha256" "${archive_name}.sha256"
upload "$staging/CHANGELOG.txt" "CHANGELOG.txt"
echo "==> Done: https://${FORGEJO_HOST}/${FORGEJO_OWNER}/-/packages/generic/${PACKAGE_NAME}/${version}"