Code fix (35f1f972) is done, tested, and pushed. Package build + Forgejo
package-registry publish is blocked on missing packaging conventions
here (no existing publish script/credentials/target-scope decision) --
see NOTES-package-publish-todo.md for what's needed to finish this.
`ipfs::cat()` used ureq's `Body::read_to_vec()`, which defaults to a
10MB response-size limit meant for typical HTTP API responses. Any LFS
object larger than that (e.g. the ~29MB thumbnail.xcf in the blog repo)
failed with "the response body is larger than request limit: 10485760",
breaking `rad lfs fetch`/`rad lfs store`-driven smudges for large files.
Raise the cap to 1GiB via ureq's `with_config().limit(...)`, since LFS
content is exactly the out-of-band-large-file case this whole feature
exists for. Verified: reproduced the exact production error with a 15MB
round-trip test against a live Kubo daemon, confirmed it fails without
the fix and passes with it, and confirmed live against the actual
29MB thumbnail.xcf object that broke deploy.
`rad lfs fetch` still exists for single-object/manual use, but
`radicle-lfs-transfer` used to spawn a fresh one per object during
`git lfs pull`/checkout -- for a private repo, each one independently
loaded the signer, so N files meant N passphrase prompts. Same bug
class `rad lfs precommit` already fixed for commits, unfixed here
until now.
Unlike precommit (which knows every staged file upfront and can read
stdin until EOF), fetch can't collect a batch before starting:
git-lfs's custom-transfer protocol requests objects one at a time.
`rad lfs fetch-batch` instead stays alive for as long as its caller
keeps its stdin open, answering one "<oid> <size> <out-path>" request
at a time with a JSON response line, reusing the same unwrapped
signer across all of them -- paired with radicle-lfs-transfer's new
FetchWorker, which spawns this once per transfer session instead of
once per object.
Also factors `fetch::run`'s core logic out into `fetch_object`,
taking a `&mut Option<MemorySigner>` the same way `store::store_object`
already does, so both the single-shot `rad lfs fetch` and the new
batch worker share one implementation.
Two known gaps, both flagged as follow-up work in LFS-IPFS.md:
`rad unseed` unconditionally unpinned every CID a repository's LFS
notes referenced. Two repositories that happen to track
byte-identical file content get the same content-addressed CID, so
unseeding one could silently evict content another still-seeded
repository needed -- that repo's next `git lfs fetch` would then fail
with no obvious cause. Fixed by checking every other currently-seeded
repository's own LFS notes before unpinning, and skipping any CID
still referenced elsewhere. No persistent refcount is kept anywhere;
this recomputes the still-needed set from git-notes each time,
consistent with the rest of this design treating notes as the single
source of truth rather than maintaining separate bookkeeping that
could drift out of sync.
`rad lfs backfill` retroactively pins any LFS-tracked file at HEAD
that doesn't yet have a note -- the situation a `--no-verify` commit
or a commit made while `ipfs`/`rad` weren't on `PATH` leaves behind
(the pre-commit hook soft-skips pinning in the latter case rather
than blocking the commit entirely). Content is read from Git LFS's
own local object cache (populated by the clean filter on `git add`
regardless of whether the hook ran), falling back to the working-tree
file. Reports what it backfilled, what already had a note, and what
it couldn't find content for locally (meaning some other peer that
does have it needs to run this instead).
Without an explicit rerun-if-changed directive, cargo has no way to
know it needs to re-run this build script just because the current
commit changed -- by default it only reruns based on files inside
the crate's own source tree, and .git is entirely outside that. This
let GIT_HEAD/RADICLE_VERSION silently go stale: several `cargo
build`s in a row during this session kept reporting an old commit's
version despite the working tree being clean at a newer commit,
because cargo decided nothing needed rebuilding at all.
`.git/logs/HEAD` (the reflog) is touched on every commit, checkout,
and merge, unlike `.git/HEAD` itself (which only changes when
switching branches) -- watching it catches the case that actually
matters here.
Matches the banner already on the root README.md -- this file is
otherwise upstream heartwood's own GitHub-specific profile readme,
unrelated to and predating the LFS-over-IPFS work, so someone landing
on it directly (or via GitHub's own rendering convention) had no
indication this repo is a fork with different setup instructions.
`rad lfs init` previously only configured a push refspec for the
notes ref (`+refs/notes/rad-lfs/local:refs/notes/rad-lfs`) -- once
any explicit push refspec exists on a remote, git stops falling back
to push.default-driven behavior for a bare `git push rad`, so it only
ever pushed the notes mapping, never the branch. That's the exact
footgun that just caused a real "blog did not update" report: `git
push rad` (bare) after a commit only sent the notes ref, silently
leaving the commit itself unpushed.
Add a second push refspec, `+refs/heads/*:refs/heads/*`, mirroring
the existing `+refs/heads/*:refs/remotes/rad/*` fetch refspec already
configured. A bare `git push rad` now pushes every local branch and
the notes mapping together in one command.
Only takes effect once `rad lfs init` has been (re-)run with this
build -- it's an additive, idempotent config change via the existing
`ensure_refspec` helper, no migration/cleanup needed since there's no
stale conflicting entry to remove this time. Updated `rad lfs init`'s
own success/info messages plus README.md/LFS-IPFS.md/
radicle-lfs-transfer's README to match -- the "push gotcha" workaround
they documented is now the *old*-repository fallback path, not the
primary instructions.
There is no `rad push`/`rad pull` subcommand -- `rad`'s actual
subcommand set is auth/checkout/clone/.../sync/watch (see `rad
--help`); pushing/pulling git refs (branch, notes) is done via plain
`git push rad`/`git pull rad`, same as any git remote. LFS-IPFS.md's
"Use" section and `rad lfs init`'s own printed success message both
said `rad push`/`rad pull`, which doesn't exist and doesn't even
gesture at the real push gotcha (branch and notes ref need two
separate pushes).
Also updates LFS-IPFS.md's "How it works" section, which still
described the pre-batching design (a pre-commit hook calling `rad lfs
store` once per file, and the bare `refs/notes/rad-lfs` as if it were
the local write target) -- both stale since the LOCAL_NOTES_REF fix
and the `rad lfs precommit` batching added this session. `rad lfs
init`'s own success message now proactively states the push gotcha,
rather than only documenting it after the fact in Troubleshooting.
publish-package.sh now checks the Forgejo package listing first and
exits early if the current version is already published, instead of
failing with a 409 on the first upload attempt. Also generates a
changelog (git log since whichever previous package's embedded
short-sha we can find) and attaches it both inside the tarball
(MANIFEST.txt, CHANGELOG.txt) and as its own uploaded file, so anyone
browsing the package page can see what changed without downloading
the archive.
Appends a semver build-metadata suffix to RADICLE_VERSION so `rad
--version` (and radicle-node/git-remote-rad, which share this build
script via symlink) unambiguously identify a binary as this fork
rather than upstream heartwood -- useful for checking what a deploy
target is actually running.
While wiring this up, found and fixed a real pre-existing bug: none
of the three `git` subprocess outputs captured here (GIT_HEAD,
RADICLE_VERSION, SOURCE_DATE_EPOCH) were trimmed, so each carried a
trailing newline into its `cargo::rustc-env=KEY=VALUE` directive.
Harmless on its own since nothing followed it, but appending the
+lfs-ipfs suffix landed after that embedded newline, splitting the
directive across two lines -- cargo silently discards the second
line as unrecognized, so the suffix never actually took effect until
this trim was added.
Also adds scripts/publish-package.sh: builds all four release
binaries (rad, radicle-node, git-remote-rad, rad-lfs-transfer) and
publishes them as a Forgejo generic package, so a deploy target
without a Rust toolchain can download instead of rebuilding from
source (see NOTES-lfs-store-note-write-bug.md's VPS deployment story
for why that matters).
Adds a copy-pasteable day-to-day cheatsheet (init, track/commit/push,
clone-elsewhere, private-repo rekey) right after the existing "Zero to
usable" build section. Also corrects/expands the existing
git-push-rad troubleshooting row in LFS-IPFS.md based on today's live
round-trip test: `rad lfs init` only configures a push refspec for the
notes ref, not the branch, so pushing a commit needs both
`git push rad <branch>` and a separate bare `git push rad` -- missing
the second step is what produces "no CID recorded for oid" on a
collaborator's clone.
`git diff --cached --name-only` C-quotes filenames with non-ASCII
bytes by default (e.g. a literal "\305\202" for a Polish "ł"), which
doesn't match any real path on disk -- the hook's `[ -f "$file" ]`
check silently skipped those files instead of erroring, so they were
committed as valid LFS pointers but never pinned to IPFS. Use `-z`
(NUL-separated, unquoted) and convert to newlines for the read loop.
Also add an idempotency fast-path to `rad lfs store`/`store_object`:
if a note already exists for an object (e.g. written moments earlier
by `rad lfs precommit`), return its CID directly instead of
re-encrypting. This avoids redundant work, and matters more than that
for private repos specifically: `git push`'s custom-transfer-agent
subprocess chain has no TTY, so without this fast path every push
would fail outright on the passphrase prompt even for objects already
stored interactively at commit time.
Previously the pre-commit hook shelled out to `rad lfs store` once per
staged LFS file, each a separate process. For private repos, each of
those needs a keystore passphrase for an ECDH key-agreement operation,
so a multi-file commit meant a separate interactive passphrase prompt
per file -- observed live against the blog repo (9+ prompts in a row).
Add `rad lfs precommit`, which reads every staged file's oid/size/path
from stdin and processes them all in one process, loading the signer
at most once and reusing it across the batch. `store::run` now
delegates to a shared `store_object` helper that both it and
`precommit::run` call.
Found live while migrating the blog's food-gallery images into LFS,
diagnosed in NOTES-lfs-store-note-write-bug.md, confirmed and fixed here.
Root cause: store.rs/rekey.rs wrote local notes to the bare
refs/notes/rad-lfs, but the earlier fetch-bug fix (1c65ee62) has the
fetch refspec populate refs/notes/rad-lfs/<peer> siblings locally --
including your own peer's copy, fetched back after your own push. A bare
refs/notes/rad-lfs ref can't coexist with refs/notes/rad-lfs/<peer> in
the same git ref namespace (a leaf ref vs. a directory prefix at the same
path) -- so the very first commit+push+fetch cycle on a repo left it
permanently unable to `rad lfs store` again, failing with "failed to
write LFS note" and no further detail (git itself hits the same conflict
from the fetch direction and just refuses that one ref cleanly; it's
specifically libgit2's ref-write path used here that fails hard instead).
Fix: stop writing to the bare ref locally at all. New
lfs_crypto::LOCAL_NOTES_REF (refs/notes/rad-lfs/local) is what
store.rs/rekey.rs write to now -- living under the same
refs/notes/rad-lfs/ prefix as every fetched peer ref makes it just
another sibling leaf ref, structurally incapable of conflicting with
them ("local" can never collide with an actual peer ID). NOTES_REF
(bare) stays exactly as it was for the remote-side/push-destination
role, unchanged. rad lfs init's push refspec becomes asymmetric
(+refs/notes/rad-lfs/local:refs/notes/rad-lfs) instead of symmetric;
fetch refspec is untouched.
Added a migration step (migrate_local_notes_ref, run by rad lfs init)
for already-affected repos: reads any existing bare-ref notes out,
deletes the bare ref, then re-writes them under the new local ref --
also removes the old symmetric push refspec config entry the same way
the earlier fetch-refspec migration did, so re-running rad lfs init
fully repairs an already-broken repo with no manual git surgery needed.
(First version of the migration itself had the identical bug -- tried to
write the new ref before deleting the old one, hit the same conflict
self-inflicted -- fixed by reading all notes into memory before deleting
the bare ref.)
Verified locally: reproduced the exact failure in the precise order that
triggers it, confirmed the fix resolves that exact same failing call in
the same repo, verified the migration path preserves note content
exactly, and confirmed a full commit->push->fetch->decrypt round trip
plus `rad lfs rekey` all still work correctly afterward.
Bug #2 (ls-remote vs fetch disagreeing on the VPS) turned out to be two
separate things, neither a bug in the fetch fix itself: a stale,
unrelated system-wide git-remote-rad binary being silently used in
ad-hoc manual testing (PATH had no ~/.radicle/bin), plus a genuine
cross-device rename bug in rad-lfs-transfer's download path (fixed
separately, ac4a27c) that only surfaced once the real pipeline actually
ran with the fetch fix for the first time. Verified live: triggered
blog-deploy.service for real, full success end-to-end, SHA-256-verified.
Private-repo LFS operations need a signer capable of ECDH, which
ssh-agent (the normal day-to-day flow after `rad auth`) can't provide --
that's a hard protocol limitation (standard ssh-agent only implements
signing, there's no key-agreement request type), not a gap in our code,
so there's no way to make it work "through the agent" the way signing
does.
Previously the only alternative to an unencrypted keystore was setting
RAD_PASSPHRASE, which breaks the normal ssh-agent-based workflow and
means putting a password in an environment variable -- not ergonomic,
and not how any other `rad` command behaves.
`rad` already has an established fallback for exactly this situation:
`crate::terminal::io::signer` tries ssh-agent first, and if that's
unavailable/unregistered and we're connected to a TTY, prompts
interactively for the passphrase via `inquire` (reusing the existing
`PassphraseValidator`). `lfs_crypto::load_signer` was missing this last
step entirely -- it only tried "unencrypted keystore" then
"RAD_PASSPHRASE" then gave up. Added the same interactive-prompt
fallback, reusing the exact same validator/prompt helpers already used
elsewhere, before falling back to a hard failure.
This covers the common case (an interactive `git commit` triggering the
pre-commit hook, which inherits the terminal's TTY) with no environment
variable needed at all. Only genuinely non-interactive contexts (CI, a
script with no TTY) still need RAD_PASSPHRASE -- and get a fast, clear
failure rather than hanging, since `inquire` returns `Ok(None)` on
`NotTTY` rather than blocking (verified: piping /dev/null as stdin still
fails immediately, not a timeout).
Verified against a real encrypted keystore through an actual pty (via
`script`, since a plain non-interactive shell doesn't have a TTY to
prompt on): the masked "Passphrase:" prompt appears, accepts the typed
passphrase, and produces a correctly encrypted note -- with
RAD_PASSPHRASE unset the entire time.
Found in a real deployment (mbator.pl blog pipeline): rad lfs init
configures a client-side fetch refspec for refs/notes/rad-lfs, but
git-remote-rad's for_fetch() never advertised any such ref -- notes only
exist per-peer, under refs/namespaces/<peer>/refs/notes/rad-lfs, with no
canonicalization step the way refs/heads/refs/tags get one. Requesting an
unadvertised refspec fails the *entire* git fetch, not just that one ref,
so any repo with `rad lfs init` run could no longer git fetch/pull at
all, LFS or not.
Fix: expose each peer's refs/notes/rad-lfs individually, as
refs/notes/rad-lfs/<peer-id> (list.rs's for_fetch), matching how patch
refs already work rather than trying to canonicalize -- notes are
per-peer contributions (any peer can commit an LFS-tracked file, not
just delegates), so unlike heads there's no single "correct" value to
resolve to; canonicalizing would silently drop non-delegate contributors'
objects. rad lfs init's fetch refspec becomes a wildcard
(+refs/notes/rad-lfs/*:refs/notes/rad-lfs/*); push is unchanged (already
worked -- for_push() turns out to be purely informational for git's
list-for-push handshake, not an enforcement gate, so the actual send-pack
call was never restricted to heads/tags despite what for_push() lists).
rad lfs store/fetch/rekey and seed/unseed's pinning (ipfs.rs::lfs_cids)
now read across every notes ref instead of assuming one canonical ref,
merging recipient lists for notes that share a CID (e.g. after a rekey
performed by a different peer than the original committer) while keeping
genuinely different ciphertexts (from two peers independently encrypting
byte-identical content) as separate candidates to try in turn.
Also added a migration step so re-running `rad lfs init` on an
already-configured repo actually fixes it, by removing the old
non-wildcard fetch refspec rather than just adding the wildcard
alongside a still-broken entry.
Verified through the real git-remote-rad transport (not the storage-copy
shortcut used for earlier testing, since this bug lives specifically in
that transport): fresh identities, a real rad:// remote matching exactly
what `rad init` configures, a genuine `git fetch rad` that used to fail
with "fatal: couldn't find remote ref refs/notes/rad-lfs" now succeeds
and pulls the notes ref, and `git lfs pull` afterward correctly decrypts
private content end-to-end (SHA-256 verified). Re-verified the rekey
flow through the same real path too.
Cargo's workspace auto-detection was pulling the submodule into the
parent workspace since it's nested under the repo root, breaking
`cargo install --path radicle-lfs-transfer` (the exact command in the
README/LFS-IPFS.md setup instructions) with "current package believes
it's in a workspace when it's not". Discovered while installing the
built binaries onto a real machine.
Radicle's Visibility::Private is enforced entirely by access control at
the replication layer -- git objects for a private repo are plain,
unencrypted objects, protected only by "unauthorized peers are never
given them" (identity/doc.rs). That protection has no jurisdiction over
IPFS's own block-exchange layer: once a legitimate collaborator's IPFS
daemon pins content and joins the network, anyone who obtains the CID by
any means can fetch the plaintext directly.
Adds envelope encryption for private repos: a random per-object content
key encrypts the file (XChaCha20-Poly1305), wrapped once per authorized
recipient (delegates + private allow-list) via ECDH between Radicle
identity keys (already Ed25519, no separate keypair needed) plus HKDF.
The wrapped keys travel in the same refs/notes/rad-lfs note as the CID.
New:
- crates/radicle-cli/src/lfs_crypto.rs -- encryption core (envelope
schema, encrypt/decrypt, key wrapping, recipient resolution, signer
loading).
- `rad lfs store`/`rad lfs fetch` -- plumbing commands that now own the
entire IPFS add/cat + git-notes read/write step (encrypting/decrypting
as needed), replacing what the pre-commit hook and rad-lfs-transfer
used to do directly. This collapses what would've become a third copy
of the note-format logic down to one canonical implementation.
- `rad lfs rekey` -- since Radicle's access control is retroactive (a
newly-authorized DID can replicate full history) but encrypted objects
aren't automatically re-wrapped for new recipients, this command lets
an already-authorized collaborator grant a newly-authorized one access
to previously-committed objects, without re-encrypting or re-uploading
content. Revocation remains unsolved, matching Radicle's own model
(already-replicated/decrypted content can't be un-known).
Operational requirement: private-repo LFS operations need a signer
capable of ECDH, which only an unencrypted keystore or RAD_PASSPHRASE
provides (an ssh-agent-only signer can sign but not do key agreement).
Fails fast with an actionable error rather than hanging on a passphrase
prompt inside a non-interactive git hook.
Breaking change: the refs/notes/rad-lfs note format moves from bare
`cid=<cid>` text to JSON (`{"v":1,"cid":"...","enc":...}`). No migration
path -- prototype, no external users yet.
Verified end-to-end with real Radicle profiles/identities (not just the
transfer-agent protocol): a private repo's IPFS-stored content is
confirmed ciphertext (byte-different from plaintext, +16 bytes AEAD tag);
an authorized peer decrypts correctly; an unauthorized peer is denied
with a clear error and no leaked content; RAD_PASSPHRASE is required and
enforced with a fast, clear failure; and rekey correctly grants a
newly-authorized peer access to a pre-existing object while leaving other
recipients' entries untouched.
Cross-repo markdown links (radicle-lfs-transfer) now use full https URLs
instead of relative paths, since Forgejo can't resolve ../other-repo across
repositories the way a plain filesystem or the git submodule mechanism can.
Adds Git LFS support for repositories, with large file content stored on
each contributor's own local IPFS node instead of a seed-hosted server:
- `rad lfs init`: one-time setup (transfer-agent config, pre-commit hook,
notes-ref push/fetch refspecs, IPFS daemon reachability check).
- `refs/notes/rad-lfs` carries the oid -> CID mapping, replicating with
the repository via Radicle's existing ref sync (verified against
`references_of` in crates/radicle/src/storage/git.rs: every ref is
replicated except refs/tmp/heads/*, so no protocol changes were needed).
- `rad seed`/`rad unseed` now pin/unpin a repo's known LFS objects in IPFS,
mirroring Radicle's existing "seeding = keep a full copy" model, and
warning (not failing) if no local IPFS daemon is reachable.
The actual byte transfer is handled by a separate binary, rad-lfs-transfer
(added here as a submodule), which implements the Git LFS custom-transfer-
agent protocol against a local IPFS daemon. See LFS-IPFS.md for the full
design and setup instructions.
Building and using `rad` for anything other than `rad lfs` requires no
IPFS dependency at all; `rad lfs init` checks for a local daemon upfront
and fails with an actionable message rather than proceeding silently.
Small improvements:
- Remove confusing comments in test
- Rename `has_active_sibling_accept`
to `has_accepted_active_sibling`
- Small refactoring to avoid `… } else { None }`
Fetching a large repository looped with
Failed to consume the pack sent by the remote
even though the pack appears valid (Git indexes it in full).
The cause appears to be a bug in zlib-rs 0.6.3 according to
<https://github.com/trifectatechfoundation/zlib-rs/pull/504>.
When a delegate creates a new `Revision`, their signature is implicitly added to the revision.
Prevent the delegate from an accept verdict on two active, sibling revisions by:
1. Checking that the `state` is `Active`, and then checking `has_active_sibling_accept`
2. If there is an existing verdict, then it the verdict from the new revision is stripped.
The creation of the `Revision` must still happen, as opposed to throwing an
error, in case of existing histories.
On `Action::RevisionAccept` use `has_active_sibling_accept` to check for a sibling accept.
If one is found, the `RevisionAccept` is logged and skipped.
This preserves the invariant of a single accept per delegate on active, sibling
revisions without throwing an error on existing histories.
Verifies that when Alice redacts a revision and Bob concurrently
accepts it, both nodes converge after sync. The CRDT evaluation
order (timestamp-based) determines which operation takes effect.
The test previously had B2 as a sibling of B1 (both children of A1),
because update() always uses self.current as parent. This meant
the test was testing multi-sibling rejection, not cascading.
Now B2 is a child of B1 via a manual transaction, so when B1 is
rejected (sibling E1 accepted), B2 is rejected via RejectedBy::Parent
(cascade), which is what the test name promises.
Introduce a property testing harness for asserting a set of properties expected
of the repository identity.
The harness uses the `Network` fixture for providing four nodes that can
interact with the repository identity. The state of the harness is advanced by
providing an actor and operation on the identity document. Invariants are then
asserted about the identity.
The current invariants are:
- The `current` revision is the one and only `Accepted` revision.
- The chain of revisions is valid. That is, `Active` revisions only have a
parent that is `Accepted`, and not `Rejected` or `Redacted`.
- `Active` revisions do not contain a majority approval.
- `Accepted` revisions contain a majority approval.
- `Rejected` revisions do not contain a majority approval.
- For each revision, at most one child is `Accepted`.
- For each revision, if any of its children is `Accepted`, all other
children are `Rejected`.
- A revision which is `Rejected(RejectedBy::Parent)` has a parent that
is `Rejected`.
- A revision which is `Redacted(RedactedBy::Parent)` has a parent that
is `Redacted`.
- For each revision that is `Rejected` or `Redacted`, none of its
children is `Active`.
- A sibling or ancestor revision's rejected state applies to its sibling or
descendant.
- The repository identity documents converge when all nodes have applied all
operations.
A `BoundedVec` that implements `Arbitrary` is useful for all property testing.
Move it from the `sigrefs` module so that it can be reused by other components.
Maciek "mab122" Bator
Lorenz Leutgeb
Rūdolfs Ošiņš
Fintan Halpenny
Adrian Duke