lfs: add fetch-batch, a long-lived worker for multi-object downloads

`rad lfs fetch` still exists for single-object/manual use, but
`radicle-lfs-transfer` used to spawn a fresh one per object during
`git lfs pull`/checkout -- for a private repo, each one independently
loaded the signer, so N files meant N passphrase prompts. Same bug
class `rad lfs precommit` already fixed for commits, unfixed here
until now.

Unlike precommit (which knows every staged file upfront and can read
stdin until EOF), fetch can't collect a batch before starting:
git-lfs's custom-transfer protocol requests objects one at a time.
`rad lfs fetch-batch` instead stays alive for as long as its caller
keeps its stdin open, answering one "<oid> <size> <out-path>" request
at a time with a JSON response line, reusing the same unwrapped
signer across all of them -- paired with radicle-lfs-transfer's new
FetchWorker, which spawns this once per transfer session instead of
once per object.

Also factors `fetch::run`'s core logic out into `fetch_object`,
taking a `&mut Option<MemorySigner>` the same way `store::store_object`
already does, so both the single-shot `rad lfs fetch` and the new
batch worker share one implementation.

LFS-IPFS.md

@@ -53,9 +53,16 @@ file-CID mapping travels with the repository itself, avoiding a single point of
   the actual work, which is a no-op re-upload if the object already has a note (e.g. one your
   own pre-commit hook already wrote).
 - **Seeding**: `rad seed` now also pins a repository's known LFS objects in your local IPFS
-  node (mirroring "seeding = keep a full copy"); `rad unseed` unpins them. Known prototype
-  limitation: no cross-repository pin refcounting, so unseeding one repository can unpin
-  content still needed by another seeded repository that happens to share the same CID.
+  node (mirroring "seeding = keep a full copy"); `rad unseed` unpins them, except any CID also
+  referenced by another currently-seeded repository's own LFS notes (e.g. two repositories that
+  happen to track byte-identical content, which hashes to the same CID) -- unpinning those would
+  silently break the other repository. There's no persistent pin refcount kept anywhere; this is
+  recomputed from every other seeded repository's notes each time `rad unseed` runs.
+- **Recovering from a skipped pin**: if the pre-commit hook didn't run (`git commit --no-verify`)
+  or soft-skipped because `ipfs`/`rad` weren't on `PATH` yet, a file can end up committed as a
+  valid LFS pointer with no corresponding note -- nothing pins it anywhere. Run `rad lfs
+  backfill` to retroactively pin every LFS-tracked file at HEAD that's missing one, using
+  content from Git LFS's own local object cache (or the working-tree file, as a fallback).
 
 ## Encryption for private repositories
 
@@ -236,7 +243,8 @@ previously-committed LFS objects too, not just ones committed after they were ad
 | `Git LFS is not installed (the 'git-lfs' binary was not found on your PATH)` | Install [Git LFS](https://git-lfs.com). |
 | `no IPFS (Kubo) daemon reachable at ...` | Start one: `ipfs daemon` (see [Run](#run) above). |
 | `rad-lfs-transfer: command not found` during `git lfs push`/`pull` | `cargo install --path radicle-lfs-transfer` didn't complete, or `~/.radicle/bin` isn't on `PATH`. |
-| Large file didn't get pinned (commit went through, but no `rad-lfs` note) | The pre-commit hook silently skips pinning if the `ipfs`/`rad` CLIs aren't on `PATH` — check `command -v ipfs` / `command -v rad`. Re-add the file and commit again once available. |
+| Large file didn't get pinned (commit went through, but no `rad-lfs` note) | The pre-commit hook silently skips pinning if the `ipfs`/`rad` CLIs aren't on `PATH` — check `command -v ipfs` / `command -v rad`. Once available, run `rad lfs backfill` to pin it retroactively rather than needing to re-commit. |
+| Committed with `git commit --no-verify` (or the pre-commit hook was skipped some other way) and now nothing's pinned | Run `rad lfs backfill` — it walks every LFS-tracked file at HEAD and pins whichever ones are missing a note, without needing a new commit. |
 | `private-repo LFS operations need to perform a key-agreement (ECDH) operation ...` | Your keystore is encrypted, ssh-agent can't do key agreement (protocol limitation), and no passphrase is available — normally you'd just be prompted interactively; this only happens in a non-interactive context (no TTY, e.g. CI). Set `RAD_PASSPHRASE`, or use an unencrypted keystore, or run it from a terminal. |
 | `you are not an authorized recipient of this object` | Either genuinely unauthorized (not a delegate/allow-listed DID for this private repo), or authorized *after* this specific object was committed and nobody has run `rad lfs rekey` since — ask an already-authorized collaborator to run it and push. |
 | `fatal: couldn't find remote ref refs/notes/rad-lfs` on `git fetch`/`git pull` (breaks *all* fetching, not just LFS) | Fixed — but if this repo had `rad lfs init` run against an older build, re-run `rad lfs init` once to replace the stale, now-invalid fetch refspec it configured (see `NOTES-lfs-notes-fetch-bug.md` for the full story). |

README.md

@@ -48,10 +48,12 @@
 > ```
 >
 > From here, use `rad` exactly as upstream describes below (`rad auth`, `rad init`, etc). The
-> only new commands are `rad lfs init` (run once per repository you want large-file support in)
-> and `rad lfs rekey` (run after granting a new collaborator access to a **private** repository,
-> so they can decrypt previously-committed LFS objects too) — see [`LFS-IPFS.md`](LFS-IPFS.md)
-> for that workflow and how private-repo content gets encrypted before it reaches IPFS.
+> only new commands are `rad lfs init` (run once per repository you want large-file support in),
+> `rad lfs rekey` (run after granting a new collaborator access to a **private** repository, so
+> they can decrypt previously-committed LFS objects too), and `rad lfs backfill` (retroactively
+> pins any LFS-tracked file that got committed without going through the pre-commit hook, e.g.
+> `--no-verify`) — see [`LFS-IPFS.md`](LFS-IPFS.md) for those workflows and how private-repo
+> content gets encrypted before it reaches IPFS.
 > **Nothing above requires IPFS**; Git LFS support specifically needs a running `ipfs daemon`,
 > and `rad lfs init` will tell you plainly if one isn't reachable rather than failing confusingly
 > later.

crates/radicle-cli/src/commands/lfs.rs

@@ -1,6 +1,8 @@
 //! `rad lfs` command implementation.
 
+pub mod backfill;
 pub mod fetch;
+pub mod fetch_batch;
 pub mod init;
 pub mod precommit;
 pub mod rekey;
@@ -20,5 +22,7 @@ pub fn run(args: Args, ctx: impl term::Context) -> anyhow::Result<()> {
         Command::Fetch { oid, size, out } => self::fetch::run(oid, size, out, ctx),
         Command::Rekey => self::rekey::run(ctx),
         Command::Precommit => self::precommit::run(ctx),
+        Command::Backfill => self::backfill::run(ctx),
+        Command::FetchBatch => self::fetch_batch::run(ctx),
     }
 }

crates/radicle-cli/src/commands/lfs/args.rs

@@ -51,4 +51,14 @@ pub enum Command {
     /// "<oid> <size> <path>" lines from stdin.
     #[command(hide = true)]
     Precommit,
+    /// Retroactively pin any LFS-tracked file at HEAD that was committed
+    /// without going through the pre-commit hook (e.g. `--no-verify`, or
+    /// the IPFS daemon/`ipfs`/`rad` weren't available at commit time)
+    Backfill,
+    /// Long-lived worker that fetches many LFS objects across one process
+    /// (invoked by the custom transfer agent; not for interactive use).
+    /// Reads "<oid> <size> <out-path>" lines from stdin, one at a time,
+    /// writing a JSON response line for each.
+    #[command(hide = true)]
+    FetchBatch,
 }

crates/radicle-cli/src/commands/lfs/backfill.rs

@@ -0,0 +1,171 @@
+//! `rad lfs backfill` — retroactively stores any already-committed
+//! LFS-tracked file that doesn't yet have a `refs/notes/rad-lfs` note (so
+//! it was never pinned to IPFS). This happens whenever the pre-commit hook
+//! didn't run or didn't have what it needed at commit time: a commit made
+//! with `--no-verify`, or one made while the `ipfs`/`rad` binaries weren't
+//! on `PATH` yet (the hook soft-skips pinning in that case rather than
+//! blocking the commit -- see `HOOK_BODY`'s `command -v` checks).
+//!
+//! Content actually needs to be available locally to backfill it from:
+//! this reads from Git LFS's own local object cache
+//! (`.git/lfs/objects/<oid prefix>/<oid>`, populated by the `git-lfs`
+//! clean filter on `git add`, which -- unlike the pre-commit *hook* --
+//! isn't skipped by `--no-verify`), falling back to the working-tree file
+//! if that's missing. A file whose content isn't available through either
+//! path can't be backfilled from this machine; some other peer that
+//! already has it needs to run this instead.
+
+use std::path::{Path, PathBuf};
+
+use anyhow::Context as _;
+
+use radicle::storage::{ReadRepository as _, ReadStorage as _};
+
+use crate::ipfs;
+use crate::terminal as term;
+
+use super::store::store_object;
+
+pub fn run(ctx: impl term::Context) -> anyhow::Result<()> {
+    let profile = ctx.profile()?;
+    let (repo, rid) = radicle::rad::cwd()
+        .context("`rad lfs backfill` must be run inside a Radicle repository working copy")?;
+    let workdir = repo
+        .workdir()
+        .ok_or_else(|| anyhow::anyhow!("`rad lfs backfill` needs a git working copy, not a bare repository"))?;
+    let doc = profile.storage.repository(rid)?.identity_doc()?.doc;
+
+    ipfs::check_daemon()?;
+
+    // Every path git currently tracks at HEAD (not the working tree or
+    // index, which may have unrelated local modifications) -- mirrors
+    // HOOK_BODY's own approach of shelling out to `git` for this rather
+    // than reimplementing tree/attribute logic against git2 directly.
+    let ls_files = radicle::git::run(Some(workdir), ["ls-tree", "-r", "-z", "--name-only", "HEAD"])
+        .context("failed to list files at HEAD")?;
+    let paths: Vec<String> = String::from_utf8_lossy(&ls_files.stdout)
+        .split('\0')
+        .filter(|s| !s.is_empty())
+        .map(str::to_string)
+        .collect();
+
+    let mut candidates = Vec::new();
+    for path in paths {
+        // Matches HOOK_BODY's own check: only files the *current*
+        // `.gitattributes` marks as `filter=lfs` are our concern.
+        let is_lfs = radicle::git::run(Some(workdir), ["check-attr", "filter", "--", &path])
+            .ok()
+            .map(|out| String::from_utf8_lossy(&out.stdout).contains(": filter: lfs"))
+            .unwrap_or(false);
+        if !is_lfs {
+            continue;
+        }
+        let blob_oid_out = radicle::git::run(Some(workdir), ["rev-parse", &format!("HEAD:{path}")])
+            .with_context(|| format!("failed to resolve blob oid for `{path}`"))?;
+        let blob_oid_str = String::from_utf8_lossy(&blob_oid_out.stdout).trim().to_string();
+        let Ok(blob_oid) = radicle::git::raw::Oid::from_str(&blob_oid_str) else {
+            continue;
+        };
+        candidates.push((path, blob_oid));
+    }
+
+    if candidates.is_empty() {
+        term::success!("No LFS-tracked files found in HEAD; nothing to backfill");
+        return Ok(());
+    }
+
+    let mut signer = None;
+    let mut backfilled = 0;
+    let mut already_pinned = 0;
+    let mut unavailable = Vec::new();
+
+    for (path, blob_oid) in candidates {
+        if !crate::lfs_crypto::find_cids(&repo, blob_oid)?.is_empty() {
+            already_pinned += 1;
+            continue;
+        }
+
+        let blob = repo
+            .find_blob(blob_oid)
+            .with_context(|| format!("failed to read blob for `{path}`"))?;
+        let Ok(pointer_text) = std::str::from_utf8(blob.content()) else {
+            continue;
+        };
+        let Some(oid) = pointer_text
+            .lines()
+            .find_map(|line| line.strip_prefix("oid sha256:"))
+        else {
+            continue;
+        };
+        let Some(size) = pointer_text
+            .lines()
+            .find_map(|line| line.strip_prefix("size "))
+            .and_then(|s| s.parse::<i64>().ok())
+        else {
+            continue;
+        };
+
+        let Some(content_path) = locate_content(workdir, oid, size, &path) else {
+            unavailable.push(path);
+            continue;
+        };
+
+        store_object(&profile, &repo, rid, &doc, oid, size, &content_path, &mut signer)
+            .with_context(|| format!("failed to backfill `{path}`"))?;
+        term::success!("Backfilled `{path}`");
+        backfilled += 1;
+    }
+
+    term::blank();
+    if backfilled > 0 {
+        term::success!("Backfilled {backfilled} object(s)");
+    }
+    if already_pinned > 0 {
+        term::info!("{already_pinned} object(s) already had a note; left untouched");
+    }
+    if !unavailable.is_empty() {
+        term::warning(format!(
+            "{} object(s) have no note and no content available locally to backfill from -- \
+             ask a peer that already has them to run `rad lfs backfill`:",
+            unavailable.len()
+        ));
+        for path in &unavailable {
+            term::warning(format!("  {path}"));
+        }
+    }
+
+    Ok(())
+}
+
+/// Git LFS's own local object cache is the reliable source: the
+/// `git-lfs` clean filter populates it on `git add` regardless of
+/// whether the pre-commit *hook* ran (`--no-verify` skips hooks, not
+/// filters). Falls back to the working-tree file, in case content is
+/// present there but the LFS cache was pruned -- rejected if the size on
+/// disk doesn't match what the pointer recorded, since that means it's
+/// just the unsmudged pointer text, not the real content.
+fn locate_content(workdir: &Path, oid: &str, size: i64, path: &str) -> Option<PathBuf> {
+    if oid.len() >= 4 {
+        let cached = workdir
+            .join(".git/lfs/objects")
+            .join(&oid[0..2])
+            .join(&oid[2..4])
+            .join(oid);
+        if matches_size(&cached, size) {
+            return Some(cached);
+        }
+    }
+
+    let working_copy = workdir.join(path);
+    if matches_size(&working_copy, size) {
+        return Some(working_copy);
+    }
+
+    None
+}
+
+fn matches_size(path: &Path, expected: i64) -> bool {
+    std::fs::metadata(path)
+        .map(|meta| meta.len() == expected as u64)
+        .unwrap_or(false)
+}

crates/radicle-cli/src/commands/lfs/fetch.rs

@@ -3,10 +3,15 @@
 //! content from the local IPFS (Kubo) node, decrypting it first if the
 //! repository is private.
 
-use std::path::PathBuf;
+use std::path::{Path, PathBuf};
 
 use anyhow::{Context as _, anyhow};
 
+use radicle::Profile;
+use radicle::git::raw::Repository;
+use radicle::identity::RepoId;
+use radicle_crypto::ssh::keystore::MemorySigner;
+
 use crate::ipfs;
 use crate::lfs_crypto;
 use crate::terminal as term;
@@ -16,13 +21,32 @@ pub fn run(oid: String, size: i64, out: PathBuf, ctx: impl term::Context) -> any
     let (repo, rid) = radicle::rad::cwd()
         .context("`rad lfs fetch` must be run inside a Radicle repository working copy")?;
 
+    let mut signer = None;
+    fetch_object(&profile, &repo, rid, &oid, size, &out, &mut signer)
+}
+
+/// Fetches one LFS object's content from IPFS (decrypting it first for
+/// private repos) and writes it to `out`. Factored out of [`run`] so
+/// `rad lfs fetch-batch` can process many objects across a single
+/// process, loading (and prompting for) the signer at most once across
+/// however many objects it ends up handling, instead of once per object.
+#[allow(clippy::too_many_arguments)]
+pub fn fetch_object(
+    profile: &Profile,
+    repo: &Repository,
+    rid: RepoId,
+    oid: &str,
+    size: i64,
+    out: &Path,
+    signer: &mut Option<MemorySigner>,
+) -> anyhow::Result<()> {
     let pointer_text =
         format!("version https://git-lfs.github.com/spec/v1\noid sha256:{oid}\nsize {size}\n");
     let blob_oid = repo
         .blob(pointer_text.as_bytes())
         .context("failed to write LFS pointer blob")?;
 
-    let candidates = lfs_crypto::find_envelopes(&repo, blob_oid)
+    let candidates = lfs_crypto::find_envelopes(repo, blob_oid)
         .with_context(|| format!("failed to read LFS notes for oid {oid}"))?;
     if candidates.is_empty() {
         anyhow::bail!(
@@ -43,13 +67,16 @@ pub fn run(oid: String, size: i64, out: PathBuf, ctx: impl term::Context) -> any
         let result = match &envelope.enc {
             None => ipfs::cat(&envelope.cid),
             Some(enc) => ipfs::cat(&envelope.cid).and_then(|ciphertext| {
-                let signer = lfs_crypto::load_signer(&profile)?;
-                lfs_crypto::decrypt_with(&ciphertext, enc, &signer, profile.did(), &rid, &oid)
+                if signer.is_none() {
+                    *signer = Some(lfs_crypto::load_signer(profile)?);
+                }
+                let signer = signer.as_ref().expect("just populated above");
+                lfs_crypto::decrypt_with(&ciphertext, enc, signer, profile.did(), &rid, oid)
             }),
         };
         match result {
             Ok(bytes) => {
-                std::fs::write(&out, bytes)
+                std::fs::write(out, bytes)
                     .with_context(|| format!("failed to write `{}`", out.display()))?;
                 return Ok(());
             }

crates/radicle-cli/src/commands/lfs/fetch_batch.rs

@@ -0,0 +1,67 @@
+//! `rad lfs fetch-batch` — long-lived plumbing worker invoked by the Git
+//! LFS custom transfer agent (`rad-lfs-transfer`) to download multiple LFS
+//! objects across a single process, kept alive for the lifetime of one
+//! `git lfs pull`/checkout session -- specifically so a private repo's
+//! passphrase is only prompted for once per session, not once per file.
+//!
+//! Unlike `rad lfs precommit`/`backfill` (which know every file upfront
+//! and can just read stdin until EOF), this can't collect a batch before
+//! starting: git-lfs's custom-transfer protocol requests objects one at a
+//! time by default (`rad-lfs-transfer` doesn't negotiate the "concurrent"
+//! capability), so the caller only knows about the next object once the
+//! previous one has been answered. This process instead stays alive and
+//! answers requests one at a time as they arrive, reusing the same
+//! unwrapped signer across all of them.
+//!
+//! Protocol (line-delimited, UTF-8, on stdin/stdout):
+//!   request:  "<oid> <size> <out-path>\n"
+//!   response: a JSON object, one line per request, in the same order:
+//!             `{"ok":true}` or `{"ok":false,"error":"<message>"}`
+//! The process exits cleanly on stdin EOF (the parent closing its write
+//! end signals the transfer session is over).
+
+use std::io::{BufRead as _, Write as _};
+use std::path::PathBuf;
+
+use anyhow::Context as _;
+use serde_json::json;
+
+use crate::terminal as term;
+
+use super::fetch::fetch_object;
+
+pub fn run(ctx: impl term::Context) -> anyhow::Result<()> {
+    let profile = ctx.profile()?;
+    let (repo, rid) = radicle::rad::cwd()
+        .context("`rad lfs fetch-batch` must be run inside a Radicle repository working copy")?;
+
+    let mut signer = None;
+    let stdin = std::io::stdin();
+    let mut stdout = std::io::stdout();
+
+    for line in stdin.lock().lines() {
+        let line = line.context("failed to read stdin")?;
+        // Each line is "<oid> <size> <out-path>" -- oid is a fixed-length
+        // hex string and size is decimal, so splitting on the first two
+        // spaces leaves the path (which may itself contain spaces) intact
+        // as the remainder. Mirrors `rad lfs precommit`'s input format.
+        let mut parts = line.splitn(3, ' ');
+        let (Some(oid), Some(size), Some(out)) = (parts.next(), parts.next(), parts.next())
+        else {
+            anyhow::bail!("malformed `rad lfs fetch-batch` input line: {line:?}");
+        };
+        let Ok(size) = size.parse::<i64>() else {
+            anyhow::bail!("invalid size in `rad lfs fetch-batch` input: {size:?}");
+        };
+        let out = PathBuf::from(out);
+
+        let response = match fetch_object(&profile, &repo, rid, oid, size, &out, &mut signer) {
+            Ok(()) => json!({"ok": true}),
+            Err(err) => json!({"ok": false, "error": format!("{err:#}")}),
+        };
+        writeln!(stdout, "{response}").context("failed to write response")?;
+        stdout.flush().context("failed to flush stdout")?;
+    }
+
+    Ok(())
+}

crates/radicle-cli/src/commands/unseed.rs

@@ -1,5 +1,8 @@
 pub mod args;
 
+use std::collections::BTreeSet;
+
+use radicle::node::policy;
 use radicle::{Node, prelude::*};
 
 use crate::terminal as term;
@@ -31,10 +34,57 @@ pub fn delete(rid: RepoId, node: &mut Node, profile: &Profile) -> anyhow::Result
     }
 
     // Mirror the seed/unseed lifecycle for IPFS pins of this repository's
-    // Git-LFS objects.
+    // Git-LFS objects -- except any CID also referenced by an LFS note in
+    // some *other* still-seeded repository (e.g. two repositories that
+    // happen to both track a byte-identical file, giving it the same
+    // content-addressed CID). Unpinning unconditionally would evict
+    // content the other repository still needs, breaking it silently the
+    // next time someone there runs `git lfs fetch`. There's no
+    // cross-repository pin refcount kept anywhere -- this recomputes the
+    // still-needed set from every other seeded repository's own notes each
+    // time, which costs a pass over every other seeded repository but
+    // requires no extra persistent state, consistent with this design's
+    // git-notes-as-source-of-truth approach everywhere else.
     if !cids.is_empty() {
-        crate::ipfs::unpin_all(&cids);
+        let still_needed = cids_still_needed_elsewhere(profile, rid)?;
+        let skipped = cids.iter().filter(|cid| still_needed.contains(*cid)).count();
+        let safe_to_unpin: Vec<String> =
+            cids.into_iter().filter(|cid| !still_needed.contains(cid)).collect();
+
+        if !safe_to_unpin.is_empty() {
+            crate::ipfs::unpin_all(&safe_to_unpin);
+        }
+        if skipped > 0 {
+            term::info!(
+                "Kept {skipped} IPFS object(s) pinned -- still referenced by another seeded repository"
+            );
+        }
     }
 
     Ok(())
 }
+
+/// The set of CIDs (among `excluding`'s own) that are also referenced by an
+/// LFS note in some other repository this node is currently seeding.
+fn cids_still_needed_elsewhere(
+    profile: &Profile,
+    excluding: RepoId,
+) -> anyhow::Result<BTreeSet<String>> {
+    let mut needed = BTreeSet::new();
+    let store = profile.policies()?;
+
+    for entry in store.seed_policies()? {
+        let policy::SeedPolicy { rid, policy } = entry?;
+        if rid == excluding {
+            continue;
+        }
+        if !matches!(policy, policy::SeedingPolicy::Allow { .. }) {
+            continue;
+        }
+        if let Ok(repo) = profile.storage.repository(rid) {
+            needed.extend(crate::ipfs::lfs_cids(&repo.backend));
+        }
+    }
+
+    Ok(needed)
+}