mab122
/
radicle-heartwood-lfs
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Compare commits

base: mab122:7c6d342ef91147e90ee28d7cc75d84f253af6622
Branches Tags
mab122:rad-lfs-ipfs
mab122:master
..
compare: mab122:581843fa923b2e9c29a88a02da87f77d63e45c0b
Branches Tags
mab122:rad-lfs-ipfs
mab122:master

No commits in common. "7c6d342ef91147e90ee28d7cc75d84f253af6622" and "581843fa923b2e9c29a88a02da87f77d63e45c0b" have entirely different histories.
7c6d342ef9 ... 581843fa92

8 changed files with 15 additions and 354 deletions
Show Stats Expand all files Collapse all files

16
LFS-IPFS.md
View File

@ -53,16 +53,9 @@ file-CID mapping travels with the repository itself, avoiding a single point of
the actual work, which is a no-op re-upload if the object already has a note (e.g. one your
own pre-commit hook already wrote).
- **Seeding**: `rad seed` now also pins a repository's known LFS objects in your local IPFS
node (mirroring "seeding = keep a full copy"); `rad unseed` unpins them, except any CID also
referenced by another currently-seeded repository's own LFS notes (e.g. two repositories that
happen to track byte-identical content, which hashes to the same CID) -- unpinning those would
silently break the other repository. There's no persistent pin refcount kept anywhere; this is
recomputed from every other seeded repository's notes each time `rad unseed` runs.
- **Recovering from a skipped pin**: if the pre-commit hook didn't run (`git commit --no-verify`)
or soft-skipped because `ipfs`/`rad` weren't on `PATH` yet, a file can end up committed as a
valid LFS pointer with no corresponding note -- nothing pins it anywhere. Run `rad lfs
backfill` to retroactively pin every LFS-tracked file at HEAD that's missing one, using
content from Git LFS's own local object cache (or the working-tree file, as a fallback).
node (mirroring "seeding = keep a full copy"); `rad unseed` unpins them. Known prototype
limitation: no cross-repository pin refcounting, so unseeding one repository can unpin
content still needed by another seeded repository that happens to share the same CID.
## Encryption for private repositories
@ -243,8 +236,7 @@ previously-committed LFS objects too, not just ones committed after they were ad
| `Git LFS is not installed (the 'git-lfs' binary was not found on your PATH)` | Install [Git LFS](https://git-lfs.com). |
| `no IPFS (Kubo) daemon reachable at ...` | Start one: `ipfs daemon` (see [Run](#run) above). |
| `rad-lfs-transfer: command not found` during `git lfs push`/`pull` | `cargo install --path radicle-lfs-transfer` didn't complete, or `~/.radicle/bin` isn't on `PATH`. |
| Large file didn't get pinned (commit went through, but no `rad-lfs` note) | The pre-commit hook silently skips pinning if the `ipfs`/`rad` CLIs aren't on `PATH` — check `command -v ipfs` / `command -v rad`. Once available, run `rad lfs backfill` to pin it retroactively rather than needing to re-commit. |
| Committed with `git commit --no-verify` (or the pre-commit hook was skipped some other way) and now nothing's pinned | Run `rad lfs backfill` — it walks every LFS-tracked file at HEAD and pins whichever ones are missing a note, without needing a new commit. |
| Large file didn't get pinned (commit went through, but no `rad-lfs` note) | The pre-commit hook silently skips pinning if the `ipfs`/`rad` CLIs aren't on `PATH` — check `command -v ipfs` / `command -v rad`. Re-add the file and commit again once available. |
| `private-repo LFS operations need to perform a key-agreement (ECDH) operation ...` | Your keystore is encrypted, ssh-agent can't do key agreement (protocol limitation), and no passphrase is available — normally you'd just be prompted interactively; this only happens in a non-interactive context (no TTY, e.g. CI). Set `RAD_PASSPHRASE`, or use an unencrypted keystore, or run it from a terminal. |
| `you are not an authorized recipient of this object` | Either genuinely unauthorized (not a delegate/allow-listed DID for this private repo), or authorized *after* this specific object was committed and nobody has run `rad lfs rekey` since — ask an already-authorized collaborator to run it and push. |
| `fatal: couldn't find remote ref refs/notes/rad-lfs` on `git fetch`/`git pull` (breaks *all* fetching, not just LFS) | Fixed — but if this repo had `rad lfs init` run against an older build, re-run `rad lfs init` once to replace the stale, now-invalid fetch refspec it configured (see `NOTES-lfs-notes-fetch-bug.md` for the full story). |

10
README.md
View File

@ -48,12 +48,10 @@
> ```
>
> From here, use `rad` exactly as upstream describes below (`rad auth`, `rad init`, etc). The
> only new commands are `rad lfs init` (run once per repository you want large-file support in),
> `rad lfs rekey` (run after granting a new collaborator access to a **private** repository, so
> they can decrypt previously-committed LFS objects too), and `rad lfs backfill` (retroactively
> pins any LFS-tracked file that got committed without going through the pre-commit hook, e.g.
> `--no-verify`) — see [`LFS-IPFS.md`](LFS-IPFS.md) for those workflows and how private-repo
> content gets encrypted before it reaches IPFS.
> only new commands are `rad lfs init` (run once per repository you want large-file support in)
> and `rad lfs rekey` (run after granting a new collaborator access to a **private** repository,
> so they can decrypt previously-committed LFS objects too) — see [`LFS-IPFS.md`](LFS-IPFS.md)
> for that workflow and how private-repo content gets encrypted before it reaches IPFS.
> **Nothing above requires IPFS**; Git LFS support specifically needs a running `ipfs daemon`,
> and `rad lfs init` will tell you plainly if one isn't reachable rather than failing confusingly
> later.

4
crates/radicle-cli/src/commands/lfs.rs
View File

@ -1,8 +1,6 @@
//! `rad lfs` command implementation.
pub mod backfill;
pub mod fetch;
pub mod fetch_batch;
pub mod init;
pub mod precommit;
pub mod rekey;
@ -22,7 +20,5 @@ pub fn run(args: Args, ctx: impl term::Context) -> anyhow::Result<()> {
Command::Fetch { oid, size, out } => self::fetch::run(oid, size, out, ctx),
Command::Rekey => self::rekey::run(ctx),
Command::Precommit => self::precommit::run(ctx),
Command::Backfill => self::backfill::run(ctx),
Command::FetchBatch => self::fetch_batch::run(ctx),
}
}

10
crates/radicle-cli/src/commands/lfs/args.rs
View File

@ -51,14 +51,4 @@ pub enum Command {
/// "<oid> <size> <path>" lines from stdin.
#[command(hide = true)]
Precommit,
/// Retroactively pin any LFS-tracked file at HEAD that was committed
/// without going through the pre-commit hook (e.g. `--no-verify`, or
/// the IPFS daemon/`ipfs`/`rad` weren't available at commit time)
Backfill,
/// Long-lived worker that fetches many LFS objects across one process
/// (invoked by the custom transfer agent; not for interactive use).
/// Reads "<oid> <size> <out-path>" lines from stdin, one at a time,
/// writing a JSON response line for each.
#[command(hide = true)]
FetchBatch,
}

171
crates/radicle-cli/src/commands/lfs/backfill.rs
View File

@ -1,171 +0,0 @@
//! `rad lfs backfill` — retroactively stores any already-committed
//! LFS-tracked file that doesn't yet have a `refs/notes/rad-lfs` note (so
//! it was never pinned to IPFS). This happens whenever the pre-commit hook
//! didn't run or didn't have what it needed at commit time: a commit made
//! with `--no-verify`, or one made while the `ipfs`/`rad` binaries weren't
//! on `PATH` yet (the hook soft-skips pinning in that case rather than
//! blocking the commit -- see `HOOK_BODY`'s `command -v` checks).
//!
//! Content actually needs to be available locally to backfill it from:
//! this reads from Git LFS's own local object cache
//! (`.git/lfs/objects/<oid prefix>/<oid>`, populated by the `git-lfs`
//! clean filter on `git add`, which -- unlike the pre-commit *hook* --
//! isn't skipped by `--no-verify`), falling back to the working-tree file
//! if that's missing. A file whose content isn't available through either
//! path can't be backfilled from this machine; some other peer that
//! already has it needs to run this instead.
use std::path::{Path, PathBuf};
use anyhow::Context as _;
use radicle::storage::{ReadRepository as _, ReadStorage as _};
use crate::ipfs;
use crate::terminal as term;
use super::store::store_object;
pub fn run(ctx: impl term::Context) -> anyhow::Result<()> {
let profile = ctx.profile()?;
let (repo, rid) = radicle::rad::cwd()
.context("`rad lfs backfill` must be run inside a Radicle repository working copy")?;
let workdir = repo
.workdir()
.ok_or_else(|| anyhow::anyhow!("`rad lfs backfill` needs a git working copy, not a bare repository"))?;
let doc = profile.storage.repository(rid)?.identity_doc()?.doc;
ipfs::check_daemon()?;
// Every path git currently tracks at HEAD (not the working tree or
// index, which may have unrelated local modifications) -- mirrors
// HOOK_BODY's own approach of shelling out to `git` for this rather
// than reimplementing tree/attribute logic against git2 directly.
let ls_files = radicle::git::run(Some(workdir), ["ls-tree", "-r", "-z", "--name-only", "HEAD"])
.context("failed to list files at HEAD")?;
let paths: Vec<String> = String::from_utf8_lossy(&ls_files.stdout)
.split('\0')
.filter(|s| !s.is_empty())
.map(str::to_string)
.collect();
let mut candidates = Vec::new();
for path in paths {
// Matches HOOK_BODY's own check: only files the *current*
// `.gitattributes` marks as `filter=lfs` are our concern.
let is_lfs = radicle::git::run(Some(workdir), ["check-attr", "filter", "--", &path])
.ok()
.map(|out| String::from_utf8_lossy(&out.stdout).contains(": filter: lfs"))
.unwrap_or(false);
if !is_lfs {
continue;
}
let blob_oid_out = radicle::git::run(Some(workdir), ["rev-parse", &format!("HEAD:{path}")])
.with_context(|| format!("failed to resolve blob oid for `{path}`"))?;
let blob_oid_str = String::from_utf8_lossy(&blob_oid_out.stdout).trim().to_string();
let Ok(blob_oid) = radicle::git::raw::Oid::from_str(&blob_oid_str) else {
continue;
};
candidates.push((path, blob_oid));
}
if candidates.is_empty() {
term::success!("No LFS-tracked files found in HEAD; nothing to backfill");
return Ok(());
}
let mut signer = None;
let mut backfilled = 0;
let mut already_pinned = 0;
let mut unavailable = Vec::new();
for (path, blob_oid) in candidates {
if !crate::lfs_crypto::find_cids(&repo, blob_oid)?.is_empty() {
already_pinned += 1;
continue;
}
let blob = repo
.find_blob(blob_oid)
.with_context(|| format!("failed to read blob for `{path}`"))?;
let Ok(pointer_text) = std::str::from_utf8(blob.content()) else {
continue;
};
let Some(oid) = pointer_text
.lines()
.find_map(|line| line.strip_prefix("oid sha256:"))
else {
continue;
};
let Some(size) = pointer_text
.lines()
.find_map(|line| line.strip_prefix("size "))
.and_then(|s| s.parse::<i64>().ok())
else {
continue;
};
let Some(content_path) = locate_content(workdir, oid, size, &path) else {
unavailable.push(path);
continue;
};
store_object(&profile, &repo, rid, &doc, oid, size, &content_path, &mut signer)
.with_context(|| format!("failed to backfill `{path}`"))?;
term::success!("Backfilled `{path}`");
backfilled += 1;
}
term::blank();
if backfilled > 0 {
term::success!("Backfilled {backfilled} object(s)");
}
if already_pinned > 0 {
term::info!("{already_pinned} object(s) already had a note; left untouched");
}
if !unavailable.is_empty() {
term::warning(format!(
"{} object(s) have no note and no content available locally to backfill from -- \
ask a peer that already has them to run `rad lfs backfill`:",
unavailable.len()
));
for path in &unavailable {
term::warning(format!(" {path}"));
}
}
Ok(())
}
/// Git LFS's own local object cache is the reliable source: the
/// `git-lfs` clean filter populates it on `git add` regardless of
/// whether the pre-commit *hook* ran (`--no-verify` skips hooks, not
/// filters). Falls back to the working-tree file, in case content is
/// present there but the LFS cache was pruned -- rejected if the size on
/// disk doesn't match what the pointer recorded, since that means it's
/// just the unsmudged pointer text, not the real content.
fn locate_content(workdir: &Path, oid: &str, size: i64, path: &str) -> Option<PathBuf> {
if oid.len() >= 4 {
let cached = workdir
.join(".git/lfs/objects")
.join(&oid[0..2])
.join(&oid[2..4])
.join(oid);
if matches_size(&cached, size) {
return Some(cached);
}
}
let working_copy = workdir.join(path);
if matches_size(&working_copy, size) {
return Some(working_copy);
}
None
}
fn matches_size(path: &Path, expected: i64) -> bool {
std::fs::metadata(path)
.map(|meta| meta.len() == expected as u64)
.unwrap_or(false)
}

37
crates/radicle-cli/src/commands/lfs/fetch.rs
View File

@ -3,15 +3,10 @@
//! content from the local IPFS (Kubo) node, decrypting it first if the
//! repository is private.
use std::path::{Path, PathBuf};
use std::path::PathBuf;
use anyhow::{Context as _, anyhow};
use radicle::Profile;
use radicle::git::raw::Repository;
use radicle::identity::RepoId;
use radicle_crypto::ssh::keystore::MemorySigner;
use crate::ipfs;
use crate::lfs_crypto;
use crate::terminal as term;
@ -21,32 +16,13 @@ pub fn run(oid: String, size: i64, out: PathBuf, ctx: impl term::Context) -> any
let (repo, rid) = radicle::rad::cwd()
.context("`rad lfs fetch` must be run inside a Radicle repository working copy")?;
let mut signer = None;
fetch_object(&profile, &repo, rid, &oid, size, &out, &mut signer)
}
/// Fetches one LFS object's content from IPFS (decrypting it first for
/// private repos) and writes it to `out`. Factored out of [`run`] so
/// `rad lfs fetch-batch` can process many objects across a single
/// process, loading (and prompting for) the signer at most once across
/// however many objects it ends up handling, instead of once per object.
#[allow(clippy::too_many_arguments)]
pub fn fetch_object(
profile: &Profile,
repo: &Repository,
rid: RepoId,
oid: &str,
size: i64,
out: &Path,
signer: &mut Option<MemorySigner>,
) -> anyhow::Result<()> {
let pointer_text =
format!("version https://git-lfs.github.com/spec/v1\noid sha256:{oid}\nsize {size}\n");
let blob_oid = repo
.blob(pointer_text.as_bytes())
.context("failed to write LFS pointer blob")?;
let candidates = lfs_crypto::find_envelopes(repo, blob_oid)
let candidates = lfs_crypto::find_envelopes(&repo, blob_oid)
.with_context(|| format!("failed to read LFS notes for oid {oid}"))?;
if candidates.is_empty() {
anyhow::bail!(
@ -67,16 +43,13 @@ pub fn fetch_object(
let result = match &envelope.enc {
None => ipfs::cat(&envelope.cid),
Some(enc) => ipfs::cat(&envelope.cid).and_then(|ciphertext| {
if signer.is_none() {
*signer = Some(lfs_crypto::load_signer(profile)?);
}
let signer = signer.as_ref().expect("just populated above");
lfs_crypto::decrypt_with(&ciphertext, enc, signer, profile.did(), &rid, oid)
let signer = lfs_crypto::load_signer(&profile)?;
lfs_crypto::decrypt_with(&ciphertext, enc, &signer, profile.did(), &rid, &oid)
}),
};
match result {
Ok(bytes) => {
std::fs::write(out, bytes)
std::fs::write(&out, bytes)
.with_context(|| format!("failed to write `{}`", out.display()))?;
return Ok(());
}

67
crates/radicle-cli/src/commands/lfs/fetch_batch.rs
View File

@ -1,67 +0,0 @@
//! `rad lfs fetch-batch` — long-lived plumbing worker invoked by the Git
//! LFS custom transfer agent (`rad-lfs-transfer`) to download multiple LFS
//! objects across a single process, kept alive for the lifetime of one
//! `git lfs pull`/checkout session -- specifically so a private repo's
//! passphrase is only prompted for once per session, not once per file.
//!
//! Unlike `rad lfs precommit`/`backfill` (which know every file upfront
//! and can just read stdin until EOF), this can't collect a batch before
//! starting: git-lfs's custom-transfer protocol requests objects one at a
//! time by default (`rad-lfs-transfer` doesn't negotiate the "concurrent"
//! capability), so the caller only knows about the next object once the
//! previous one has been answered. This process instead stays alive and
//! answers requests one at a time as they arrive, reusing the same
//! unwrapped signer across all of them.
//!
//! Protocol (line-delimited, UTF-8, on stdin/stdout):
//! request: "<oid> <size> <out-path>\n"
//! response: a JSON object, one line per request, in the same order:
//! `{"ok":true}` or `{"ok":false,"error":"<message>"}`
//! The process exits cleanly on stdin EOF (the parent closing its write
//! end signals the transfer session is over).
use std::io::{BufRead as _, Write as _};
use std::path::PathBuf;
use anyhow::Context as _;
use serde_json::json;
use crate::terminal as term;
use super::fetch::fetch_object;
pub fn run(ctx: impl term::Context) -> anyhow::Result<()> {
let profile = ctx.profile()?;
let (repo, rid) = radicle::rad::cwd()
.context("`rad lfs fetch-batch` must be run inside a Radicle repository working copy")?;
let mut signer = None;
let stdin = std::io::stdin();
let mut stdout = std::io::stdout();
for line in stdin.lock().lines() {
let line = line.context("failed to read stdin")?;
// Each line is "<oid> <size> <out-path>" -- oid is a fixed-length
// hex string and size is decimal, so splitting on the first two
// spaces leaves the path (which may itself contain spaces) intact
// as the remainder. Mirrors `rad lfs precommit`'s input format.
let mut parts = line.splitn(3, ' ');
let (Some(oid), Some(size), Some(out)) = (parts.next(), parts.next(), parts.next())
else {
anyhow::bail!("malformed `rad lfs fetch-batch` input line: {line:?}");
};
let Ok(size) = size.parse::<i64>() else {
anyhow::bail!("invalid size in `rad lfs fetch-batch` input: {size:?}");
};
let out = PathBuf::from(out);
let response = match fetch_object(&profile, &repo, rid, oid, size, &out, &mut signer) {
Ok(()) => json!({"ok": true}),
Err(err) => json!({"ok": false, "error": format!("{err:#}")}),
};
writeln!(stdout, "{response}").context("failed to write response")?;
stdout.flush().context("failed to flush stdout")?;
}
Ok(())
}

54
crates/radicle-cli/src/commands/unseed.rs
View File

@ -1,8 +1,5 @@
pub mod args;
use std::collections::BTreeSet;
use radicle::node::policy;
use radicle::{Node, prelude::*};
use crate::terminal as term;
@ -34,57 +31,10 @@ pub fn delete(rid: RepoId, node: &mut Node, profile: &Profile) -> anyhow::Result
}
// Mirror the seed/unseed lifecycle for IPFS pins of this repository's
// Git-LFS objects -- except any CID also referenced by an LFS note in
// some *other* still-seeded repository (e.g. two repositories that
// happen to both track a byte-identical file, giving it the same
// content-addressed CID). Unpinning unconditionally would evict
// content the other repository still needs, breaking it silently the
// next time someone there runs `git lfs fetch`. There's no
// cross-repository pin refcount kept anywhere -- this recomputes the
// still-needed set from every other seeded repository's own notes each
// time, which costs a pass over every other seeded repository but
// requires no extra persistent state, consistent with this design's
// git-notes-as-source-of-truth approach everywhere else.
// Git-LFS objects.
if !cids.is_empty() {
let still_needed = cids_still_needed_elsewhere(profile, rid)?;
let skipped = cids.iter().filter(|cid| still_needed.contains(*cid)).count();
let safe_to_unpin: Vec<String> =
cids.into_iter().filter(|cid| !still_needed.contains(cid)).collect();
if !safe_to_unpin.is_empty() {
crate::ipfs::unpin_all(&safe_to_unpin);
}
if skipped > 0 {
term::info!(
"Kept {skipped} IPFS object(s) pinned -- still referenced by another seeded repository"
);
}
crate::ipfs::unpin_all(&cids);
}
Ok(())
}
/// The set of CIDs (among `excluding`'s own) that are also referenced by an
/// LFS note in some other repository this node is currently seeding.
fn cids_still_needed_elsewhere(
profile: &Profile,
excluding: RepoId,
) -> anyhow::Result<BTreeSet<String>> {
let mut needed = BTreeSet::new();
let store = profile.policies()?;
for entry in store.seed_policies()? {
let policy::SeedPolicy { rid, policy } = entry?;
if rid == excluding {
continue;
}
if !matches!(policy, policy::SeedingPolicy::Allow { .. }) {
continue;
}
if let Ok(repo) = profile.storage.repository(rid) {
needed.extend(crate::ipfs::lfs_cids(&repo.backend));
}
}
Ok(needed)
}