Add rad lfs init and IPFS-backed pinning for seed/unseed
Adds Git LFS support for repositories, with large file content stored on each contributor's own local IPFS node instead of a seed-hosted server: - `rad lfs init`: one-time setup (transfer-agent config, pre-commit hook, notes-ref push/fetch refspecs, IPFS daemon reachability check). - `refs/notes/rad-lfs` carries the oid -> CID mapping, replicating with the repository via Radicle's existing ref sync (verified against `references_of` in crates/radicle/src/storage/git.rs: every ref is replicated except refs/tmp/heads/*, so no protocol changes were needed). - `rad seed`/`rad unseed` now pin/unpin a repo's known LFS objects in IPFS, mirroring Radicle's existing "seeding = keep a full copy" model, and warning (not failing) if no local IPFS daemon is reachable. The actual byte transfer is handled by a separate binary, rad-lfs-transfer (added here as a submodule), which implements the Git LFS custom-transfer- agent protocol against a local IPFS daemon. See LFS-IPFS.md for the full design and setup instructions. Building and using `rad` for anything other than `rad lfs` requires no IPFS dependency at all; `rad lfs init` checks for a local daemon upfront and fails with an actionable message rather than proceeding silently.
This commit is contained in:
Maciek "mab122" Bator
parent
b5e6d82769
commit
bb8608e984
|
|
@ -0,0 +1,3 @@
|
|||
[submodule "radicle-lfs-transfer"]
|
||||
path = radicle-lfs-transfer
|
||||
url = ../radicle-lfs-transfer.git
|
||||
|
|
@ -1846,6 +1846,22 @@ dependencies = [
|
|||
"digest",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "http"
|
||||
version = "1.4.2"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "6970f50e31d6fc17d3fa27329444bfa74e196cf62e95052a3f6fee181dba6425"
|
||||
dependencies = [
|
||||
"bytes",
|
||||
"itoa",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "httparse"
|
||||
version = "1.10.1"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "6dbf3de79e51f3d586ab4cb9d5c3e2c14aa28ed23d180cf89b4df0454a69cc87"
|
||||
|
||||
[[package]]
|
||||
name = "human-panic"
|
||||
version = "2.0.6"
|
||||
|
|
@ -2992,6 +3008,7 @@ dependencies = [
|
|||
"tree-sitter-rust",
|
||||
"tree-sitter-toml-ng",
|
||||
"tree-sitter-typescript",
|
||||
"ureq",
|
||||
"winsplit",
|
||||
]
|
||||
|
||||
|
|
@ -4689,6 +4706,31 @@ dependencies = [
|
|||
"subtle",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "ureq"
|
||||
version = "3.3.0"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "dea7109cdcd5864d4eeb1b58a1648dc9bf520360d7af16ec26d0a9354bafcfc0"
|
||||
dependencies = [
|
||||
"base64 0.22.1",
|
||||
"log",
|
||||
"percent-encoding",
|
||||
"ureq-proto",
|
||||
"utf8-zero",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "ureq-proto"
|
||||
version = "0.6.0"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "e994ba84b0bd1b1b0cf92878b7ef898a5c1760108fe7b6010327e274917a808c"
|
||||
dependencies = [
|
||||
"base64 0.22.1",
|
||||
"http",
|
||||
"httparse",
|
||||
"log",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "url"
|
||||
version = "2.5.8"
|
||||
|
|
@ -4701,6 +4743,12 @@ dependencies = [
|
|||
"serde",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "utf8-zero"
|
||||
version = "0.8.1"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "b8c0a043c9540bae7c578c88f91dda8bd82e59ae27c21baca69c8b191aaf5a6e"
|
||||
|
||||
[[package]]
|
||||
name = "utf8_iter"
|
||||
version = "1.0.4"
|
||||
|
|
|
|||
|
|
@ -75,6 +75,7 @@ sqlite = "0.37.0"
|
|||
tempfile = "3.3.0"
|
||||
thiserror = { version = "2", default-features = false }
|
||||
uds_windows = "1.1.0"
|
||||
ureq = { version = "3", default-features = false }
|
||||
windows = "0.62"
|
||||
winpipe = "0.1.1"
|
||||
winsplit = "0.1.0"
|
||||
|
|
|
|||
|
|
@ -0,0 +1,68 @@
|
|||
# Git LFS support, backed by IPFS
|
||||
|
||||
This is a fork of [Radicle's `heartwood`][heartwood] adding Git LFS (large file) support,
|
||||
with large file content stored on each contributor's own local IPFS node rather than on a
|
||||
central server.
|
||||
|
||||
[heartwood]: https://github.com/radicle-dev/heartwood
|
||||
|
||||
## Why not a server?
|
||||
|
||||
Radicle's own FAQ notes that its first-generation protocol was built on IPFS and moved away
|
||||
from it — IPFS is a content-addressed store, and repositories are mutable, so it wasn't a fit
|
||||
for Radicle's core storage. That objection doesn't apply here: Git LFS's own pattern already
|
||||
separates a small, mutable pointer file (which Radicle replicates exactly as it replicates any
|
||||
other git object) from immutable large-file content addressed by hash — which is precisely
|
||||
what IPFS is good at. This fork only uses IPFS for that second part.
|
||||
|
||||
A seed-hosted HTTP LFS server was considered and prototyped first, but rejected: it would have
|
||||
recreated a single point of failure, and Radicle's node/address book has no existing mechanism
|
||||
to advertise a seed's HTTP endpoint (only P2P gossip addresses), so real seed discovery would
|
||||
have needed new protocol work outside this fork's scope. Instead, every peer uses their own
|
||||
local IPFS node, and the file-CID mapping travels with the repository itself.
|
||||
|
||||
## How it works
|
||||
|
||||
- **The mapping**: a `cid=<cid>` git note on `refs/notes/rad-lfs`, attached to each LFS
|
||||
pointer file's own git blob hash (not the LFS oid — a different value). Verified against
|
||||
this repo's actual replication code (`references_of` in
|
||||
`crates/radicle/src/storage/git.rs`) that Radicle replicates every ref under a peer's
|
||||
namespace except `refs/tmp/heads/*` — there's no fixed allowlist, so this notes ref
|
||||
replicates the same way `refs/heads`/`refs/tags`/`refs/cobs` do, once the push/fetch
|
||||
refspecs below are configured.
|
||||
- **On commit**: a pre-commit hook (installed by `rad lfs init`) adds each staged LFS-tracked
|
||||
file to your local IPFS daemon, pins it, and writes the note above.
|
||||
- **On push/pull**: the actual bytes move through
|
||||
[`rad-lfs-transfer`][rad-lfs-transfer], a Git LFS custom transfer agent — see that repo for
|
||||
details. It's a separate binary/repository since it has nothing Radicle-specific in it; it
|
||||
only needs a local IPFS daemon and a git repository with the notes above.
|
||||
- **Seeding**: `rad seed` now also pins a repository's known LFS objects in your local IPFS
|
||||
node (mirroring "seeding = keep a full copy"); `rad unseed` unpins them. Known prototype
|
||||
limitation: no cross-repository pin refcounting, so unseeding one repository can unpin
|
||||
content still needed by another seeded repository that happens to share the same CID.
|
||||
|
||||
[rad-lfs-transfer]: ../radicle-lfs-transfer
|
||||
|
||||
## Setup
|
||||
|
||||
```sh
|
||||
# Build & install both binaries (from a checkout with the submodule fetched)
|
||||
git submodule update --init
|
||||
cargo install --path .
|
||||
cargo install --path radicle-lfs-transfer
|
||||
|
||||
# One-time, per repository
|
||||
rad lfs init
|
||||
```
|
||||
|
||||
`rad lfs init` requires a local IPFS (Kubo) daemon to be reachable — it checks upfront and
|
||||
fails with a clear message (e.g. "start one with `ipfs daemon`") rather than silently
|
||||
continuing. **Nothing else in this fork requires IPFS.** Building, installing, and using `rad`
|
||||
for anything other than the `lfs` command works exactly as upstream, with no IPFS dependency
|
||||
at all. `rad seed`/`rad unseed` will warn (not fail) if a local IPFS daemon isn't reachable
|
||||
when they try to pin/unpin LFS objects — seeding/unseeding itself still succeeds.
|
||||
|
||||
## Status
|
||||
|
||||
This is a prototype, not upstream Radicle functionality. See the commits on the
|
||||
`rad-lfs-ipfs` branch for the full change set relative to upstream `heartwood`.
|
||||
|
|
@ -56,6 +56,7 @@ tree-sitter-ruby = "0.23.1"
|
|||
tree-sitter-rust = "0.23.2"
|
||||
tree-sitter-toml-ng = "0.6.0"
|
||||
tree-sitter-typescript = "0.23.2"
|
||||
ureq = { workspace = true }
|
||||
|
||||
[target.'cfg(unix)'.dependencies]
|
||||
backtrace = { workspace = true, optional = true }
|
||||
|
|
|
|||
|
|
@ -14,6 +14,7 @@ pub mod inbox;
|
|||
pub mod init;
|
||||
pub mod inspect;
|
||||
pub mod issue;
|
||||
pub mod lfs;
|
||||
pub mod ls;
|
||||
pub mod node;
|
||||
pub mod patch;
|
||||
|
|
|
|||
|
|
@ -0,0 +1,20 @@
|
|||
//! `rad lfs` command implementation.
|
||||
|
||||
pub mod init;
|
||||
|
||||
mod args;
|
||||
|
||||
use crate::terminal as term;
|
||||
|
||||
pub use args::Args;
|
||||
use args::Command;
|
||||
|
||||
pub fn run(args: Args, ctx: impl term::Context) -> anyhow::Result<()> {
|
||||
// No subcommand currently needs a `Profile`, but we take `ctx` to stay
|
||||
// consistent with the other top-level commands' `run` signature.
|
||||
let _ = ctx;
|
||||
|
||||
match args.command {
|
||||
Command::Init => self::init::run(),
|
||||
}
|
||||
}
|
||||
|
|
@ -0,0 +1,25 @@
|
|||
use clap::{Parser, Subcommand};
|
||||
|
||||
const ABOUT: &str = "Manage Git LFS (large file) support, backed by IPFS";
|
||||
|
||||
const LONG_ABOUT: &str = r#"
|
||||
The `lfs` command manages Git LFS (Large File Storage) support for a
|
||||
Radicle repository. Large file content is stored on the contributor's
|
||||
local IPFS (Kubo) node rather than on a seed-hosted HTTP server: there
|
||||
is no additional server-side infrastructure to run.
|
||||
|
||||
See `rad lfs init --help` for the one-time setup command.
|
||||
"#;
|
||||
|
||||
#[derive(Parser, Debug)]
|
||||
#[command(about = ABOUT, long_about = LONG_ABOUT, disable_version_flag = true)]
|
||||
pub struct Args {
|
||||
#[command(subcommand)]
|
||||
pub command: Command,
|
||||
}
|
||||
|
||||
#[derive(Subcommand, Debug)]
|
||||
pub enum Command {
|
||||
/// Set up Git LFS support for this repository, backed by IPFS
|
||||
Init,
|
||||
}
|
||||
|
|
@ -0,0 +1,249 @@
|
|||
//! `rad lfs init` — one-time, idempotent setup of Git LFS support for a
|
||||
//! Radicle repository, backed by the contributor's local IPFS (Kubo) node.
|
||||
//!
|
||||
//! This deliberately does *not* talk to any seed-hosted HTTP LFS server:
|
||||
//! large file content lives on each peer's own IPFS node, and the
|
||||
//! oid -> CID mapping is carried in a git-notes ref (`refs/notes/rad-lfs`)
|
||||
//! that replicates alongside the rest of the repository once the
|
||||
//! push/fetch refspecs below are configured.
|
||||
|
||||
use std::fs;
|
||||
#[cfg(unix)]
|
||||
use std::os::unix::fs::PermissionsExt;
|
||||
use std::path::Path;
|
||||
use std::process::Command;
|
||||
|
||||
use anyhow::{Context as _, anyhow};
|
||||
|
||||
use radicle::git::raw::Repository;
|
||||
|
||||
use crate::git;
|
||||
use crate::ipfs::{self, LFS_NOTES_REF as NOTES_REF};
|
||||
use crate::terminal as term;
|
||||
|
||||
/// Name under which our custom transfer agent is registered with Git LFS.
|
||||
const TRANSFER_AGENT_NAME: &str = "rad-ipfs";
|
||||
|
||||
/// Binary implementing the custom transfer agent. Must be on the user's
|
||||
/// `PATH`; we intentionally don't assume a specific install location.
|
||||
const TRANSFER_AGENT_BIN: &str = "rad-lfs-transfer";
|
||||
|
||||
/// Name of the remote that `rad init` sets up.
|
||||
const RAD_REMOTE: &str = "rad";
|
||||
|
||||
/// Markers delimiting the block we manage inside `.git/hooks/pre-commit`,
|
||||
/// so re-running `rad lfs init` updates our block in place instead of
|
||||
/// duplicating it, and so we don't clobber any hook content that was
|
||||
/// already there.
|
||||
const HOOK_BEGIN: &str = "# >>> rad-lfs-init managed pre-commit hook >>>";
|
||||
const HOOK_END: &str = "# <<< rad-lfs-init managed pre-commit hook <<<";
|
||||
|
||||
/// Body of the managed pre-commit hook block (POSIX `sh`).
|
||||
///
|
||||
/// For every file staged in the commit that's tracked by Git LFS (i.e.
|
||||
/// matches a `filter=lfs` pattern in `.gitattributes`), this:
|
||||
/// 1. `ipfs add`s the real file content to the local Kubo daemon,
|
||||
/// 2. `ipfs pin add`s the resulting CID,
|
||||
/// 3. computes the git blob hash of the LFS pointer text for that file, and
|
||||
/// 4. writes a `cid=<cid>` git note on that blob hash under
|
||||
/// `refs/notes/rad-lfs`.
|
||||
///
|
||||
/// This mirrors `radicle-lfs-server`'s `notes::pointer_blob_hash` /
|
||||
/// `notes::write_cid` logic (see that crate's `src/notes.rs`), reimplemented
|
||||
/// here as a shell script since a git hook has no business depending on a
|
||||
/// separate Rust crate.
|
||||
const HOOK_BODY: &str = r#"rad_lfs_precommit() {
|
||||
command -v ipfs >/dev/null 2>&1 || {
|
||||
echo "rad-lfs: 'ipfs' CLI not found on PATH; skipping CID pinning for this commit" >&2
|
||||
return 0
|
||||
}
|
||||
|
||||
sha256() {
|
||||
if command -v sha256sum >/dev/null 2>&1; then
|
||||
sha256sum "$1" | awk '{ print $1 }'
|
||||
else
|
||||
shasum -a 256 "$1" | awk '{ print $1 }'
|
||||
fi
|
||||
}
|
||||
|
||||
staged=$(mktemp) || return 1
|
||||
git diff --cached --name-only --diff-filter=ACM > "$staged"
|
||||
|
||||
while IFS= read -r file; do
|
||||
[ -n "$file" ] || continue
|
||||
[ -f "$file" ] || continue
|
||||
|
||||
attr=$(git check-attr filter -- "$file" | sed -n 's/.*: filter: //p')
|
||||
[ "$attr" = "lfs" ] || continue
|
||||
|
||||
oid=$(sha256 "$file") || { rm -f "$staged"; exit 1; }
|
||||
size=$(wc -c < "$file" | tr -d ' ')
|
||||
|
||||
cid=$(ipfs add -Q -- "$file") || {
|
||||
echo "rad-lfs: 'ipfs add' failed for $file" >&2
|
||||
rm -f "$staged"
|
||||
exit 1
|
||||
}
|
||||
ipfs pin add -- "$cid" >/dev/null || {
|
||||
echo "rad-lfs: 'ipfs pin add' failed for $cid" >&2
|
||||
rm -f "$staged"
|
||||
exit 1
|
||||
}
|
||||
|
||||
blob_sha=$(printf 'version https://git-lfs.github.com/spec/v1\noid sha256:%s\nsize %s\n' "$oid" "$size" | git hash-object --stdin -t blob) || {
|
||||
rm -f "$staged"
|
||||
exit 1
|
||||
}
|
||||
|
||||
git notes --ref=refs/notes/rad-lfs add -f -m "cid=$cid" "$blob_sha" >/dev/null || {
|
||||
echo "rad-lfs: failed to write git note for $file" >&2
|
||||
rm -f "$staged"
|
||||
exit 1
|
||||
}
|
||||
done < "$staged"
|
||||
|
||||
rm -f "$staged"
|
||||
}
|
||||
|
||||
rad_lfs_precommit"#;
|
||||
|
||||
pub fn run() -> anyhow::Result<()> {
|
||||
let (repo, _rid) = radicle::rad::cwd()
|
||||
.context("`rad lfs init` must be run inside a Radicle repository working copy")?;
|
||||
let workdir = repo.workdir().ok_or_else(|| {
|
||||
anyhow!("`rad lfs init` must be run inside a git working copy (not a bare repository)")
|
||||
})?;
|
||||
|
||||
ipfs::check_daemon()?;
|
||||
term::success!("Found a local IPFS (Kubo) daemon at {}", ipfs::kubo_api_url());
|
||||
|
||||
ensure_git_lfs_available()?;
|
||||
git::git(workdir, ["lfs", "install", "--local"])
|
||||
.context("failed to run `git lfs install --local`")?;
|
||||
term::success!("Initialized Git LFS for this repository");
|
||||
|
||||
git::git(
|
||||
workdir,
|
||||
["config", "lfs.standalonetransferagent", TRANSFER_AGENT_NAME],
|
||||
)
|
||||
.context("failed to configure the LFS standalone transfer agent")?;
|
||||
git::git(
|
||||
workdir,
|
||||
[
|
||||
"config",
|
||||
&format!("lfs.customtransfer.{TRANSFER_AGENT_NAME}.path"),
|
||||
TRANSFER_AGENT_BIN,
|
||||
],
|
||||
)
|
||||
.context("failed to configure the LFS custom transfer agent path")?;
|
||||
term::success!(
|
||||
"Configured Git LFS to transfer objects via `{TRANSFER_AGENT_BIN}` (backed by IPFS)"
|
||||
);
|
||||
|
||||
let notes_refspec = format!("+{NOTES_REF}:{NOTES_REF}");
|
||||
ensure_refspec(workdir, &format!("remote.{RAD_REMOTE}.push"), ¬es_refspec)?;
|
||||
ensure_refspec(workdir, &format!("remote.{RAD_REMOTE}.fetch"), ¬es_refspec)?;
|
||||
term::success!(
|
||||
"Configured the `{RAD_REMOTE}` remote to push/fetch the `{NOTES_REF}` notes ref"
|
||||
);
|
||||
|
||||
install_pre_commit_hook(&repo)?;
|
||||
term::success!("Installed the `rad-lfs` pre-commit hook");
|
||||
|
||||
term::blank();
|
||||
term::success!("Git LFS is now configured for this repository, backed by your local IPFS node.");
|
||||
term::info!(
|
||||
"File CIDs are recorded in the `{NOTES_REF}` git-notes ref, which now travels with every `rad push` / `rad pull`."
|
||||
);
|
||||
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Make sure the `git-lfs` binary is installed before we shell out to it,
|
||||
/// so we can give a clear, actionable error instead of a raw "unknown git
|
||||
/// command" failure.
|
||||
fn ensure_git_lfs_available() -> anyhow::Result<()> {
|
||||
match Command::new("git-lfs").arg("version").output() {
|
||||
Ok(output) if output.status.success() => Ok(()),
|
||||
Ok(output) => Err(anyhow!(
|
||||
"`git-lfs version` failed: {}",
|
||||
String::from_utf8_lossy(&output.stderr)
|
||||
)),
|
||||
Err(err) if err.kind() == std::io::ErrorKind::NotFound => Err(anyhow!(
|
||||
"Git LFS is not installed (the `git-lfs` binary was not found on your PATH). \
|
||||
Install Git LFS — see https://git-lfs.com — and try again."
|
||||
)),
|
||||
Err(err) => Err(err).context("failed to check for `git-lfs`"),
|
||||
}
|
||||
}
|
||||
|
||||
/// Add `value` to the (potentially multi-valued) git config key `key`,
|
||||
/// unless it's already present, so re-running `rad lfs init` doesn't
|
||||
/// duplicate refspecs.
|
||||
fn ensure_refspec(workdir: &Path, key: &str, value: &str) -> anyhow::Result<()> {
|
||||
// `git config --get-all` exits non-zero when the key is unset, which
|
||||
// isn't an error for us, so we use the lower-level `run` rather than
|
||||
// `git::git` (which treats a non-zero exit as failure).
|
||||
let output = radicle::git::run(Some(workdir), ["config", "--get-all", key])
|
||||
.with_context(|| format!("failed to read git config `{key}`"))?;
|
||||
let existing = String::from_utf8_lossy(&output.stdout);
|
||||
if existing.lines().any(|line| line.trim() == value) {
|
||||
return Ok(());
|
||||
}
|
||||
|
||||
git::git(workdir, ["config", "--add", key, value])
|
||||
.with_context(|| format!("failed to set git config `{key}`"))?;
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Install (or update, or chain onto) the `.git/hooks/pre-commit` hook that
|
||||
/// pins staged LFS objects to IPFS and records their CID as a git note.
|
||||
fn install_pre_commit_hook(repo: &Repository) -> anyhow::Result<()> {
|
||||
let hooks_dir = repo.path().join("hooks");
|
||||
fs::create_dir_all(&hooks_dir)
|
||||
.with_context(|| format!("failed to create hooks directory `{}`", hooks_dir.display()))?;
|
||||
let hook_path = hooks_dir.join("pre-commit");
|
||||
|
||||
let existing = fs::read_to_string(&hook_path).unwrap_or_default();
|
||||
|
||||
let new_content = if let Some((before, after_begin)) = existing.split_once(HOOK_BEGIN) {
|
||||
// We've already installed our block before: replace it in place so
|
||||
// re-running `rad lfs init` doesn't duplicate it, while preserving
|
||||
// anything the user added before/after it.
|
||||
let after = after_begin
|
||||
.split_once(HOOK_END)
|
||||
.map_or("", |(_, after)| after);
|
||||
format!("{before}{HOOK_BEGIN}\n{HOOK_BODY}\n{HOOK_END}{after}")
|
||||
} else if existing.trim().is_empty() {
|
||||
format!("#!/bin/sh\n\n{HOOK_BEGIN}\n{HOOK_BODY}\n{HOOK_END}\n")
|
||||
} else {
|
||||
// Some other hook (not ours) is already installed (`git lfs
|
||||
// install` itself only installs a pre-push hook, so this is
|
||||
// usually a hook the user or another tool set up). Chain onto it
|
||||
// rather than clobbering it.
|
||||
let mut content = existing;
|
||||
if !content.ends_with('\n') {
|
||||
content.push('\n');
|
||||
}
|
||||
content.push('\n');
|
||||
content.push_str(HOOK_BEGIN);
|
||||
content.push('\n');
|
||||
content.push_str(HOOK_BODY);
|
||||
content.push('\n');
|
||||
content.push_str(HOOK_END);
|
||||
content.push('\n');
|
||||
content
|
||||
};
|
||||
|
||||
fs::write(&hook_path, new_content)
|
||||
.with_context(|| format!("failed to write hook `{}`", hook_path.display()))?;
|
||||
|
||||
#[cfg(unix)]
|
||||
{
|
||||
let mut perms = fs::metadata(&hook_path)?.permissions();
|
||||
perms.set_mode(perms.mode() | 0o111);
|
||||
fs::set_permissions(&hook_path, perms)?;
|
||||
}
|
||||
|
||||
Ok(())
|
||||
}
|
||||
|
|
@ -70,6 +70,19 @@ pub fn update(
|
|||
term::format::tertiary(rid),
|
||||
);
|
||||
|
||||
// Since seeding a repository means keeping a full local copy of it,
|
||||
// also pin any of its Git-LFS objects that are stored in IPFS, so
|
||||
// that they follow the same seed/unseed lifecycle.
|
||||
if let Ok(repo) = profile.storage.repository(rid) {
|
||||
let cids = crate::ipfs::lfs_cids(&repo.backend);
|
||||
if !cids.is_empty() {
|
||||
let pinned = crate::ipfs::pin_all(&cids);
|
||||
if pinned > 0 {
|
||||
term::success!("Pinned {pinned} LFS object(s) in IPFS");
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
Ok(())
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -18,8 +18,23 @@ pub fn run(args: Args, ctx: impl term::Context) -> anyhow::Result<()> {
|
|||
}
|
||||
|
||||
pub fn delete(rid: RepoId, node: &mut Node, profile: &Profile) -> anyhow::Result<()> {
|
||||
// Gather the repository's known LFS objects *before* unseeding, since
|
||||
// we still need access to its git notes at that point.
|
||||
let cids = profile
|
||||
.storage
|
||||
.repository(rid)
|
||||
.map(|repo| crate::ipfs::lfs_cids(&repo.backend))
|
||||
.unwrap_or_default();
|
||||
|
||||
if profile.unseed(rid, node)? {
|
||||
term::success!("Seeding policy for {} removed", term::format::tertiary(rid));
|
||||
}
|
||||
|
||||
// Mirror the seed/unseed lifecycle for IPFS pins of this repository's
|
||||
// Git-LFS objects.
|
||||
if !cids.is_empty() {
|
||||
crate::ipfs::unpin_all(&cids);
|
||||
}
|
||||
|
||||
Ok(())
|
||||
}
|
||||
|
|
|
|||
|
|
@ -0,0 +1,149 @@
|
|||
//! Pinning of Git-LFS-backed content in a local IPFS (Kubo) node.
|
||||
//!
|
||||
//! This is a prototype of "Git-LFS over IPFS": large files tracked via
|
||||
//! Git LFS are stored in IPFS rather than as git blobs, and the mapping
|
||||
//! from an LFS pointer file's git blob oid to the corresponding IPFS CID
|
||||
//! is recorded as a git note on `refs/notes/rad-lfs` (note content
|
||||
//! `cid=<cid>`).
|
||||
//!
|
||||
//! Since seeding a repository already means "keep a full local copy of
|
||||
//! it", we extend that lifecycle to IPFS: seeding a repository pins all
|
||||
//! of its known LFS objects locally, and unseeding unpins them.
|
||||
|
||||
use std::env;
|
||||
|
||||
use anyhow::anyhow;
|
||||
use radicle::git::raw::Repository;
|
||||
|
||||
use crate::terminal as term;
|
||||
|
||||
/// Git notes ref under which oid -> CID mappings for LFS objects are stored.
|
||||
pub const LFS_NOTES_REF: &str = "refs/notes/rad-lfs";
|
||||
|
||||
/// Default HTTP API address of a local Kubo (IPFS) daemon.
|
||||
const DEFAULT_KUBO_API_URL: &str = "http://127.0.0.1:5001";
|
||||
|
||||
/// Returns the Kubo HTTP API base URL, respecting the `KUBO_API_URL`
|
||||
/// environment variable if set.
|
||||
pub(crate) fn kubo_api_url() -> String {
|
||||
env::var("KUBO_API_URL").unwrap_or_else(|_| DEFAULT_KUBO_API_URL.to_string())
|
||||
}
|
||||
|
||||
/// Check that a local IPFS (Kubo) daemon is reachable, failing with a
|
||||
/// clear, actionable error otherwise. Used by `rad lfs init` to refuse to
|
||||
/// proceed silently when there's no daemon to back LFS content with.
|
||||
pub fn check_daemon() -> anyhow::Result<()> {
|
||||
let base = kubo_api_url();
|
||||
let url = format!("{base}/api/v0/id");
|
||||
|
||||
match ureq::post(&url).send_empty() {
|
||||
Ok(_) => Ok(()),
|
||||
Err(err) if is_daemon_unreachable(&err) => Err(anyhow!(
|
||||
"no IPFS (Kubo) daemon reachable at {base} — start one with `ipfs daemon`.\n\
|
||||
This repo's large-file (LFS) support is backed by your local IPFS node."
|
||||
)),
|
||||
Err(err) => Err(anyhow!("IPFS daemon at {base} responded unexpectedly: {err}")),
|
||||
}
|
||||
}
|
||||
|
||||
/// Collect the set of CIDs referenced by LFS notes on [`LFS_NOTES_REF`].
|
||||
///
|
||||
/// Returns an empty vector if the notes ref doesn't exist, i.e. the
|
||||
/// repository has no LFS objects tracked yet. This is not an error.
|
||||
#[must_use]
|
||||
pub fn lfs_cids(repo: &Repository) -> Vec<String> {
|
||||
let notes = match repo.notes(Some(LFS_NOTES_REF)) {
|
||||
Ok(notes) => notes,
|
||||
// The `refs/notes/rad-lfs` ref doesn't exist, which just means
|
||||
// there are no LFS objects tracked for this repository yet.
|
||||
Err(_) => return Vec::new(),
|
||||
};
|
||||
|
||||
let mut cids = Vec::new();
|
||||
for note in notes {
|
||||
let Ok((_, annotated_id)) = note else {
|
||||
continue;
|
||||
};
|
||||
let Ok(note) = repo.find_note(Some(LFS_NOTES_REF), annotated_id) else {
|
||||
continue;
|
||||
};
|
||||
let Some(message) = note.message() else {
|
||||
continue;
|
||||
};
|
||||
if let Some(cid) = message.trim().strip_prefix("cid=") {
|
||||
cids.push(cid.to_string());
|
||||
}
|
||||
}
|
||||
cids
|
||||
}
|
||||
|
||||
/// Whether a `ureq` error indicates that the local IPFS daemon could
|
||||
/// not be reached at all, as opposed to an HTTP-level error response.
|
||||
fn is_daemon_unreachable(err: &ureq::Error) -> bool {
|
||||
matches!(
|
||||
err,
|
||||
ureq::Error::ConnectionFailed
|
||||
| ureq::Error::Io(_)
|
||||
| ureq::Error::HostNotFound
|
||||
| ureq::Error::Timeout(_)
|
||||
)
|
||||
}
|
||||
|
||||
/// Pin the given CIDs recursively in the local IPFS daemon.
|
||||
///
|
||||
/// Returns the number of CIDs successfully pinned. If the local daemon
|
||||
/// is not reachable, a warning is printed and `0` is returned; this is
|
||||
/// not treated as a hard failure, since seeding itself already succeeded.
|
||||
#[must_use]
|
||||
pub fn pin_all(cids: &[String]) -> usize {
|
||||
let base = kubo_api_url();
|
||||
let mut pinned = 0;
|
||||
|
||||
for cid in cids {
|
||||
let url = format!("{base}/api/v0/pin/add?arg={cid}&recursive=true");
|
||||
match ureq::post(&url).send_empty() {
|
||||
Ok(_) => pinned += 1,
|
||||
Err(err) if is_daemon_unreachable(&err) => {
|
||||
term::warning(format!(
|
||||
"Could not reach local IPFS daemon at {base}; LFS objects were not pinned"
|
||||
));
|
||||
return pinned;
|
||||
}
|
||||
Err(err) => {
|
||||
term::warning(format!("Failed to pin LFS object {cid} in IPFS: {err}"));
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
pinned
|
||||
}
|
||||
|
||||
/// Unpin the given CIDs from the local IPFS daemon.
|
||||
///
|
||||
/// If the local daemon is not reachable, a warning is printed and the
|
||||
/// function returns without failing; this is not treated as a hard
|
||||
/// failure, since unseeding itself already succeeded.
|
||||
pub fn unpin_all(cids: &[String]) {
|
||||
let base = kubo_api_url();
|
||||
|
||||
for cid in cids {
|
||||
let url = format!("{base}/api/v0/pin/rm?arg={cid}");
|
||||
// NOTE: this unconditionally unpins `cid`, even if it's still
|
||||
// referenced by LFS objects of another repository that remains
|
||||
// seeded. This prototype has no cross-repo pin refcounting, so
|
||||
// a CID shared between repositories will be unpinned here
|
||||
// regardless of whether other seeded repos still need it.
|
||||
match ureq::post(&url).send_empty() {
|
||||
Ok(_) => {}
|
||||
Err(err) if is_daemon_unreachable(&err) => {
|
||||
term::warning(format!(
|
||||
"Could not reach local IPFS daemon at {base}; LFS objects were not unpinned"
|
||||
));
|
||||
return;
|
||||
}
|
||||
Err(err) => {
|
||||
term::warning(format!("Failed to unpin LFS object {cid} in IPFS: {err}"));
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -4,6 +4,7 @@
|
|||
#![deny(clippy::print_stdout)]
|
||||
pub mod commands;
|
||||
pub mod git;
|
||||
pub mod ipfs;
|
||||
pub mod node;
|
||||
pub mod pager;
|
||||
pub mod project;
|
||||
|
|
|
|||
|
|
@ -76,6 +76,7 @@ enum Command {
|
|||
#[command(alias = ".")]
|
||||
Inspect(inspect::Args),
|
||||
Issue(issue::Args),
|
||||
Lfs(lfs::Args),
|
||||
Ls(ls::Args),
|
||||
Node(node::Args),
|
||||
Patch(patch::Args),
|
||||
|
|
@ -271,6 +272,7 @@ fn run_command(command: Command, ctx: impl term::Context) -> Result<(), anyhow::
|
|||
Command::Init(args) => init::run(args, ctx),
|
||||
Command::Inspect(args) => inspect::run(args, ctx),
|
||||
Command::Issue(args) => issue::run(args, ctx),
|
||||
Command::Lfs(args) => lfs::run(args, ctx),
|
||||
Command::Ls(args) => ls::run(args, ctx),
|
||||
Command::Node(args) => node::run(args, ctx),
|
||||
Command::Patch(args) => patch::run(args, ctx),
|
||||
|
|
|
|||
|
|
@ -0,0 +1 @@
|
|||
Subproject commit fb95fd0c68eb260b671839f74235d03b252b19d7
|
||||
Loading…
Reference in New Issue