mab122
/
radicle-lfs-transfer
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
radicle-lfs-transfer/README.md

5.8 KiB
Raw Permalink Blame History

radicle-lfs-transfer

A Git LFS custom transfer agent that backs large file storage with IPFS, for use with Radicle repositories.

This is the companion binary for the rad lfs support added in radicle-heartwood-lfs (a fork of Radicle's core node/CLI, heartwood). Most people want to start there, not here — that repo has the full zero-to-usable setup guide. This repo is what you're reading if you want to build/understand the transfer agent on its own, or you're following a link back from there.

rad-lfs-transfer is not meant to be invoked directly — git lfs spawns it automatically, once a repository has been set up with rad lfs init (from the fork above).

How it works

Every peer uses their own local IPFS node — there's no shared server. All of the actual IPFS add/pin work and git-notes bookkeeping lives in the rad lfs store/rad lfs fetch/ rad lfs precommit subcommands, implemented in the companion radicle-heartwood-lfs fork — that's also where content gets encrypted for private repositories before it ever reaches IPFS. This crate deliberately has no dependency on Radicle's identity/crypto stack; it just shells out:

  • On git add/git commit, a pre-commit hook (installed by rad lfs init) collects every staged LFS-tracked file's oid/size and passes them all to a single rad lfs precommit call, which adds each one to your local IPFS daemon, pins it, encrypts it first if the repo is private, and records the resulting CID as a git note (batched into one call, rather than one per file, so a private repo's passphrase is only prompted for once per commit). rad lfs init configures the rad remote so a single git push rad pushes your commit and its notes mapping together (see radicle-heartwood-lfs's LFS-IPFS.md for the full explanation) — as long as it's been (re-)run with a build that includes that fix.
  • rad-lfs-transfer implements the other half: when git lfs push/pull runs, it's invoked per-object over stdin/stdout JSON (the standard Git LFS custom-transfer-agent protocol) and:
    • upload: shells out to rad lfs store --oid <oid> --size <size> <path>, which is authoritative and idempotent — it always ends up added, pinned, and noted correctly, whether or not the pre-commit hook already did it (e.g. if it was bypassed with --no-verify).
    • download: shells out to rad lfs fetch --oid <oid> --size <size> --out <path>, which looks up the note, fetches from IPFS (pulling from the wider network if not local yet), decrypts if needed, and writes the plaintext to <path>.

There is no central server anywhere in this design — see radicle-heartwood-lfs's LFS-IPFS.md for the full rationale and how rad seed/rad unseed tie IPFS pinning to Radicle's existing seeding lifecycle.

Dependencies

On Arch Linux:

# To build and install
sudo pacman -S --needed rust

# To actually use it (not needed to build/install — see "Requirements" below)
sudo pacman -S --needed kubo

On other distributions, install a Rust toolchain (e.g. via rustup) — and, only when you want to use LFS, not to build this — Kubo.

Requirements

None of these are needed to build or install this binary — only at runtime, when git lfs push/pull actually invokes it:

  • The rad binary (built from radicle-heartwood-lfs) on your PATH, with the repository already set up via rad lfs init. rad-lfs-transfer shells out to rad lfs store/rad lfs fetch for every object — it does not talk to git notes or IPFS directly itself anymore (that's all inside those subcommands now), and it has no dependency on git either.
  • A local IPFS (Kubo) daemon reachable at http://127.0.0.1:5001 (or wherever KUBO_API_URL points) — used both by rad lfs store/fetch themselves and by this binary's own startup healthcheck on the init event, so a daemon-not-running failure is reported clearly up front (ipfs daemon) rather than confusingly mid-transfer.
  • Git LFS installed (git-lfs on your PATH) — git-lfs package on Arch (sudo pacman -S --needed git-lfs).

Build & install

git clone ssh://git@git.hswro.org:9022/mab122/radicle-lfs-transfer.git
cd radicle-lfs-transfer
cargo install --path .

This installs the rad-lfs-transfer binary to ~/.cargo/bin by default — make sure that's on your PATH. If you're installing it alongside radicle-heartwood-lfs (the usual case), install it to that project's shared root instead so everything ends up in one place:

cargo install --path . --force --locked --root ~/.radicle

(You normally don't need to clone this repo separately at all for that — radicle-heartwood-lfs includes it as a git submodule and its own setup instructions cover installing both together.)

Configuration

Environment variable Default Purpose
KUBO_API_URL http://127.0.0.1:5001 Base URL of your local Kubo HTTP RPC API

You normally don't need to set this yourself — rad lfs init wires up everything else (the transfer-agent config, the pre-commit hook, and the notes-ref refspecs) automatically.

License

Licensed under either of Apache License, Version 2.0 or MIT license at your option.