mab122
/
radicle-lfs-transfer
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
A Git LFS custom transfer agent that backs large file storage with IPFS, for use with Radicle repositories.
8 Commits 1 Branch 0 Tags 77 KiB
Rust 100%
Go to file
Maciek "mab122" Bator 3215f231d8 Fetch multiple LFS objects through one long-lived worker, not per-object 
download_inner used to spawn a fresh `rad lfs fetch` subprocess for
every object, each one independently loading the signer -- for a
private repo, checking out N files meant N separate passphrase
prompts (or N failures without RAD_PASSPHRASE). Same bug class the
pre-commit hook had before it was batched into `rad lfs precommit`,
just on the download side.

Fetch can't batch the same way: git-lfs's custom-transfer protocol
requests objects one at a time, so unlike the pre-commit hook (which
knows every staged file upfront), there's no point where the full
object set is known before starting. Instead, `fetch_worker::FetchWorker`
spawns `rad lfs fetch-batch` (new, radicle-heartwood-lfs) lazily on
first use and keeps it alive for the rest of the session, one request
per download, one JSON response line back -- so the signer is loaded
(and the passphrase prompted for) at most once per session.

If the worker process itself dies (broken pipe, unexpected exit) --
as opposed to a legitimate per-object failure like a missing note or
wrong passphrase, which isn't retried -- one respawn-and-retry is
attempted before giving up.
 		2026-07-17 23:42:17 +02:00
src Fetch multiple LFS objects through one long-lived worker, not per-object 2026-07-17 23:42:17 +02:00
.gitignore Initial commit: rad-lfs-transfer, a Git LFS custom transfer agent backed by IPFS 2026-07-14 12:29:36 +02:00
Cargo.lock Initial commit: rad-lfs-transfer, a Git LFS custom transfer agent backed by IPFS 2026-07-14 12:29:36 +02:00
Cargo.toml Initial commit: rad-lfs-transfer, a Git LFS custom transfer agent backed by IPFS 2026-07-14 12:29:36 +02:00
LICENSE-APACHE Initial commit: rad-lfs-transfer, a Git LFS custom transfer agent backed by IPFS 2026-07-14 12:29:36 +02:00
LICENSE-MIT Initial commit: rad-lfs-transfer, a Git LFS custom transfer agent backed by IPFS 2026-07-14 12:29:36 +02:00
README.md docs: single git push rad now pushes commit + notes together 2026-07-16 15:23:13 +02:00

README.md

radicle-lfs-transfer

A Git LFS custom transfer agent that backs large file storage with IPFS, for use with Radicle repositories.

This is the companion binary for the rad lfs support added in radicle-heartwood-lfs (a fork of Radicle's core node/CLI, heartwood). Most people want to start there, not here — that repo has the full zero-to-usable setup guide. This repo is what you're reading if you want to build/understand the transfer agent on its own, or you're following a link back from there.

rad-lfs-transfer is not meant to be invoked directly — git lfs spawns it automatically, once a repository has been set up with rad lfs init (from the fork above).

How it works

Every peer uses their own local IPFS node — there's no shared server. All of the actual IPFS add/pin work and git-notes bookkeeping lives in the rad lfs store/rad lfs fetch/ rad lfs precommit subcommands, implemented in the companion radicle-heartwood-lfs fork — that's also where content gets encrypted for private repositories before it ever reaches IPFS. This crate deliberately has no dependency on Radicle's identity/crypto stack; it just shells out:

  • On git add/git commit, a pre-commit hook (installed by rad lfs init) collects every staged LFS-tracked file's oid/size and passes them all to a single rad lfs precommit call, which adds each one to your local IPFS daemon, pins it, encrypts it first if the repo is private, and records the resulting CID as a git note (batched into one call, rather than one per file, so a private repo's passphrase is only prompted for once per commit). rad lfs init configures the rad remote so a single git push rad pushes your commit and its notes mapping together (see radicle-heartwood-lfs's LFS-IPFS.md for the full explanation) — as long as it's been (re-)run with a build that includes that fix.
  • rad-lfs-transfer implements the other half: when git lfs push/pull runs, it's invoked per-object over stdin/stdout JSON (the standard Git LFS custom-transfer-agent protocol) and:
    • upload: shells out to rad lfs store --oid <oid> --size <size> <path>, which is authoritative and idempotent — it always ends up added, pinned, and noted correctly, whether or not the pre-commit hook already did it (e.g. if it was bypassed with --no-verify).
    • download: shells out to rad lfs fetch --oid <oid> --size <size> --out <path>, which looks up the note, fetches from IPFS (pulling from the wider network if not local yet), decrypts if needed, and writes the plaintext to <path>.

There is no central server anywhere in this design — see radicle-heartwood-lfs's LFS-IPFS.md for the full rationale and how rad seed/rad unseed tie IPFS pinning to Radicle's existing seeding lifecycle.

Dependencies

On Arch Linux:

# To build and install
sudo pacman -S --needed rust

# To actually use it (not needed to build/install — see "Requirements" below)
sudo pacman -S --needed kubo

On other distributions, install a Rust toolchain (e.g. via rustup) — and, only when you want to use LFS, not to build this — Kubo.

Requirements

None of these are needed to build or install this binary — only at runtime, when git lfs push/pull actually invokes it:

  • The rad binary (built from radicle-heartwood-lfs) on your PATH, with the repository already set up via rad lfs init. rad-lfs-transfer shells out to rad lfs store/rad lfs fetch for every object — it does not talk to git notes or IPFS directly itself anymore (that's all inside those subcommands now), and it has no dependency on git either.
  • A local IPFS (Kubo) daemon reachable at http://127.0.0.1:5001 (or wherever KUBO_API_URL points) — used both by rad lfs store/fetch themselves and by this binary's own startup healthcheck on the init event, so a daemon-not-running failure is reported clearly up front (ipfs daemon) rather than confusingly mid-transfer.
  • Git LFS installed (git-lfs on your PATH) — git-lfs package on Arch (sudo pacman -S --needed git-lfs).

Build & install

git clone ssh://git@git.hswro.org:9022/mab122/radicle-lfs-transfer.git
cd radicle-lfs-transfer
cargo install --path .

This installs the rad-lfs-transfer binary to ~/.cargo/bin by default — make sure that's on your PATH. If you're installing it alongside radicle-heartwood-lfs (the usual case), install it to that project's shared root instead so everything ends up in one place:

cargo install --path . --force --locked --root ~/.radicle

(You normally don't need to clone this repo separately at all for that — radicle-heartwood-lfs includes it as a git submodule and its own setup instructions cover installing both together.)

Configuration

Environment variable Default Purpose
KUBO_API_URL http://127.0.0.1:5001 Base URL of your local Kubo HTTP RPC API

You normally don't need to set this yourself — rad lfs init wires up everything else (the transfer-agent config, the pre-commit hook, and the notes-ref refspecs) automatically.

License

Licensed under either of Apache License, Version 2.0 or MIT license at your option.