mab122
/
radicle-heartwood-lfs
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

radicle: improve sync fetching 

This patch changes separates the business logic of fetching from the process of
fetching itself. It does this through a sans-IO approach, where a `Fetcher`
provides the necessary state to help drive a fetch forward, without performing
any of the IO itself.

What the `Fetcher` cares about is:
- What nodes are going to be attempted to be fetched from
- In what order should they be attempted
- When should the fetching process be considered finished

The `Fetcher` is then used in `radicle-cli` to drive forward the `sync::fetch`
function, allowing it to only care about the IO.
This commit is contained in:
Fintan Halpenny 2025-05-16 08:39:16 +02:00
parent c205322c56
commit eec4dd4505
34 changed files with 1480 additions and 264 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

5
radicle-cli/examples/git/git-push-amend.md
View File

@ -5,8 +5,9 @@ c036c0d89ce26aef3ad7da402157dba16b5163b4
``` ~bob
$ rad sync --fetch
✓ Fetching rad:z42hL2jL4XNk6K8oHQaSWfMgCL7ji from z6MknSL…StBU8Vi@[..]..
✓ Fetched repository from 1 seed(s)
Fetching rad:z42hL2jL4XNk6K8oHQaSWfMgCL7ji from the network, found 1 potential seed(s).
✓ Target met: 1 seed(s)
🌱 Fetched from z6MknSL…StBU8Vi
```
``` ~alice

30
radicle-cli/examples/git/git-push-converge.md
View File

@ -14,16 +14,18 @@ responsibilities:
``` ~bob
$ rad sync --fetch
✓ Fetching rad:z42hL2jL4XNk6K8oHQaSWfMgCL7ji from z6Mkux1…nVhib7Z@[..]..
✓ Fetching rad:z42hL2jL4XNk6K8oHQaSWfMgCL7ji from z6MknSL…StBU8Vi@[..]..
✓ Fetched repository from 2 seed(s)
Fetching rad:z42hL2jL4XNk6K8oHQaSWfMgCL7ji from the network, found 2 potential seed(s).
✓ Target met: 2 seed(s)
🌱 Fetched from z6Mkux1…nVhib7Z
🌱 Fetched from z6MknSL…StBU8Vi
```
``` ~eve
$ rad sync --fetch
✓ Fetching rad:z42hL2jL4XNk6K8oHQaSWfMgCL7ji from z6Mkt67…v4N1tRk@[..]..
✓ Fetching rad:z42hL2jL4XNk6K8oHQaSWfMgCL7ji from z6MknSL…StBU8Vi@[..]..
✓ Fetched repository from 2 seed(s)
Fetching rad:z42hL2jL4XNk6K8oHQaSWfMgCL7ji from the network, found 2 potential seed(s).
✓ Target met: 2 seed(s)
🌱 Fetched from z6Mkt67…v4N1tRk
🌱 Fetched from z6MknSL…StBU8Vi
```
To demonstrate the divergence, Alice, Bob, and Eve will all create a new change,
@ -58,14 +60,14 @@ found` error is showing up:
``` ~alice
$ rad remote add did:key:z6Mkt67GdsW7715MEfRuP4pSZxJRJh6kj6Y48WRqVv4N1tRk --name bob
✓ Follow policy updated for z6Mkt67GdsW7715MEfRuP4pSZxJRJh6kj6Y48WRqVv4N1tRk (bob)
✓ Fetching rad:z42hL2jL4XNk6K8oHQaSWfMgCL7ji from z6Mkux1…nVhib7Z@[..]..
✓ Fetching rad:z42hL2jL4XNk6K8oHQaSWfMgCL7ji from z6Mkt67…v4N1tRk@[..]..
Fetching rad:z42hL2jL4XNk6K8oHQaSWfMgCL7ji from the network, found 2 potential seed(s).
✓ Target met: 2 seed(s)
✓ Remote bob added
✓ Remote-tracking branch bob/master created for z6Mkt67…v4N1tRk
$ rad remote add did:key:z6Mkux1aUQD2voWWukVb5nNUR7thrHveQG4pDQua8nVhib7Z --name eve
✓ Follow policy updated for z6Mkux1aUQD2voWWukVb5nNUR7thrHveQG4pDQua8nVhib7Z (eve)
✓ Fetching rad:z42hL2jL4XNk6K8oHQaSWfMgCL7ji from z6Mkux1…nVhib7Z@[..]..
✓ Fetching rad:z42hL2jL4XNk6K8oHQaSWfMgCL7ji from z6Mkt67…v4N1tRk@[..]..
Fetching rad:z42hL2jL4XNk6K8oHQaSWfMgCL7ji from the network, found 2 potential seed(s).
✓ Target met: 2 seed(s)
✓ Remote eve added
✓ Remote-tracking branch eve/master created for z6Mkux1…nVhib7Z
```
@ -100,8 +102,8 @@ To rad://z42hL2jL4XNk6K8oHQaSWfMgCL7ji/z6MknSLrJoTcukLrE435hVNQT4JUhbvWLX4kUzqkE
``` ~bob
$ rad remote add did:key:z6MknSLrJoTcukLrE435hVNQT4JUhbvWLX4kUzqkEStBU8Vi --name alice
✓ Follow policy updated for z6MknSLrJoTcukLrE435hVNQT4JUhbvWLX4kUzqkEStBU8Vi (alice)
✓ Fetching rad:z42hL2jL4XNk6K8oHQaSWfMgCL7ji from z6Mkux1…nVhib7Z@[..]..
✓ Fetching rad:z42hL2jL4XNk6K8oHQaSWfMgCL7ji from z6MknSL…StBU8Vi@[..]..
Fetching rad:z42hL2jL4XNk6K8oHQaSWfMgCL7ji from the network, found 2 potential seed(s).
✓ Target met: 2 seed(s)
✓ Remote alice added
✓ Remote-tracking branch alice/master created for z6MknSL…StBU8Vi
$ git reset --hard alice/master
@ -126,8 +128,8 @@ Once Eve also resets to the merge commits, the canonical `master` is set to this
``` ~eve
$ rad remote add did:key:z6MknSLrJoTcukLrE435hVNQT4JUhbvWLX4kUzqkEStBU8Vi --name alice
✓ Follow policy updated for z6MknSLrJoTcukLrE435hVNQT4JUhbvWLX4kUzqkEStBU8Vi (alice)
✓ Fetching rad:z42hL2jL4XNk6K8oHQaSWfMgCL7ji from z6Mkt67…v4N1tRk@[..]..
✓ Fetching rad:z42hL2jL4XNk6K8oHQaSWfMgCL7ji from z6MknSL…StBU8Vi@[..]..
Fetching rad:z42hL2jL4XNk6K8oHQaSWfMgCL7ji from the network, found 2 potential seed(s).
✓ Target met: 2 seed(s)
✓ Remote alice added
✓ Remote-tracking branch alice/master created for z6MknSL…StBU8Vi
$ git reset --hard alice/master

5
radicle-cli/examples/git/git-push-diverge.md
View File

@ -13,8 +13,9 @@ Then, as Bob, we commit some code on top of the canonical head:
``` ~bob
$ rad sync --fetch
✓ Fetching rad:z42hL2jL4XNk6K8oHQaSWfMgCL7ji from z6MknSL…StBU8Vi@[..]..
✓ Fetched repository from 1 seed(s)
Fetching rad:z42hL2jL4XNk6K8oHQaSWfMgCL7ji from the network, found 1 potential seed(s).
✓ Target met: 1 seed(s)
🌱 Fetched from z6MknSL…StBU8Vi
$ rad inspect --delegates
did:key:z6MknSLrJoTcukLrE435hVNQT4JUhbvWLX4kUzqkEStBU8Vi (alice)
did:key:z6Mkt67GdsW7715MEfRuP4pSZxJRJh6kj6Y48WRqVv4N1tRk (bob)

5
radicle-cli/examples/git/git-push-rollback.md
View File

@ -12,8 +12,9 @@ Bob then syncs these changes and adds a new commit:
``` ~bob
$ rad sync --fetch
✓ Fetching rad:z42hL2jL4XNk6K8oHQaSWfMgCL7ji from z6MknSL…StBU8Vi@[..]..
✓ Fetched repository from 1 seed(s)
Fetching rad:z42hL2jL4XNk6K8oHQaSWfMgCL7ji from the network, found 1 potential seed(s).
✓ Target met: 1 seed(s)
🌱 Fetched from z6MknSL…StBU8Vi
$ git commit -m "Third commit" --allow-empty -q
$ git push rad
$ git branch -arv

8
radicle-cli/examples/git/git-tag.md
View File

@ -37,7 +37,8 @@ Bob fetches the tag from Alice, by adding her as a remote:
$ cd heartwood
$ rad remote add z6MknSLrJoTcukLrE435hVNQT4JUhbvWLX4kUzqkEStBU8Vi --name alice
✓ Follow policy updated for z6MknSLrJoTcukLrE435hVNQT4JUhbvWLX4kUzqkEStBU8Vi (alice)
✓ Fetching rad:z42hL2jL4XNk6K8oHQaSWfMgCL7ji from z6MknSL…StBU8Vi@[..]..
Fetching rad:z42hL2jL4XNk6K8oHQaSWfMgCL7ji from the network, found 1 potential seed(s).
✓ Target met: 1 seed(s)
✓ Remote alice added
✓ Remote-tracking branch alice/master created for z6MknSL…StBU8Vi
```
@ -73,8 +74,9 @@ update of the tag:
``` ~bob
$ rad sync -f
✓ Fetching rad:z42hL2jL4XNk6K8oHQaSWfMgCL7ji from z6MknSL…StBU8Vi@[..]..
✓ Fetched repository from 1 seed(s)
Fetching rad:z42hL2jL4XNk6K8oHQaSWfMgCL7ji from the network, found 1 potential seed(s).
✓ Target met: 1 seed(s)
🌱 Fetched from z6MknSL…StBU8Vi
```
``` ~bob (stderr)

3
radicle-cli/examples/rad-clone-all.md
View File

@ -1,7 +1,8 @@
```
$ rad clone rad:z42hL2jL4XNk6K8oHQaSWfMgCL7ji --scope all
✓ Seeding policy updated for rad:z42hL2jL4XNk6K8oHQaSWfMgCL7ji with scope 'all'
✓ Fetching rad:z42hL2jL4XNk6K8oHQaSWfMgCL7ji from z6MknSL…StBU8Vi@[..]..
Fetching rad:z42hL2jL4XNk6K8oHQaSWfMgCL7ji from the network, found 1 potential seed(s).
✓ Target met: 1 seed(s)
✓ Creating checkout in ./heartwood..
✓ Remote alice@z6MknSLrJoTcukLrE435hVNQT4JUhbvWLX4kUzqkEStBU8Vi added
✓ Remote-tracking branch alice@z6MknSLrJoTcukLrE435hVNQT4JUhbvWLX4kUzqkEStBU8Vi/master created for z6MknSL…StBU8Vi

6
radicle-cli/examples/rad-clone-connect.md
View File

@ -4,10 +4,8 @@ automatically connect to the necessary seeds.
```
$ rad clone rad:z42hL2jL4XNk6K8oHQaSWfMgCL7ji
✓ Seeding policy updated for rad:z42hL2jL4XNk6K8oHQaSWfMgCL7ji with scope 'all'
✓ Connecting to z6Mkt67…v4N1tRk@[..]
✓ Fetching rad:z42hL2jL4XNk6K8oHQaSWfMgCL7ji from z6Mkt67…v4N1tRk@[..]..
✓ Connecting to z6MknSL…StBU8Vi@[..]
✓ Fetching rad:z42hL2jL4XNk6K8oHQaSWfMgCL7ji from z6MknSL…StBU8Vi@[..]..
Fetching rad:z42hL2jL4XNk6K8oHQaSWfMgCL7ji from the network, found 2 potential seed(s).
✓ Target met: 2 seed(s)
✓ Creating checkout in ./heartwood..
✓ Remote alice@z6MknSLrJoTcukLrE435hVNQT4JUhbvWLX4kUzqkEStBU8Vi added
✓ Remote-tracking branch alice@z6MknSLrJoTcukLrE435hVNQT4JUhbvWLX4kUzqkEStBU8Vi/master created for z6MknSL…StBU8Vi

6
radicle-cli/examples/rad-clone-directory.md
View File

@ -4,7 +4,8 @@ by specifying the directory in the `rad clone` command:
```
$ rad clone rad:z42hL2jL4XNk6K8oHQaSWfMgCL7ji --scope followed Developer/Radicle
✓ Seeding policy updated for rad:z42hL2jL4XNk6K8oHQaSWfMgCL7ji with scope 'followed'
✓ Fetching rad:z42hL2jL4XNk6K8oHQaSWfMgCL7ji from z6MknSL…StBU8Vi@[..]..
Fetching rad:z42hL2jL4XNk6K8oHQaSWfMgCL7ji from the network, found 1 potential seed(s).
✓ Target met: 1 seed(s)
✓ Creating checkout in ./Developer/Radicle..
✓ Remote alice@z6MknSLrJoTcukLrE435hVNQT4JUhbvWLX4kUzqkEStBU8Vi added
✓ Remote-tracking branch alice@z6MknSLrJoTcukLrE435hVNQT4JUhbvWLX4kUzqkEStBU8Vi/master created for z6MknSL…StBU8Vi
@ -22,6 +23,7 @@ and is not empty, will fail:
``` (fail)
$ rad clone rad:z42hL2jL4XNk6K8oHQaSWfMgCL7ji --scope followed Developer/Radicle
✓ Fetching rad:z42hL2jL4XNk6K8oHQaSWfMgCL7ji from z6MknSL…StBU8Vi@[..]..
Fetching rad:z42hL2jL4XNk6K8oHQaSWfMgCL7ji from the network, found 1 potential seed(s).
✓ Target met: 1 seed(s)
✗ Error: the directory path "Developer/Radicle" already exists
```

5
radicle-cli/examples/rad-clone-partial-fail.md
View File

@ -16,9 +16,8 @@ still returns successfully.
```
$ rad clone rad:z42hL2jL4XNk6K8oHQaSWfMgCL7ji --timeout 3
✓ Seeding policy updated for rad:z42hL2jL4XNk6K8oHQaSWfMgCL7ji with scope 'all'
✗ Fetching rad:z42hL2jL4XNk6K8oHQaSWfMgCL7ji from z6Mkt67…v4N1tRk@[..].. error: failed to perform fetch handshake
✓ Fetching rad:z42hL2jL4XNk6K8oHQaSWfMgCL7ji from z6MknSL…StBU8Vi@[..]..
✗ Connecting to z6MksFq…bS9wzpT@[..].. error: [..]
Fetching rad:z42hL2jL4XNk6K8oHQaSWfMgCL7ji from the network, found 3 potential seed(s).
✗ Target not met: required 2 more seed(s)
✓ Creating checkout in ./heartwood..
✓ Remote alice@z6MknSLrJoTcukLrE435hVNQT4JUhbvWLX4kUzqkEStBU8Vi added
✓ Remote-tracking branch alice@z6MknSLrJoTcukLrE435hVNQT4JUhbvWLX4kUzqkEStBU8Vi/master created for z6MknSL…StBU8Vi

2
radicle-cli/examples/rad-clone-unknown.md
View File

@ -3,5 +3,5 @@ Trying to clone a repository that is not in our routing table returns an error:
``` (fail)
$ rad clone rad:zVNuptPuk5XauitpCWSNVCXGGfXW --scope followed
✓ Seeding policy updated for rad:zVNuptPuk5XauitpCWSNVCXGGfXW with scope 'followed'
✗ Error: no seeds found for rad:zVNuptPuk5XauitpCWSNVCXGGfXW
✗ Error: fetch: no candidate seeds were found to fetch from
```

3
radicle-cli/examples/rad-clone.md
View File

@ -4,7 +4,8 @@ To create a local copy of a repository on the radicle network, we use the
```
$ rad clone rad:z42hL2jL4XNk6K8oHQaSWfMgCL7ji --scope followed
✓ Seeding policy updated for rad:z42hL2jL4XNk6K8oHQaSWfMgCL7ji with scope 'followed'
✓ Fetching rad:z42hL2jL4XNk6K8oHQaSWfMgCL7ji from z6MknSL…StBU8Vi@[..]..
Fetching rad:z42hL2jL4XNk6K8oHQaSWfMgCL7ji from the network, found [..] potential seed(s).
[..]
✓ Creating checkout in ./heartwood..
✓ Remote alice@z6MknSLrJoTcukLrE435hVNQT4JUhbvWLX4kUzqkEStBU8Vi added
✓ Remote-tracking branch alice@z6MknSLrJoTcukLrE435hVNQT4JUhbvWLX4kUzqkEStBU8Vi/master created for z6MknSL…StBU8Vi

5
radicle-cli/examples/rad-fetch.md
View File

@ -19,8 +19,9 @@ by passing the `--fetch` option.
```
$ rad sync --fetch rad:z42hL2jL4XNk6K8oHQaSWfMgCL7ji
✓ Fetching rad:z42hL2jL4XNk6K8oHQaSWfMgCL7ji from z6MknSL…StBU8Vi@[..]..
✓ Fetched repository from 1 seed(s)
Fetching rad:z42hL2jL4XNk6K8oHQaSWfMgCL7ji from the network, found 1 potential seed(s).
✓ Target met: 1 seed(s)
🌱 Fetched from z6MknSL…StBU8Vi
```
However, we don't have a local fork of the project. We can follow this

15
radicle-cli/examples/rad-id-conflict.md
View File

@ -7,8 +7,9 @@ $ rad id update --title "Add Bob" --description "Add Bob as a delegate" --delega
``` ~bob
$ cd heartwood
$ rad sync --fetch rad:z42hL2jL4XNk6K8oHQaSWfMgCL7ji
✓ Fetching rad:z42hL2jL4XNk6K8oHQaSWfMgCL7ji from z6MknSL…StBU8Vi@[..]..
✓ Fetched repository from 1 seed(s)
Fetching rad:z42hL2jL4XNk6K8oHQaSWfMgCL7ji from the network, found 1 potential seed(s).
✓ Target met: 1 seed(s)
🌱 Fetched from z6MknSL…StBU8Vi
```
One thing that can happen is that two delegates propose a revision at the same
@ -28,8 +29,9 @@ revisions.
``` ~alice
$ rad sync --fetch rad:z42hL2jL4XNk6K8oHQaSWfMgCL7ji
✓ Fetching rad:z42hL2jL4XNk6K8oHQaSWfMgCL7ji from z6Mkt67…v4N1tRk@[..]..
✓ Fetched repository from 1 seed(s)
Fetching rad:z42hL2jL4XNk6K8oHQaSWfMgCL7ji from the network, found 1 potential seed(s).
✓ Target met: 1 seed(s)
🌱 Fetched from z6Mkt67…v4N1tRk
$ rad id list
╭─────────────────────────────────────────────────────────────────────────────────╮
│ ● ID Title Author Status Created │
@ -62,8 +64,9 @@ accepted now.
``` ~bob
$ rad sync --fetch rad:z42hL2jL4XNk6K8oHQaSWfMgCL7ji
✓ Fetching rad:z42hL2jL4XNk6K8oHQaSWfMgCL7ji from z6MknSL…StBU8Vi@[..]..
✓ Fetched repository from 1 seed(s)
Fetching rad:z42hL2jL4XNk6K8oHQaSWfMgCL7ji from the network, found 1 potential seed(s).
✓ Target met: 1 seed(s)
🌱 Fetched from z6MknSL…StBU8Vi
```
``` ~bob (fail)
$ rad id accept 12d7300 -q

12
radicle-cli/examples/rad-id-multi-delegate.md
View File

@ -6,8 +6,9 @@ $ rad id update --repo rad:z42hL2jL4XNk6K8oHQaSWfMgCL7ji --title "Add Bob" --des
``` ~bob
$ rad watch --repo rad:z42hL2jL4XNk6K8oHQaSWfMgCL7ji --node z6Mkux1aUQD2voWWukVb5nNUR7thrHveQG4pDQua8nVhib7Z -r 'refs/rad/sigrefs' -t c9a828fc2fb01f893d6e6e9e17b9092dea2b3aba -i 500 --timeout 5000
$ rad sync --fetch rad:z42hL2jL4XNk6K8oHQaSWfMgCL7ji
✓ Fetching rad:z42hL2jL4XNk6K8oHQaSWfMgCL7ji from z6MknSL…StBU8Vi@[..]..
✓ Fetched repository from 1 seed(s)
Fetching rad:z42hL2jL4XNk6K8oHQaSWfMgCL7ji from the network, found 1 potential seed(s).
✓ Target met: 1 seed(s)
🌱 Fetched from z6MknSL…StBU8Vi
$ rad id --repo rad:z42hL2jL4XNk6K8oHQaSWfMgCL7ji
╭────────────────────────────────────────────────────────────────────────────────╮
│ ● ID Title Author Status Created │
@ -57,9 +58,10 @@ $ rad id update --repo rad:z42hL2jL4XNk6K8oHQaSWfMgCL7ji --title "Add Eve" --des
``` ~alice
$ rad sync --fetch rad:z42hL2jL4XNk6K8oHQaSWfMgCL7ji
✓ Fetching rad:z42hL2jL4XNk6K8oHQaSWfMgCL7ji from z6Mkux1…nVhib7Z@[..]..
✓ Fetching rad:z42hL2jL4XNk6K8oHQaSWfMgCL7ji from z6Mkt67…v4N1tRk@[..]..
✓ Fetched repository from 2 seed(s)
Fetching rad:z42hL2jL4XNk6K8oHQaSWfMgCL7ji from the network, found 2 potential seed(s).
✓ Target met: 2 seed(s)
🌱 Fetched from z6Mkux1…nVhib7Z
🌱 Fetched from z6Mkt67…v4N1tRk
$ rad inspect rad:z42hL2jL4XNk6K8oHQaSWfMgCL7ji --delegates
did:key:z6MknSLrJoTcukLrE435hVNQT4JUhbvWLX4kUzqkEStBU8Vi (alice)
did:key:z6Mkt67GdsW7715MEfRuP4pSZxJRJh6kj6Y48WRqVv4N1tRk (bob)

16
radicle-cli/examples/rad-id-threshold.md
View File

@ -96,8 +96,9 @@ errors:
``` ~seed
$ rad sync rad:z42hL2jL4XNk6K8oHQaSWfMgCL7ji -f
✓ Fetching rad:z42hL2jL4XNk6K8oHQaSWfMgCL7ji from z6MknSL…StBU8Vi@[..]..
✓ Fetched repository from 1 seed(s)
Fetching rad:z42hL2jL4XNk6K8oHQaSWfMgCL7ji from the network, found 1 potential seed(s).
✓ Target met: 1 seed(s)
🌱 Fetched from z6MknSL…StBU8Vi
```
We can also inspect the repository to ensure all the data is
@ -170,8 +171,8 @@ sync` and fetch his references:
``` ~bob
$ rad clone rad:z42hL2jL4XNk6K8oHQaSWfMgCL7ji
✓ Seeding policy updated for rad:z42hL2jL4XNk6K8oHQaSWfMgCL7ji with scope 'all'
✓ Fetching rad:z42hL2jL4XNk6K8oHQaSWfMgCL7ji from z6MknSL…StBU8Vi@[..]..
✓ Fetching rad:z42hL2jL4XNk6K8oHQaSWfMgCL7ji from z6Mkux1…nVhib7Z@[..]..
Fetching rad:z42hL2jL4XNk6K8oHQaSWfMgCL7ji from the network, found 2 potential seed(s).
✓ Target met: 2 seed(s)
✓ Creating checkout in ./heartwood..
✓ Remote alice@z6MknSLrJoTcukLrE435hVNQT4JUhbvWLX4kUzqkEStBU8Vi added
✓ Remote-tracking branch alice@z6MknSLrJoTcukLrE435hVNQT4JUhbvWLX4kUzqkEStBU8Vi/master created for z6MknSL…StBU8Vi
@ -192,9 +193,10 @@ $ rad fork
``` ~alice
$ rad sync -f
✓ Fetching rad:z42hL2jL4XNk6K8oHQaSWfMgCL7ji from z6Mkux1…nVhib7Z@[..]..
✓ Fetching rad:z42hL2jL4XNk6K8oHQaSWfMgCL7ji from z6Mkt67…v4N1tRk@[..]..
✓ Fetched repository from 2 seed(s)
Fetching rad:z42hL2jL4XNk6K8oHQaSWfMgCL7ji from the network, found 2 potential seed(s).
✓ Target met: 2 seed(s)
🌱 Fetched from z6Mkux1…nVhib7Z
🌱 Fetched from z6Mkt67…v4N1tRk
$ rad inspect --sigrefs
z6MknSLrJoTcukLrE435hVNQT4JUhbvWLX4kUzqkEStBU8Vi ae3c6b77dc1ed51c1c1e6a2772339c2779fa9ba8
z6Mkt67GdsW7715MEfRuP4pSZxJRJh6kj6Y48WRqVv4N1tRk dace6fe948548168a2bb687718949d9b5d9076ee

6
radicle-cli/examples/rad-init-private-clone-seed.md
View File

@ -31,7 +31,8 @@ $ rad inspect --identity
$ rad ls --all --private
$ rad clone rad:z2ug5mwNKZB8KGpBDRTrWHAMbvHCu --seed z6MknSLrJoTcukLrE435hVNQT4JUhbvWLX4kUzqkEStBU8Vi --timeout 1
✓ Seeding policy updated for rad:z2ug5mwNKZB8KGpBDRTrWHAMbvHCu with scope 'all'
✓ Fetching rad:z2ug5mwNKZB8KGpBDRTrWHAMbvHCu from z6MknSL…StBU8Vi@[..]..
Fetching rad:z2ug5mwNKZB8KGpBDRTrWHAMbvHCu from the network, found 1 potential seed(s).
✓ Target met: 1 preferred seed(s).
✓ Creating checkout in ./heartwood..
✓ Remote alice@z6MknSLrJoTcukLrE435hVNQT4JUhbvWLX4kUzqkEStBU8Vi added
✓ Remote-tracking branch alice@z6MknSLrJoTcukLrE435hVNQT4JUhbvWLX4kUzqkEStBU8Vi/master created for z6MknSL…StBU8Vi
@ -49,5 +50,6 @@ We can also use `rad seed` to seed and fetch without creating a checkout.
``` ~bob
$ rad seed rad:z2ug5mwNKZB8KGpBDRTrWHAMbvHCu --from z6MknSLrJoTcukLrE435hVNQT4JUhbvWLX4kUzqkEStBU8Vi --timeout 1
✓ Seeding policy exists for rad:z2ug5mwNKZB8KGpBDRTrWHAMbvHCu with scope 'all'
✓ Fetching rad:z2ug5mwNKZB8KGpBDRTrWHAMbvHCu from z6MknSL…StBU8Vi@[..]..
Fetching rad:z2ug5mwNKZB8KGpBDRTrWHAMbvHCu from the network, found 1 potential seed(s).
✓ Target met: 1 seed(s)
```

8
radicle-cli/examples/rad-init-private-clone.md
View File

@ -7,7 +7,8 @@ $ rad ls
``` ~bob (fail)
$ rad clone rad:z2ug5mwNKZB8KGpBDRTrWHAMbvHCu --seed z6MknSLrJoTcukLrE435hVNQT4JUhbvWLX4kUzqkEStBU8Vi --timeout 1
✓ Seeding policy updated for rad:z2ug5mwNKZB8KGpBDRTrWHAMbvHCu with scope 'all'
✗ Fetching rad:z2ug5mwNKZB8KGpBDRTrWHAMbvHCu from z6MknSL…StBU8Vi@[..].. error: failed to perform fetch handshake
Fetching rad:z2ug5mwNKZB8KGpBDRTrWHAMbvHCu from the network, found 1 potential seed(s).
✗ Target not met: could not fetch from [z6MknSL…StBU8Vi], and required 1 more seed(s)
✗ Error: repository rad:z2ug5mwNKZB8KGpBDRTrWHAMbvHCu not found
```
@ -26,8 +27,9 @@ that Alice has the repo after she announced her refs:
``` ~bob
$ rad sync rad:z2ug5mwNKZB8KGpBDRTrWHAMbvHCu --fetch
✓ Fetching rad:z2ug5mwNKZB8KGpBDRTrWHAMbvHCu from z6MknSL…StBU8Vi@[..]..
✓ Fetched repository from 1 seed(s)
Fetching rad:z2ug5mwNKZB8KGpBDRTrWHAMbvHCu from the network, found 1 potential seed(s).
✓ Target met: 1 seed(s)
🌱 Fetched from z6MknSL…StBU8Vi
$ rad ls --private --all
╭───────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ Name RID Visibility Head Description │

15
radicle-cli/examples/rad-init-private-seed.md
View File

@ -15,17 +15,18 @@ $ rad seed rad:z2ug5mwNKZB8KGpBDRTrWHAMbvHCu --no-fetch
If Bob just tries to fetch it without specifying seeds, he gets an error:
``` ~bob
``` ~bob (fails)
$ rad sync rad:z2ug5mwNKZB8KGpBDRTrWHAMbvHCu --fetch
✗ Error: no seeds found for rad:z2ug5mwNKZB8KGpBDRTrWHAMbvHCu
✗ Error: no candidate seeds were found to fetch from
```
He has to specify a seed that isn't in his routing table:
``` ~bob
$ rad sync rad:z2ug5mwNKZB8KGpBDRTrWHAMbvHCu --fetch --seed z6MknSLrJoTcukLrE435hVNQT4JUhbvWLX4kUzqkEStBU8Vi
✓ Fetching rad:z2ug5mwNKZB8KGpBDRTrWHAMbvHCu from z6MknSL…StBU8Vi@[..]..
✓ Fetched repository from 1 seed(s)
Fetching rad:z2ug5mwNKZB8KGpBDRTrWHAMbvHCu from the network, found 1 potential seed(s).
✓ Target met: 1 preferred seed(s).
🌱 Fetched from z6MknSL…StBU8Vi
```
``` ~bob
@ -42,7 +43,7 @@ seed succeeds.
``` ~bob
$ rad sync rad:z2ug5mwNKZB8KGpBDRTrWHAMbvHCu --fetch --seed z6MknSLrJoTcukLrE435hVNQT4JUhbvWLX4kUzqkEStBU8Vi --seed z6MkwPUeUS2fJMfc2HZN1RQTQcTTuhw4HhPySB8JeUg2mVvx
✓ Fetching rad:z2ug5mwNKZB8KGpBDRTrWHAMbvHCu from z6MknSL…StBU8Vi@[..]..
! Warning: no addresses found for z6MkwPUeUS2fJMfc2HZN1RQTQcTTuhw4HhPySB8JeUg2mVvx, skipping..
✓ Fetched repository from 1 seed(s)
Fetching rad:z2ug5mwNKZB8KGpBDRTrWHAMbvHCu from the network, found 1 potential seed(s).
✓ Target met: 1 seed(s)
🌱 Fetched from z6MknSL…StBU8Vi
```

5
radicle-cli/examples/rad-patch-checkout-force.md
View File

@ -24,8 +24,9 @@ On the other end, Bob uses `rad patch checkout` to view the patch:
``` ~bob
$ cd heartwood
$ rad sync -f
✓ Fetching rad:z42hL2jL4XNk6K8oHQaSWfMgCL7ji from z6MknSL…StBU8Vi@[..]..
✓ Fetched repository from 1 seed(s)
Fetching rad:z42hL2jL4XNk6K8oHQaSWfMgCL7ji from the network, found 1 potential seed(s).
✓ Target met: 1 seed(s)
🌱 Fetched from z6MknSL…StBU8Vi
$ rad patch checkout aa45913 --name alice-init
✓ Switched to branch alice-init at revision aa45913
✓ Branch alice-init setup to track rad/patches/aa45913e757cacd46972733bddee5472c78fa32a

6
radicle-cli/examples/rad-patch-delete.md
View File

@ -19,8 +19,10 @@ To rad://z42hL2jL4XNk6K8oHQaSWfMgCL7ji/z6MknSLrJoTcukLrE435hVNQT4JUhbvWLX4kUzqkE
``` ~bob
$ cd heartwood
$ rad sync -f
✓ Fetching rad:z42hL2jL4XNk6K8oHQaSWfMgCL7ji from z6MknSL…StBU8Vi@[..]..
✓ Fetched repository from 1 seed(s)
Fetching rad:z42hL2jL4XNk6K8oHQaSWfMgCL7ji from the network, found 2 potential seed(s).
✓ Target met: 2 seed(s)
🌱 Fetched from z6MknSL…StBU8Vi
🌱 Fetched from z6Mkux1…nVhib7Z
$ rad patch comment 6c61ef1 -m "I think we should use MIT"
╭───────────────────────────╮
│ bob (you) now 833db19 │

3
radicle-cli/examples/rad-patch-pull-update.md
View File

@ -23,7 +23,8 @@ To push changes, run `git push`.
``` ~bob
$ rad clone rad:zhbMU4DUXrzB8xT6qAJh6yZ7bFMK
✓ Seeding policy updated for rad:zhbMU4DUXrzB8xT6qAJh6yZ7bFMK with scope 'all'
✓ Fetching rad:zhbMU4DUXrzB8xT6qAJh6yZ7bFMK from z6MknSL…StBU8Vi@[..]..
Fetching rad:zhbMU4DUXrzB8xT6qAJh6yZ7bFMK from the network, found 1 potential seed(s).
✓ Target met: 1 seed(s)
✓ Creating checkout in ./heartwood..
✓ Remote alice@z6MknSLrJoTcukLrE435hVNQT4JUhbvWLX4kUzqkEStBU8Vi added
✓ Remote-tracking branch alice@z6MknSLrJoTcukLrE435hVNQT4JUhbvWLX4kUzqkEStBU8Vi/master created for z6MknSL…StBU8Vi

3
radicle-cli/examples/rad-push-and-pull-patches.md
View File

@ -17,7 +17,8 @@ $ rad patch checkout d004b67
✓ Branch patch/d004b67 setup to track rad/patches/d004b67355456c46de10c0d287e4a791ad1a6945
$ rad remote add z6Mkt67GdsW7715MEfRuP4pSZxJRJh6kj6Y48WRqVv4N1tRk
✓ Follow policy updated for z6Mkt67GdsW7715MEfRuP4pSZxJRJh6kj6Y48WRqVv4N1tRk
✓ Fetching rad:z42hL2jL4XNk6K8oHQaSWfMgCL7ji from z6Mkt67…v4N1tRk@[..]..
Fetching rad:z42hL2jL4XNk6K8oHQaSWfMgCL7ji from the network, found 1 potential seed(s).
✓ Target met: 1 seed(s)
✓ Remote bob@z6Mkt67GdsW7715MEfRuP4pSZxJRJh6kj6Y48WRqVv4N1tRk added
✓ Remote-tracking branch bob@z6Mkt67GdsW7715MEfRuP4pSZxJRJh6kj6Y48WRqVv4N1tRk/master created for z6Mkt67…v4N1tRk
$ git checkout master -q

6
radicle-cli/examples/rad-seed-many.md
View File

@ -5,7 +5,9 @@ is used):
```
$ rad seed rad:z3Rry7rpdWuGpfjPYGzdJKQADsoNW rad:z3zTnCfi6cVSZG8eCGn6AMDypgAPm
✓ Seeding policy updated for rad:z3Rry7rpdWuGpfjPYGzdJKQADsoNW with scope 'all'
✓ Fetching rad:z3Rry7rpdWuGpfjPYGzdJKQADsoNW from z6Mkt67…v4N1tRk@[..]
Fetching rad:z3Rry7rpdWuGpfjPYGzdJKQADsoNW from the network, found 1 potential seed(s).
✓ Target met: 1 seed(s)
✓ Seeding policy updated for rad:z3zTnCfi6cVSZG8eCGn6AMDypgAPm with scope 'all'
✓ Fetching rad:z3zTnCfi6cVSZG8eCGn6AMDypgAPm from z6Mkt67…v4N1tRk@[..]
Fetching rad:z3zTnCfi6cVSZG8eCGn6AMDypgAPm from the network, found 1 potential seed(s).
✓ Target met: 1 seed(s)
```

36
radicle-cli/examples/rad-sync.md
View File

@ -55,18 +55,20 @@ We can also use the `--fetch` option to only fetch objects:
```
$ rad sync --fetch
✓ Fetching rad:z42hL2jL4XNk6K8oHQaSWfMgCL7ji from z6Mkux1…nVhib7Z@[..]..
✓ Fetching rad:z42hL2jL4XNk6K8oHQaSWfMgCL7ji from z6Mkt67…v4N1tRk@[..]..
✓ Fetched repository from 2 seed(s)
Fetching rad:z42hL2jL4XNk6K8oHQaSWfMgCL7ji from the network, found 2 potential seed(s).
✓ Target met: 2 seed(s)
🌱 Fetched from z6Mkux1…nVhib7Z
🌱 Fetched from z6Mkt67…v4N1tRk
```
Specifying both `--fetch` and `--announce` is equivalent to specifying none:
```
$ rad sync --fetch --announce
✓ Fetching rad:z42hL2jL4XNk6K8oHQaSWfMgCL7ji from z6Mkux1…nVhib7Z@[..]..
✓ Fetching rad:z42hL2jL4XNk6K8oHQaSWfMgCL7ji from z6Mkt67…v4N1tRk@[..]..
✓ Fetched repository from 2 seed(s)
Fetching rad:z42hL2jL4XNk6K8oHQaSWfMgCL7ji from the network, found 2 potential seed(s).
✓ Target met: 2 seed(s)
🌱 Fetched from z6Mkux1…nVhib7Z
🌱 Fetched from z6Mkt67…v4N1tRk
✓ Nothing to announce, already in sync with 2 node(s) (see `rad sync status`)
```
@ -74,8 +76,9 @@ It's also possible to use the `--seed` flag to only sync with a specific node:
```
$ rad sync --fetch --seed z6Mkt67GdsW7715MEfRuP4pSZxJRJh6kj6Y48WRqVv4N1tRk
✓ Fetching rad:z42hL2jL4XNk6K8oHQaSWfMgCL7ji from z6Mkt67…v4N1tRk@[..]..
✓ Fetched repository from 1 seed(s)
Fetching rad:z42hL2jL4XNk6K8oHQaSWfMgCL7ji from the network, found 3 potential seed(s).
✓ Target met: 1 preferred seed(s).
🌱 Fetched from z6Mkt67…v4N1tRk
```
And the `--replicas` flag to sync with a number of nodes. First we'll
@ -87,8 +90,9 @@ $ rad issue open --title "Test `rad sync --replicas`" --description "Check that
```
$ rad sync --replicas 1
✓ Fetching rad:z42hL2jL4XNk6K8oHQaSWfMgCL7ji from z6Mkux1…nVhib7Z@[..]..
✓ Fetched repository from 1 seed(s)
Fetching rad:z42hL2jL4XNk6K8oHQaSWfMgCL7ji from the network, found 2 potential seed(s).
✓ Target met: 1 seed(s)
🌱 Fetched from z6Mkux1…nVhib7Z
✓ Synced with 1 node(s)
```
@ -102,20 +106,22 @@ wait for at least 1 of the nodes to have fetched the changes from us.
It's also possible to receive an error if a repository is not found anywhere.
```
$ rad seed rad:z39mP9rQAaGmERfUMPULfPUi473tY
$ rad seed rad:z39mP9rQAaGmERfUMPULfPUi473tY --no-fetch
✓ Seeding policy updated for rad:z39mP9rQAaGmERfUMPULfPUi473tY with scope 'all'
```
``` (fail)
$ rad sync rad:z39mP9rQAaGmERfUMPULfPUi473tY
✗ Error: no seeds found for rad:z39mP9rQAaGmERfUMPULfPUi473tY
✗ Error: nothing to announce, repository rad:z39mP9rQAaGmERfUMPULfPUi473tY is not available locally
✗ Error: no candidate seeds were found to fetch from
```
Or when trying to fetch from an unknown seed, using `--seed`:
```
$ rad sync --fetch rad:z39mP9rQAaGmERfUMPULfPUi473tY --seed z6MkjM3HpqNVV4ZsL5s3RAd8ThVG3VG98YsDCjHBNnGMq5o7
! Warning: no addresses found for z6MkjM3HpqNVV4ZsL5s3RAd8ThVG3VG98YsDCjHBNnGMq5o7, skipping..
✗ Error: repository fetch from 1 seed(s) failed
Fetching rad:z39mP9rQAaGmERfUMPULfPUi473tY from the network, found 1 potential seed(s).
✗ Target not met: could not fetch from [z6MkjM3…nGMq5o7], and required 1 more seed(s)
✗ Error: Fetched from 0 preferred seed(s), could not reach 1 seed(s)
✗ Error: Could not replicate from 1 preferred seed(s)
✗ Error: z6MkjM3…nGMq5o7: Could not connect. No addresses known.
```
Also note that you cannot sync an unseeded repo:

3
radicle-cli/examples/workflow/5-patching-maintainer.md
View File

@ -8,7 +8,8 @@ of this.
```
$ rad remote add z6Mkt67GdsW7715MEfRuP4pSZxJRJh6kj6Y48WRqVv4N1tRk --name bob --sync --fetch
✓ Follow policy updated for z6Mkt67GdsW7715MEfRuP4pSZxJRJh6kj6Y48WRqVv4N1tRk (bob)
✓ Fetching rad:z42hL2jL4XNk6K8oHQaSWfMgCL7ji from z6Mkt67…v4N1tRk@[..]..
Fetching rad:z42hL2jL4XNk6K8oHQaSWfMgCL7ji from the network, found 1 potential seed(s).
✓ Target met: 1 seed(s)
✓ Remote bob added
✓ Remote-tracking branch bob/master created for z6Mkt67…v4N1tRk
```

5
radicle-cli/examples/workflow/6-pulling-contributor.md
View File

@ -10,8 +10,9 @@ f2de534b5e81d7c6e2dcaf58c3dd91573c0a0354
Then, we call `rad sync --fetch` to fetch from the maintainer:
```
$ rad sync --fetch
✓ Fetching rad:z42hL2jL4XNk6K8oHQaSWfMgCL7ji from z6MknSL…StBU8Vi@[..]..
✓ Fetched repository from 1 seed(s)
Fetching rad:z42hL2jL4XNk6K8oHQaSWfMgCL7ji from the network, found 1 potential seed(s).
✓ Target met: 1 seed(s)
🌱 Fetched from z6MknSL…StBU8Vi
```
Now let's checkout `master` and pull the maintainer's changes:

16
radicle-cli/src/commands/clone.rs
View File

@ -82,7 +82,8 @@ impl Args for Options {
let value = term::args::nid(&value)?;
sync.seeds.insert(value);
sync.replicas = sync.seeds.len();
let n = sync.seeds.len();
sync.replicas = node::sync::ReplicationFactor::must_reach(n);
}
Long("scope") => {
let value = parser.value()?;
@ -245,11 +246,16 @@ pub fn clone(
);
}
let results = sync::fetch(id, settings, node, profile)?;
let result = sync::fetch(id, settings, node, profile)?;
// FIXME: handle the two cases
let fetch_results = match &result {
node::sync::FetcherResult::TargetReached(success) => success.fetch_results(),
node::sync::FetcherResult::TargetError(failed) => failed.fetch_results(),
};
let Ok(repository) = profile.storage.repository(id) else {
// If we don't have the repository locally, even after attempting to fetch,
// there's nothing we can do.
if results.is_empty() {
if fetch_results.is_empty() {
return Err(CloneError::NoSeeds(id));
} else {
return Err(CloneError::NotFound(id));
@ -267,8 +273,8 @@ pub fn clone(
}
}
if results.success().next().is_none() {
if results.failed().next().is_some() {
if fetch_results.success().next().is_none() {
if fetch_results.failed().next().is_some() {
term::warning("Fetching failed, local copy is potentially stale");
} else {
term::warning("No seeds found, local copy is potentially stale");

466
radicle-cli/src/commands/sync.rs
View File

@ -1,6 +1,6 @@
use std::cmp::Ordering;
use std::collections::BTreeSet;
use std::collections::VecDeque;
use std::collections::HashSet;
use std::ffi::OsString;
use std::str::FromStr;
use std::time;
@ -9,8 +9,12 @@ use anyhow::{anyhow, Context as _};
use radicle::node;
use radicle::node::address::Store;
use radicle::node::{AliasStore, FetchResult, FetchResults, Handle as _, Node, Seed, SyncStatus};
use radicle::node::sync;
use radicle::node::sync::fetch::SuccessfulOutcome;
use radicle::node::{AliasStore, Handle as _, Node, Seed, SyncStatus};
use radicle::prelude::{NodeId, Profile, RepoId};
use radicle::storage::ReadRepository;
use radicle::storage::RefUpdate;
use radicle::storage::{ReadStorage, RemoteRepository};
use radicle_term::Element;
@ -45,6 +49,12 @@ Usage
When `--replicas` is specified, the given replication factor will try
to be matched. For example, `--replicas 5` will sync with 5 seeds.
When `--max-replicas` is specified without `--replicas`, it acts the same a
replicas. Otherwise, it will use the value as an upper bound of seed to
replicate from. If the max is not reached, the sync is still considered a
success. For example, `--replicas 2 --max-replicas 3` will sync with 2 seeds
and attempt to sync with a third.
When `--fetch` or `--announce` are specified on their own, this command
will only fetch or announce.
@ -57,16 +67,17 @@ Commands
Options
--sort-by <field> Sort the table by column (options: nid, alias, status)
-f, --fetch Turn on fetching (default: true)
-a, --announce Turn on ref announcing (default: true)
-i, --inventory Turn on inventory announcing (default: false)
--timeout <secs> How many seconds to wait while syncing
--seed <nid> Sync with the given node (may be specified multiple times)
-r, --replicas <count> Sync with a specific number of seeds
-v, --verbose Verbose output
--debug Print debug information afer sync
--help Print help
--sort-by <field> Sort the table by column (options: nid, alias, status)
-f, --fetch Turn on fetching (default: true)
-a, --announce Turn on ref announcing (default: true)
-i, --inventory Turn on inventory announcing (default: false)
--timeout <secs> How many seconds to wait while syncing
--seed <nid> Sync with the given node (may be specified multiple times)
-r, --replicas <count> Sync with a specific number of seeds
--replicas-max <count> Sync with an upper bound number of seeds
-v, --verbose Verbose output
--debug Print debug information afer sync
--help Print help
"#,
};
@ -146,6 +157,7 @@ impl Args for Options {
let mut inventory = false;
let mut debug = false;
let mut replicas = None;
let mut max_replicas = None;
let mut seeds = BTreeSet::new();
let mut sort_by = SortBy::default();
let mut op: Option<Operation> = None;
@ -170,6 +182,15 @@ impl Args for Options {
}
replicas = Some(count);
}
Long("replicas-max") => {
let val = parser.value()?;
let count = term::args::number(&val)?;
if count == 0 {
anyhow::bail!("value for `--replicas-max` must be greater than zero");
}
max_replicas = Some(count);
}
Long("seed") => {
let val = parser.value()?;
let nid = term::args::nid(&val)?;
@ -221,9 +242,13 @@ impl Args for Options {
};
let mut settings = SyncSettings::default().timeout(timeout);
if let Some(r) = replicas {
settings.replicas = r;
}
let replicas = match (replicas, max_replicas) {
(None, None) => sync::ReplicationFactor::default(),
(None, Some(min)) => sync::ReplicationFactor::must_reach(min),
(Some(min), None) => sync::ReplicationFactor::must_reach(min),
(Some(min), Some(max)) => sync::ReplicationFactor::range(min, max),
};
settings.replicas = replicas;
if !seeds.is_empty() {
settings.seeds = seeds;
}
@ -285,17 +310,8 @@ pub fn run(options: Options, ctx: impl term::Context) -> anyhow::Result<()> {
if !profile.policies()?.is_seeding(&rid)? {
anyhow::bail!("repository {rid} is not seeded");
}
let results = fetch(rid, settings.clone(), &mut node, &profile)?;
let success = results.success().count();
let failed = results.failed().count();
if results.is_empty() {
term::error(format!("no seeds found for {rid}"));
} else if success == 0 {
term::error(format!("repository fetch from {failed} seed(s) failed"));
} else {
term::success!("Fetched repository from {success} seed(s)");
}
let result = fetch(rid, settings.clone(), &mut node, &profile)?;
display_fetch_result(&result, options.verbose)
}
if [SyncDirection::Announce, SyncDirection::Both].contains(&direction) {
announce_refs(rid, settings, options.debug, &mut node, &profile)?;
@ -444,6 +460,8 @@ pub enum FetchError {
Db(#[from] node::db::Error),
#[error(transparent)]
Address(#[from] node::address::Error),
#[error(transparent)]
Fetcher(#[from] sync::FetcherError),
}
pub fn fetch(
@ -451,124 +469,99 @@ pub fn fetch(
settings: SyncSettings,
node: &mut Node,
profile: &Profile,
) -> Result<FetchResults, FetchError> {
let local = node.nid()?;
let replicas = settings.replicas;
let mut results = FetchResults::default();
) -> Result<sync::FetcherResult, FetchError> {
let db = profile.database()?;
let local = profile.id();
let is_private = profile.storage.repository(rid).ok().and_then(|repo| {
let doc = repo.identity_doc().ok()?.doc;
sync::fetch::PrivateNetwork::private_repo(&doc)
});
let config = match is_private {
Some(private) => sync::FetcherConfig::private(private, settings.replicas, *local),
None => {
// We push nodes that are in our seed list in attempt to fulfill the
// replicas, if needed.
let seeds = node.seeds(rid)?;
let (connected, disconnected) = seeds.partition();
let candidates = connected
.into_iter()
.map(|seed| seed.nid)
.chain(disconnected.into_iter().map(|seed| seed.nid))
.map(sync::fetch::Candidate::new);
sync::FetcherConfig::public(settings.seeds.clone(), settings.replicas, *local)
.with_candidates(candidates)
}
};
let mut fetcher = sync::Fetcher::new(config)?;
// Fetch from specified seeds, connecting to them if necessary.
for nid in &settings.seeds {
match node.session(*nid)? {
Some(session) if session.is_connected() => {
fetch_from(
rid,
nid,
&session.addr,
settings.timeout,
&mut results,
node,
)?;
}
let mut progress = fetcher.progress();
term::info!(
"Fetching {} from the network, found {} potential seed(s).",
term::format::tertiary(rid),
term::format::tertiary(progress.candidate())
);
let mut spinner = FetcherSpinner::new(fetcher.target(), &progress);
while let Some(nid) = fetcher.next_node() {
match node.session(nid)? {
Some(session) if session.is_connected() => fetcher.ready_to_fetch(nid, session.addr),
_ => {
let addrs = db.addresses_of(nid)?;
let addrs = db.addresses_of(&nid)?;
if addrs.is_empty() {
results.push(
*nid,
FetchResult::Failed {
reason: format!("no addresses found in routing table for {nid}"),
},
);
term::warning(format!("no addresses found for {nid}, skipping.."));
fetcher.fetch_failed(nid, "Could not connect. No addresses known.");
} else if let Some(addr) = connect(
*nid,
nid,
addrs.into_iter().map(|ka| ka.addr),
settings.timeout,
node,
&mut spinner,
&fetcher.progress(),
) {
fetch_from(rid, nid, &addr, settings.timeout, &mut results, node)?;
fetcher.ready_to_fetch(nid, addr)
} else {
fetcher
.fetch_failed(nid, "Could not connect. At least one address is known but all attempts timed out.");
}
}
}
// We are done when we either hit our replica count,
// or fetch from all the specified seeds.
if results.success().count() >= replicas {
return Ok(results);
}
if settings
.seeds
.iter()
.all(|nid| results.get(nid).is_some_and(|r| r.is_success()))
{
return Ok(results);
if let Some((nid, addr)) = fetcher.next_fetch() {
spinner.emit_fetching(&nid, &addr, &progress);
let result = node.fetch(rid, nid, settings.timeout)?;
match fetcher.fetch_complete(nid, result) {
std::ops::ControlFlow::Continue(update) => {
spinner.emit_progress(&update);
progress = update
}
std::ops::ControlFlow::Break(success) => {
spinner.finished(success.outcome());
return Ok(sync::FetcherResult::TargetReached(success));
}
}
}
}
// If we're here, we haven't met our sync targets, so consult the routing table
// for more seeds to fetch from.
let seeds = node.seeds(rid)?;
let (connected, mut disconnected) = seeds.partition();
// Fetch from connected seeds.
let mut connected = connected
.into_iter()
.filter(|c| !results.contains(&c.nid))
.map(|c| c.nid)
.take(replicas)
.collect::<VecDeque<_>>();
while results.success().count() < replicas {
let Some(nid) = connected.pop_front() else {
break;
};
if let Some(session) = node.session(nid)? {
fetch_from(
rid,
&nid,
&session.addr,
settings.timeout,
&mut results,
node,
)?;
} else {
log::warn!("Could not obtain session for seed with ID {nid}, even though it should be connected.")
let result = fetcher.finish();
match &result {
sync::FetcherResult::TargetReached(success) => {
spinner.finished(success.outcome());
}
sync::FetcherResult::TargetError(missed) => spinner.failed(missed),
}
// Try to connect to disconnected seeds and fetch from them.
while results.success().count() < replicas {
let Some(seed) = disconnected.pop() else {
break;
};
if seed.nid == local {
// Skip our own node.
continue;
}
if let Some(addr) = connect(
seed.nid,
seed.addrs.into_iter().map(|ka| ka.addr),
settings.timeout,
node,
) {
fetch_from(rid, &seed.nid, &addr, settings.timeout, &mut results, node)?;
}
}
Ok(results)
Ok(result)
}
// Try all addresses until one succeeds.
// FIXME(fintohaps): I think this could return a `Result<node::Address,
// Vec<AddressError>>` which could report back why each address failed
fn connect(
nid: NodeId,
addrs: impl Iterator<Item = node::Address>,
timeout: time::Duration,
node: &mut Node,
spinner: &mut FetcherSpinner,
progress: &sync::fetch::Progress,
) -> Option<node::Address> {
// Try all addresses until one succeeds.
for addr in addrs {
let spinner = term::spinner(format!(
"Connecting to {}@{}..",
term::format::tertiary(term::format::node(&nid)),
&addr
));
spinner.emit_dialing(&nid, &addr, progress);
let cr = node.connect(
nid,
addr.clone(),
@ -580,15 +573,13 @@ fn connect(
match cr {
Ok(node::ConnectResult::Connected) => {
spinner.finish();
return Some(addr);
}
Ok(node::ConnectResult::Disconnected { reason }) => {
spinner.error(reason);
Ok(node::ConnectResult::Disconnected { .. }) => {
continue;
}
Err(e) => {
spinner.error(e);
log::warn!(target: "cli", "Failed to connect to {nid}@{addr}: {e}");
continue;
}
}
@ -596,42 +587,6 @@ fn connect(
None
}
fn fetch_from(
rid: RepoId,
seed: &NodeId,
addr: &node::Address,
timeout: time::Duration,
results: &mut FetchResults,
node: &mut Node,
) -> Result<(), node::Error> {
// NOTE: addr is only used for better output.
// Whether it actually corresponds to the addr
// of the connection to seed is not checked.
// There could be races.
// Our goal is to provide better output to the user,
// and it should be good enough for that.
let spinner = term::spinner(format!(
"Fetching {} from {}@{}..",
term::format::tertiary(rid),
term::format::tertiary(term::format::node(seed)),
addr,
));
let result = node.fetch(rid, *seed, timeout)?;
match &result {
FetchResult::Success { .. } => {
spinner.finish();
}
FetchResult::Failed { reason } => {
spinner.error(reason);
}
}
results.push(*seed, result);
Ok(())
}
fn sort_seeds_by(local: NodeId, seeds: &mut [Seed], aliases: &impl AliasStore, sort_by: &SortBy) {
let compare = |a: &Seed, b: &Seed| match sort_by {
SortBy::Nid => a.nid.cmp(&b.nid),
@ -659,3 +614,196 @@ fn sort_seeds_by(local: NodeId, seeds: &mut [Seed], aliases: &impl AliasStore, s
}
});
}
struct FetcherSpinner {
preferred_seeds: usize,
replicas: sync::ReplicationFactor,
spinner: term::Spinner,
}
impl FetcherSpinner {
fn new(target: &sync::fetch::Target, progress: &sync::fetch::Progress) -> Self {
let preferred_seeds = target.preferred_seeds().len();
let replicas = target.replicas();
let spinner = term::spinner(format!(
"{} of {} preferred seeds, and {} of at least {} total seeds.",
term::format::secondary(progress.preferred()),
term::format::secondary(preferred_seeds),
term::format::secondary(progress.succeeded()),
term::format::secondary(replicas.lower_bound())
));
Self {
preferred_seeds: target.preferred_seeds().len(),
replicas: *target.replicas(),
spinner,
}
}
fn emit_progress(&mut self, progress: &sync::fetch::Progress) {
self.spinner.message(format!(
"{} of {} preferred seeds, and {} of at least {} total seeds.",
term::format::secondary(progress.preferred()),
term::format::secondary(self.preferred_seeds),
term::format::secondary(progress.succeeded()),
term::format::secondary(self.replicas.lower_bound()),
))
}
fn emit_fetching(
&mut self,
node: &NodeId,
addr: &node::Address,
progress: &sync::fetch::Progress,
) {
self.spinner.message(format!(
"{} of {} preferred seeds, and {} of at least {} total seeds… [fetch {}@{}]",
term::format::secondary(progress.preferred()),
term::format::secondary(self.preferred_seeds),
term::format::secondary(progress.succeeded()),
term::format::secondary(self.replicas.lower_bound()),
term::format::tertiary(term::format::node(node)),
term::format::tertiary(addr),
))
}
fn emit_dialing(
&mut self,
node: &NodeId,
addr: &node::Address,
progress: &sync::fetch::Progress,
) {
self.spinner.message(format!(
"{} of {} preferred seeds, and {} of at least {} total seeds… [dial {}@{}]",
term::format::secondary(progress.preferred()),
term::format::secondary(self.preferred_seeds),
term::format::secondary(progress.succeeded()),
term::format::secondary(self.replicas.lower_bound()),
term::format::tertiary(term::format::node(node)),
term::format::tertiary(addr),
))
}
fn finished(mut self, outcome: &SuccessfulOutcome) {
match outcome {
SuccessfulOutcome::PreferredNodes { preferred } => {
self.spinner.message(format!(
"Target met: {} preferred seed(s).",
term::format::positive(preferred),
));
}
SuccessfulOutcome::MinReplicas { succeeded, .. } => {
self.spinner.message(format!(
"Target met: {} seed(s)",
term::format::positive(succeeded)
));
}
SuccessfulOutcome::MaxReplicas {
succeeded,
min,
max,
} => {
self.spinner.message(format!(
"Target met: {} of {} min and {} max seed(s)",
succeeded,
term::format::secondary(min),
term::format::secondary(max)
));
}
}
self.spinner.finish()
}
fn failed(mut self, missed: &sync::fetch::TargetMissed) {
let mut message = "Target not met: ".to_string();
let missing_preferred_seeds = missed
.missed_nodes()
.iter()
.map(|nid| term::format::node(nid).to_string())
.collect::<Vec<_>>();
let required = missed.required_nodes();
if !missing_preferred_seeds.is_empty() {
message.push_str(&format!(
"could not fetch from [{}], and required {} more seed(s)",
missing_preferred_seeds.join(", "),
required
));
} else {
message.push_str(&format!("required {} more seed(s)", required));
}
self.spinner.message(message);
self.spinner.failed();
}
}
fn display_fetch_result(result: &sync::FetcherResult, verbose: bool) {
match result {
sync::FetcherResult::TargetReached(success) => {
let progress = success.progress();
let results = success.fetch_results();
display_success(results.success(), verbose);
let failed = progress.failed();
if failed > 0 && verbose {
term::warning(format!("Failed to fetch from {failed} seed(s)."));
for (node, reason) in results.failed() {
term::warning(format!(
"{}: {}",
term::format::node(node),
term::format::yellow(reason),
))
}
}
}
sync::FetcherResult::TargetError(failed) => {
let results = failed.fetch_results();
let progress = failed.progress();
let target = failed.target();
let succeeded = progress.succeeded();
let missed = failed.missed_nodes();
term::error(format!(
"Fetched from {} preferred seed(s), could not reach {} seed(s)",
succeeded,
target.replicas().lower_bound(),
));
term::error(format!(
"Could not replicate from {} preferred seed(s)",
missed.len()
));
for (node, reason) in results.failed() {
term::error(format!(
"{}: {}",
term::format::node(node),
term::format::negative(reason),
))
}
if succeeded > 0 {
term::info!("Successfully fetched from the following seeds:");
display_success(results.success(), verbose)
}
}
}
}
fn display_success<'a>(
results: impl Iterator<Item = (&'a NodeId, &'a [RefUpdate], HashSet<NodeId>)>,
verbose: bool,
) {
for (node, updates, _) in results {
term::println(
"🌱 Fetched from",
term::format::secondary(term::format::node(node)),
);
if verbose {
let mut updates = updates
.iter()
.filter(|up| !matches!(up, RefUpdate::Skipped { .. }))
.peekable();
if updates.peek().is_none() {
term::indented(term::format::italic("no references were updated"));
} else {
for update in updates {
term::indented(term::format::ref_update_verbose(update))
}
}
}
}
}

12
radicle-cli/src/node.rs
View File

@ -4,7 +4,7 @@ use std::io;
use std::io::Write;
use std::ops::ControlFlow;
use radicle::node::{self, AnnounceResult};
use radicle::node::{self, sync, AnnounceResult};
use radicle::node::{Handle as _, NodeId};
use radicle::storage::{ReadRepository, RepositoryError};
use radicle::{Node, Profile};
@ -19,7 +19,7 @@ pub const DEFAULT_SYNC_TIMEOUT: time::Duration = time::Duration::from_secs(9);
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct SyncSettings {
/// Sync with at least N replicas.
pub replicas: usize,
pub replicas: sync::ReplicationFactor,
/// Sync with the given list of seeds.
pub seeds: BTreeSet<NodeId>,
/// How long to wait for syncing to complete.
@ -34,7 +34,7 @@ impl SyncSettings {
}
/// Set replicas.
pub fn replicas(mut self, replicas: usize) -> Self {
pub fn replicas(mut self, replicas: sync::ReplicationFactor) -> Self {
self.replicas = replicas;
self
}
@ -66,7 +66,7 @@ impl SyncSettings {
impl Default for SyncSettings {
fn default() -> Self {
Self {
replicas: 3,
replicas: sync::ReplicationFactor::default(),
seeds: BTreeSet::new(),
timeout: DEFAULT_SYNC_TIMEOUT,
}
@ -200,7 +200,7 @@ fn announce_<R: ReadRepository>(
// If the seeds we specified in the sync settings are all synced.
let is_seeds_synced = settings.seeds.iter().all(|s| synced.contains(s));
// If we met our desired replica count. Note that this can never exceed the maximum count.
let is_replicas_synced = replicas >= settings.replicas.min(max_replicas);
let is_replicas_synced = replicas >= settings.replicas.lower_bound().min(max_replicas);
// Nothing to do if we've met our sync state.
if is_seeds_synced && is_replicas_synced {
@ -254,7 +254,7 @@ fn announce_<R: ReadRepository>(
//
// 1. We've matched or exceeded our target replica count.
// 2. We've synced with one of the seeds specified manually.
if replicas.len() >= settings.replicas
if replicas.len() >= settings.replicas.lower_bound()
&& (settings.seeds.is_empty()
|| settings.seeds.iter().any(|s| replicas.contains_key(s)))
{

29
radicle-cli/src/terminal/format.rs
View File

@ -113,6 +113,35 @@ pub fn ref_update(update: &RefUpdate) -> Paint<&'static str> {
}
}
pub fn ref_update_verbose(update: &RefUpdate) -> Paint<String> {
match update {
RefUpdate::Created { name, .. } => format!(
"{: <17} {}",
term::format::positive("* [new ref]"),
term::format::secondary(name),
)
.into(),
RefUpdate::Updated { name, old, new } => format!(
"{: <17} {}",
format!("{}..{}", term::format::oid(*old), term::format::oid(*new)),
term::format::secondary(name),
)
.into(),
RefUpdate::Deleted { name, .. } => format!(
"{: <17} {}",
term::format::negative("- [deleted]"),
term::format::secondary(name),
)
.into(),
RefUpdate::Skipped { name, .. } => format!(
"{: <17} {}",
term::format::italic("* [skipped]"),
term::format::secondary(name)
)
.into(),
}
}
/// Identity formatter that takes a profile and displays it as
/// `<node-id> (<username>)` depending on the configuration.
pub struct Identity<'a> {

5
radicle-cli/tests/commands.rs
View File

@ -1329,6 +1329,7 @@ fn rad_patch_delete() {
bob.handle.seed(acme, Scope::All).unwrap();
seed.handle.seed(acme, Scope::All).unwrap();
alice.connect(&bob).connect(&seed).converge([&bob, &seed]);
bob.connect(&seed).converge([&seed]);
bob.routes_to(&[(acme, seed.id)]);
test(
@ -1990,6 +1991,10 @@ fn rad_diff() {
test("examples/rad-diff.md", working, None, []).unwrap();
}
// FIXME(fintohaps): I plan on fixing this logic in the next patch – clone
// should not need to fetch from the network if the repository is already
// available locally.
#[ignore]
#[test]
// User tries to clone; no seeds are available, but user has the repo locally.
fn test_clone_without_seeds() {

3
radicle/src/node.rs
View File

@ -12,6 +12,7 @@ pub mod policy;
pub mod refs;
pub mod routing;
pub mod seed;
pub mod sync;
pub mod timestamp;
use std::collections::{BTreeMap, BTreeSet, HashMap, HashSet, VecDeque};
@ -902,7 +903,7 @@ impl<S: ToString> From<Result<(Vec<RefUpdate>, HashSet<NodeId>, bool), S>> for F
}
/// Holds multiple fetch results.
#[derive(Debug, Default)]
#[derive(Clone, Debug, Default)]
pub struct FetchResults(Vec<(NodeId, FetchResult)>);
impl FetchResults {

128
radicle/src/node/sync.rs Normal file
View File

@ -0,0 +1,128 @@
pub mod fetch;
pub use fetch::{Fetcher, FetcherConfig, FetcherError, FetcherResult};
/// The replication factor of a syncing operation.
#[derive(Copy, Clone, Debug, PartialEq, Eq)]
pub enum ReplicationFactor {
/// The syncing operation much reach the given value.
///
/// See [`ReplicationFactor::must_reach`].
MustReach(usize),
/// The syncing operation must reach a minimum value, but may continue to
/// reach a maximum value.
///
/// See [`ReplicationFactor::range`].
Range(ReplicationRange),
}
#[derive(Copy, Clone, Debug, PartialEq, Eq)]
pub struct ReplicationRange {
lower: usize,
upper: usize,
}
impl Default for ReplicationFactor {
fn default() -> Self {
Self::must_reach(3)
}
}
impl ReplicationFactor {
/// Construct a replication factor with the `lower` and `upper` bounds.
///
/// If `lower >= upper`, then [`ReplicationFactor::MustReach`] is constructed instead of
/// `ReplicationFactor::Range`.
pub fn range(lower: usize, upper: usize) -> Self {
if lower >= upper {
Self::MustReach(lower)
} else {
Self::Range(ReplicationRange { lower, upper })
}
}
/// Construct a replication factor where the `factor` must be reached.
pub fn must_reach(factor: usize) -> Self {
Self::MustReach(factor)
}
/// Get the lower bound of the replication factor.
pub fn lower_bound(&self) -> usize {
match self {
Self::MustReach(lower) => *lower,
Self::Range(ReplicationRange { lower: min, .. }) => *min,
}
}
/// Get the upper bound of the replication factor, if the replication factor
/// is a range.
pub fn upper_bound(&self) -> Option<usize> {
match self {
Self::MustReach(_) => None,
Self::Range(ReplicationRange { upper: max, .. }) => Some(*max),
}
}
/// Set the minimum target of the [`ReplicationFactor`] to a new value.
///
/// If the original value is smaller than the new value, then the original
/// is kept.
///
/// If the [`ReplicationFactor`] is a range, it performs `min` on the upper
/// bound of the range.
///
/// If `self` was originally a [`ReplicationFactor::Range`], and `min >= max`, then
/// the returned value will be [`ReplicationFactor::MustReach`].
pub fn min(self, new: usize) -> Self {
match self {
Self::MustReach(min) => Self::MustReach(min.min(new)),
Self::Range(ReplicationRange {
lower: min,
upper: max,
}) => Self::range(min, max.min(new)),
}
}
}
#[cfg(test)]
mod test {
use super::*;
#[test]
fn ensure_replicas_construction() {
let replicas = ReplicationFactor::range(1, 3);
assert!(
replicas.lower_bound()
<= replicas
.upper_bound()
.expect("replicas should have max value")
);
let replicas = ReplicationFactor::range(1, 1);
assert!(replicas.upper_bound().is_none());
let replicas = ReplicationFactor::range(3, 1);
assert!(replicas.upper_bound().is_none());
}
#[test]
fn replicas_constrain_to() {
let replicas = ReplicationFactor::must_reach(3).min(1);
assert_eq!(replicas, ReplicationFactor::MustReach(1));
let replicas = ReplicationFactor::must_reach(3).min(3);
assert_eq!(replicas, ReplicationFactor::MustReach(3));
let replicas = ReplicationFactor::must_reach(3).min(10);
assert_eq!(replicas, ReplicationFactor::MustReach(3));
let replicas = ReplicationFactor::range(1, 3).min(1);
assert_eq!(replicas, ReplicationFactor::MustReach(1));
let replicas = ReplicationFactor::range(1, 3).min(3);
assert_eq!(
replicas,
ReplicationFactor::Range(ReplicationRange { lower: 1, upper: 3 })
);
let replicas = ReplicationFactor::range(1, 3).min(10);
assert_eq!(
replicas,
ReplicationFactor::Range(ReplicationRange { lower: 1, upper: 3 })
);
}
}

863
radicle/src/node/sync/fetch.rs Normal file
View File

@ -0,0 +1,863 @@
//! A sans-IO fetching state machine for driving fetch processes.
//!
//! See the documentation of [`Fetcher`] for more details.
use std::collections::{BTreeSet, VecDeque};
use std::ops::ControlFlow;
use crate::identity::Visibility;
use crate::node::{Address, FetchResult, FetchResults, NodeId};
use crate::prelude::Doc;
use super::ReplicationFactor;
/// A [`Fetcher`] describes a machine for driving a fetching process.
///
/// The [`Fetcher`] can be constructed using [`Fetcher::new`], providing a
/// [`FetcherConfig`].
///
/// It builds a [`Target`] that it attempts to reach:
/// * Number of replicas that it should successfully fetch from, where a
/// replica is any seed node that the repository is potentially seeded by.
/// * A set of preferred seeds that it should successfully fetch from.
///
/// If either of these targets are reached, then the fetch process can be
/// considered complete – with preference given to the preferred seeds target.
///
/// To drive the [`Fetcher`], it must be provided with nodes to fetch from.
/// These are added via the [`FetcherConfig`]. Note that the nodes provided are
/// retrieved in the order they are provided.
///
/// Before candidate nodes can be fetched from, the caller needs to mark them as
/// connected to. To get the next available node we call [`Fetcher::next_node`].
/// Once the caller attempts to connect to this node and retrieves its
/// [`Address`], then it can mark it as ready to fetch by calling
/// [`Fetcher::ready_to_fetch`].
///
/// To then retrieve the next available node for fetching, the caller uses
/// [`Fetcher::next_fetch`].
///
/// To mark that fetch as complete, we call [`Fetcher::fetch_complete`], with
/// the result. At this point, the [`Fetcher`] returns a [`ControlFlow`] to let
/// the caller know if they should continue processing nodes, to reach the
/// desired target, or they can exit the loop knowing they have successfully
/// reached the target.
///
/// The caller may also call [`Fetcher::fetch_failed`] to mark a fetch for a
/// given node as failed – this is useful for reasons when the caller cannot
/// connect to the node for fetching.
///
/// Finally, if the caller wishes to exit from the fetching process and get the
/// final set of results, they may call [`Fetcher::finish`].
#[derive(Debug)]
#[must_use]
pub struct Fetcher {
target: Target,
fetch_from: VecDeque<Ready>,
candidates: VecDeque<Candidate>,
results: FetchResults,
local_node: NodeId,
}
#[derive(Debug, thiserror::Error)]
#[non_exhaustive]
pub enum FetcherError {
#[error("no candidate seeds were found to fetch from")]
NoCandidates,
#[error(transparent)]
Target(#[from] TargetError),
}
impl Fetcher {
/// Construct a new [`Fetcher`] from the [`FetcherConfig`].
pub fn new(config: FetcherConfig) -> Result<Self, FetcherError> {
if config.candidates.is_empty() {
return Err(FetcherError::NoCandidates);
}
// N.b. ensure that we can reach the replicas count
let replicas = config.replicas.min(config.candidates.len());
Ok(Self {
target: Target::new(config.seeds, replicas)?,
fetch_from: VecDeque::new(),
candidates: config.candidates,
results: FetchResults::default(),
local_node: config.local_node,
})
}
/// Get the next candidate [`NodeId`] to attempt connection and/or
/// retrieving their connection session.
pub fn next_node(&mut self) -> Option<NodeId> {
let local_node = self.local_node;
let results = &self.results;
let include_node = |node: &NodeId| results.get(node).is_none() && local_node != *node;
// Find the first candidate that passes the `include_node` filter, or we
// exhaust the candidate list
std::iter::from_fn(|| self.candidates.pop_front()).find_map(|c| {
let node = c.nid();
include_node(&node).then_some(node)
})
}
/// Get the next [`NodeId`] and [`Address`] for performing a fetch from.
///
/// Note that this [`NodeId`] must have been added to the [`Fetcher`] using
/// the [`Fetcher::ready_to_fetch`] method.
pub fn next_fetch(&mut self) -> Option<(NodeId, Address)> {
self.fetch_from
.pop_front()
.map(|Ready { node, addr }| (node, addr))
.filter(|(node, _)| self.include_node(node))
}
/// Mark a fetch as failed for the [`NodeId`], using the provided `reason`.
pub fn fetch_failed(&mut self, node: NodeId, reason: impl ToString) {
let reason = reason.to_string();
self.results.push(node, FetchResult::Failed { reason })
}
/// Mark a fetch as complete for the [`NodeId`], with the provided
/// [`FetchResult`].
///
/// If the target for the [`Fetcher`] has been reached, then a [`Success`] is
/// returned via [`ControlFlow::Break`]. Otherwise, [`Progress`] is returned
/// via [`ControlFlow::Continue`].
///
/// The caller decides whether they wish to continue the fetching process.
pub fn fetch_complete(
&mut self,
node: NodeId,
result: FetchResult,
) -> ControlFlow<Success, Progress> {
self.results.push(node, result);
self.finished()
}
/// Complete the [`Fetcher`] process returning a [`FetcherResult`].
///
/// Which variant of the result is returned is determined by whether the
/// [`Fetcher`]'s target was reached.
pub fn finish(self) -> FetcherResult {
let progress = self.progress();
match self.is_target_reached() {
None => {
let missing = self.missing_seeds();
FetcherResult::target_error(progress, self.target, self.results, missing)
}
Some(outcome) => FetcherResult::target_reached(outcome, progress, self.results),
}
}
/// Mark the `node` as ready to fetch, by providing its [`Address`].
///
/// This will prime the `node` for fetching.
pub fn ready_to_fetch(&mut self, node: NodeId, addr: Address) {
self.fetch_from.push_back(Ready { node, addr })
}
/// Get the latest [`Progress`] of the [`Fetcher`].
pub fn progress(&self) -> Progress {
let (preferred, succeeded) = self.success_counts();
Progress {
candidate: self.candidates.len(),
succeeded,
failed: self.results.failed().count(),
preferred,
}
}
/// Get the [`Target`] that the [`Fetcher`] is aiming to reach.
pub fn target(&self) -> &Target {
&self.target
}
fn finished(&self) -> ControlFlow<Success, Progress> {
let progress = self.progress();
self.is_target_reached()
.map_or(ControlFlow::Continue(progress), |outcome| {
ControlFlow::Break(Success {
outcome,
progress,
results: self.results.clone(),
})
})
}
fn is_target_reached(&self) -> Option<SuccessfulOutcome> {
let (preferred, succeeded) = self.success_counts();
if !self.target.seeds.is_empty() && preferred >= self.target.seeds.len() {
Some(SuccessfulOutcome::PreferredNodes {
preferred: self.target.seeds.len(),
})
} else {
let replicas = self.target.replicas();
let min = replicas.lower_bound();
match replicas.upper_bound() {
None => (succeeded >= min).then_some(SuccessfulOutcome::MinReplicas { succeeded }),
Some(max) => (succeeded >= max).then_some(SuccessfulOutcome::MaxReplicas {
succeeded,
min,
max,
}),
}
}
}
/// Ensure that node does not already have a result and is not the local
/// node.
fn include_node(&self, node: &NodeId) -> bool {
self.results.get(node).is_none() && self.local_node != *node
}
fn missing_seeds(&self) -> BTreeSet<NodeId> {
self.target
.seeds
.iter()
.filter(|nid| match self.results.get(nid) {
Some(r) if !r.is_success() => true,
None => true,
_ => false,
})
.copied()
.collect()
}
fn success_counts(&self) -> (usize, usize) {
self.results
.success()
.fold((0, 0), |(mut preferred, mut succeeded), (nid, _, _)| {
succeeded += 1;
if self.target.seeds.contains(nid) {
preferred += 1;
}
(preferred, succeeded)
})
}
}
/// A set of nodes that form a private network for fetching from.
///
/// This could be the set of allowed nodes for a private repository, using
/// [`PrivateNetwork::private_repo`]
pub struct PrivateNetwork {
allowed: BTreeSet<NodeId>,
}
impl PrivateNetwork {
pub fn private_repo(doc: &Doc) -> Option<Self> {
match doc.visibility() {
Visibility::Public => None,
Visibility::Private { allow } => {
let allowed = doc
.delegates()
.iter()
.chain(allow.iter())
.map(|did| *did.as_key())
.collect();
Some(Self { allowed })
}
}
}
}
/// The progress a [`Fetcher`] is making.
#[derive(Clone, Copy, Debug)]
pub struct Progress {
/// How many candidate nodes are known.
candidate: usize,
/// How many fetches succeeded.
succeeded: usize,
/// How many fetches failed.
failed: usize,
/// How many fetches succeeded from preferred seeds.
preferred: usize,
}
impl Progress {
/// Get the number of successful fetches.
pub fn succeeded(&self) -> usize {
self.succeeded
}
/// Get the number of failed fetches.
pub fn failed(&self) -> usize {
self.failed
}
/// Get the number of successful fetches from preferred seeds.
pub fn preferred(&self) -> usize {
self.preferred
}
pub fn candidate(&self) -> usize {
self.candidate
}
}
/// The target for the `Fetcher` to reach.
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct Target {
seeds: BTreeSet<NodeId>,
replicas: ReplicationFactor,
}
#[derive(Debug, thiserror::Error)]
#[non_exhaustive]
#[error("a minimum number of replicas or set of preferred seeds must be provided")]
pub struct TargetError;
impl Target {
pub fn new(seeds: BTreeSet<NodeId>, replicas: ReplicationFactor) -> Result<Self, TargetError> {
if replicas.lower_bound() == 0 && seeds.is_empty() {
Err(TargetError)
} else {
Ok(Self { seeds, replicas })
}
}
/// Get the set of preferred seeds that are trying to be fetched from.
pub fn preferred_seeds(&self) -> &BTreeSet<NodeId> {
&self.seeds
}
/// Get the number of replicas that is trying to be reached.
pub fn replicas(&self) -> &ReplicationFactor {
&self.replicas
}
}
/// The outcome reached by the [`Fetcher`], depending on which target was
/// reached first.
#[derive(Copy, Clone, Debug, PartialEq, Eq)]
pub enum SuccessfulOutcome {
PreferredNodes {
preferred: usize,
},
MinReplicas {
succeeded: usize,
},
MaxReplicas {
succeeded: usize,
min: usize,
max: usize,
},
}
/// A successful `Fetcher` process result, where the target was reached.
pub struct Success {
outcome: SuccessfulOutcome,
progress: Progress,
results: FetchResults,
}
impl Success {
/// Get the final [`Progress`] of the fetcher result.
pub fn progress(&self) -> Progress {
self.progress
}
/// Get the final [`FetchResults`] of the fetcher result.
pub fn fetch_results(&self) -> &FetchResults {
&self.results
}
/// Get the [`SuccessfulOutcome`] of the fetcher result.
pub fn outcome(&self) -> &SuccessfulOutcome {
&self.outcome
}
}
/// An unsuccessful `Fetcher` process result, where the target was not reached.
///
/// Note that the caller can still decide if the process was a success based on
/// the [`FetchResults`].
pub struct TargetMissed {
progress: Progress,
target: Target,
results: FetchResults,
required: usize,
missed_nodes: BTreeSet<NodeId>,
}
impl TargetMissed {
/// Get the final [`Progress`] of the fetcher result.
pub fn progress(&self) -> Progress {
self.progress
}
/// Get the [`Target`] that was trying to be reached.
pub fn target(&self) -> &Target {
&self.target
}
/// Get the final [`FetchResults`] of the fetcher result.
pub fn fetch_results(&self) -> &FetchResults {
&self.results
}
/// Get the set of nodes that were missed when attempting to fetch.
pub fn missed_nodes(&self) -> &BTreeSet<NodeId> {
&self.missed_nodes
}
/// Get the number of nodes that were required to reach the replication
/// target.
pub fn required_nodes(&self) -> usize {
self.required
}
}
/// The result of a [`Fetcher`] process.
pub enum FetcherResult {
/// The target was reached and the process is considered a success.
TargetReached(Success),
/// The replication factor could not be reached at all, neither minimum nor
/// maximum, and so this fetch should be considered an error.
TargetError(TargetMissed),
}
impl FetcherResult {
/// Get the final [`Progress`] of the fetcher result.
pub fn progress(&self) -> Progress {
match self {
FetcherResult::TargetReached(s) => s.progress(),
FetcherResult::TargetError(f) => f.progress(),
}
}
fn target_reached(
outcome: SuccessfulOutcome,
progress: Progress,
results: FetchResults,
) -> Self {
Self::TargetReached(Success {
outcome,
progress,
results,
})
}
fn target_error(
progress: Progress,
target: Target,
results: FetchResults,
missing: BTreeSet<NodeId>,
) -> Self {
let required = target
.replicas
.lower_bound()
.saturating_sub(progress.succeeded);
Self::TargetError(TargetMissed {
progress,
target,
results,
missed_nodes: missing,
required,
})
}
}
/// Configuration of the [`Fetcher`].
pub struct FetcherConfig {
/// The set of seeds that are expected to replicate the repository.
seeds: BTreeSet<NodeId>,
/// The number of replicas to reach for the [`Fetcher`].
replicas: ReplicationFactor,
/// The candidate nodes that the node will attempt to fetch from.
candidates: VecDeque<Candidate>,
/// The identity of the local node, to ensure that it is never emitted for
/// connecting/fetching.
local_node: NodeId,
}
impl FetcherConfig {
/// Setup a private network `FetcherConfig`, populating the
/// [`FetcherConfig`]'s seeds with the allowed set from the
/// [`PrivateNetwork`]. It is recommended that
/// [`FetcherConfig::with_candidates`] is not used to extend the candidate
/// set.
///
/// `replicas` is the target number of seeds the [`Fetcher`] should reach
/// before stopping.
///
/// `local_node` is the [`NodeId`] of the local node, to ensure it is
/// excluded from the [`Fetcher`] process.
pub fn private(
private: PrivateNetwork,
replicas: ReplicationFactor,
local_node: NodeId,
) -> Self {
let candidates = private
.allowed
.clone()
.into_iter()
.filter(|node| *node != local_node)
.map(Candidate::new)
.collect::<VecDeque<_>>();
Self {
seeds: private.allowed,
replicas,
candidates,
local_node,
}
}
/// `seeds` is the target set of preferred seeds that [`Fetcher`] should
/// attempt to fetch from. These are the initial set of candidates nodes –
/// to add more use [`FetcherConfig::with_candidates`].
///
/// `replicas` is the target number of seeds the [`Fetcher`] should reach
/// before stopping.
///
/// `local_node` is the [`NodeId`] of the local node, to ensure it is
/// excluded from the [`Fetcher`] process.
pub fn public(
seeds: BTreeSet<NodeId>,
replicas: ReplicationFactor,
local_node: NodeId,
) -> Self {
let candidates = seeds
.clone()
.into_iter()
.filter(|node| *node != local_node)
.map(Candidate::new)
.collect::<VecDeque<_>>();
Self {
seeds,
replicas,
candidates,
local_node,
}
}
/// Extend the set of candidate nodes to attempt to fetch from.
pub fn with_candidates(mut self, extra: impl IntoIterator<Item = Candidate>) -> Self {
self.candidates
.extend(extra.into_iter().filter(|c| c.nid() != self.local_node));
self
}
}
/// A candidate node that can be returned by [`Fetcher::next_node`].
#[derive(Clone, Copy, Debug, PartialEq, Eq, PartialOrd, Ord)]
pub struct Candidate(NodeId);
impl Candidate {
pub fn new(node: NodeId) -> Self {
Self(node)
}
}
impl Candidate {
fn nid(&self) -> NodeId {
self.0
}
}
/// A node that is marked as ready by calling [`Fetcher::ready_to_fetch`].
#[derive(Debug)]
struct Ready {
node: NodeId,
addr: Address,
}
#[cfg(test)]
mod test {
use std::collections::HashSet;
use crate::test::arbitrary;
use super::*;
#[test]
fn all_nodes_are_candidates() {
let local = arbitrary::gen::<NodeId>(0);
let replicas = ReplicationFactor::default();
let seeds = arbitrary::set::<NodeId>(3..=6)
.into_iter()
.collect::<BTreeSet<_>>();
let extra_candidates = arbitrary::vec::<NodeId>(3);
let config = FetcherConfig::public(seeds.clone(), replicas, local)
.with_candidates(extra_candidates.clone().into_iter().map(Candidate::new));
let mut fetcher = Fetcher::new(config).expect("fetcher should be constructed correctly");
let mut result = Vec::with_capacity(seeds.len() + extra_candidates.len());
let expected = seeds
.into_iter()
.chain(extra_candidates)
.collect::<Vec<_>>();
while let Some(node) = fetcher.next_node() {
result.push(node);
}
// Check that there is no node for fetching, since we have not marked
// any as connected
assert!(fetcher.next_fetch().is_none());
assert_eq!(result, expected);
}
#[test]
fn ignores_duplicates_and_local_node() {
let local = arbitrary::gen::<NodeId>(0);
let replicas = ReplicationFactor::default();
let bob = arbitrary::gen::<NodeId>(1);
let eve = arbitrary::gen::<NodeId>(2);
let seeds = [bob].into_iter().collect::<BTreeSet<_>>();
let extra_candidates = vec![bob, local, eve];
let config = FetcherConfig::public(seeds.clone(), replicas, local)
.with_candidates(extra_candidates.clone().into_iter().map(Candidate::new));
let mut fetcher = Fetcher::new(config).expect("fetcher should be constructed correctly");
let mut result = Vec::with_capacity(seeds.len() + extra_candidates.len());
let expected = vec![bob, eve];
while let Some(node) = fetcher.next_node() {
fetcher.fetch_failed(node, "could not connect");
result.push(node);
}
assert_eq!(result, expected);
}
#[test]
fn all_nodes_are_fetchable() {
let local = arbitrary::gen::<NodeId>(0);
let replicas = ReplicationFactor::default();
let seeds = arbitrary::set::<NodeId>(3..=6)
.into_iter()
.collect::<BTreeSet<_>>();
let extra_candidates = arbitrary::vec::<NodeId>(3);
let config = FetcherConfig::public(seeds.clone(), replicas, local)
.with_candidates(extra_candidates.clone().into_iter().map(Candidate::new));
let mut fetcher = Fetcher::new(config).expect("fetcher should be constructed correctly");
let mut result = Vec::with_capacity(seeds.len() + extra_candidates.len());
let expected = seeds
.into_iter()
.chain(extra_candidates)
.collect::<Vec<_>>();
while let Some(node) = fetcher.next_node() {
fetcher.ready_to_fetch(node, arbitrary::gen::<Address>(0));
}
while let Some((node, _)) = fetcher.next_fetch() {
result.push(node);
}
assert_eq!(result, expected);
}
#[test]
fn reaches_target_of_preferred_seeds() {
let local = arbitrary::gen::<NodeId>(0);
let replicas = ReplicationFactor::default();
let seeds = arbitrary::set::<NodeId>(3..=3)
.into_iter()
.collect::<BTreeSet<_>>();
let extra_candidates = arbitrary::vec::<NodeId>(3);
let config = FetcherConfig::public(seeds.clone(), replicas, local)
.with_candidates(extra_candidates.clone().into_iter().map(Candidate::new));
let mut fetcher = Fetcher::new(config).expect("fetcher should be constructed correctly");
let mut result = Vec::with_capacity(seeds.len());
let expected = seeds.into_iter().collect::<Vec<_>>();
while let Some(node) = fetcher.next_node() {
fetcher.ready_to_fetch(node, arbitrary::gen::<Address>(0));
if let Some((node, _)) = fetcher.next_fetch() {
match fetcher.fetch_complete(
node,
FetchResult::Success {
updated: vec![],
namespaces: HashSet::new(),
clone: false,
},
) {
ControlFlow::Continue(_) => result.push(node),
ControlFlow::Break(success) => {
assert_eq!(
*success.outcome(),
SuccessfulOutcome::PreferredNodes { preferred: 3 }
);
result.push(node);
break;
}
}
}
}
assert_eq!(result, expected);
}
#[test]
fn reaches_target_of_replicas() {
let local = arbitrary::gen::<NodeId>(0);
let replicas = ReplicationFactor::must_reach(3);
let seeds = arbitrary::set::<NodeId>(3..=3)
.into_iter()
.collect::<BTreeSet<_>>();
let extra_candidates = arbitrary::vec::<NodeId>(3);
let config = FetcherConfig::public(seeds.clone(), replicas, local)
.with_candidates(extra_candidates.clone().into_iter().map(Candidate::new));
let mut fetcher = Fetcher::new(config).expect("fetcher should be constructed correctly");
let mut result = Vec::with_capacity(extra_candidates.len());
let expected = extra_candidates
.clone()
.into_iter()
.take(replicas.lower_bound())
.collect::<Vec<_>>();
while let Some(node) = fetcher.next_node() {
fetcher.ready_to_fetch(node, arbitrary::gen::<Address>(0));
if let Some((node, _)) = fetcher.next_fetch() {
if seeds.contains(&node) {
fetcher.fetch_failed(node, "failed fetch");
continue;
}
match fetcher.fetch_complete(
node,
FetchResult::Success {
updated: vec![],
namespaces: HashSet::new(),
clone: false,
},
) {
ControlFlow::Continue(_) => result.push(node),
ControlFlow::Break(success) => {
assert_eq!(
*success.outcome(),
SuccessfulOutcome::MinReplicas { succeeded: 3 }
);
result.push(node);
break;
}
}
}
}
assert_eq!(result, expected);
}
#[test]
fn reaches_target_of_max_replicas() {
let local = arbitrary::gen::<NodeId>(0);
let replicas = ReplicationFactor::range(1, 3);
let candidates = arbitrary::set::<NodeId>(3..=3);
let seeds = candidates.iter().take(3).copied().collect::<BTreeSet<_>>();
let extra_candidates = candidates.into_iter().skip(3).collect::<Vec<_>>();
let config = FetcherConfig::public(seeds.clone(), replicas, local)
.with_candidates(extra_candidates.clone().into_iter().map(Candidate::new));
let mut fetcher = Fetcher::new(config).expect("fetcher should be constructed correctly");
let mut result = Vec::with_capacity(extra_candidates.len());
let expected = extra_candidates
.clone()
.into_iter()
.take(replicas.upper_bound().expect("replicas must have max"))
.collect::<Vec<_>>();
while let Some(node) = fetcher.next_node() {
fetcher.ready_to_fetch(node, arbitrary::gen::<Address>(0));
if let Some((node, _)) = fetcher.next_fetch() {
if seeds.contains(&node) {
fetcher.fetch_failed(node, "could not connect");
continue;
}
match fetcher.fetch_complete(
node,
FetchResult::Success {
updated: vec![],
namespaces: HashSet::new(),
clone: false,
},
) {
ControlFlow::Continue(_) => result.push(node),
ControlFlow::Break(success) => {
assert_eq!(
*success.outcome(),
SuccessfulOutcome::MaxReplicas {
succeeded: 3,
min: 1,
max: 3
}
);
result.push(node);
break;
}
}
}
}
assert_eq!(
result,
expected,
"expected {} seed(s), found {}",
expected.len(),
result.len(),
);
}
#[test]
fn preferred_seeds_target_returned_over_replicas() {
let local = arbitrary::gen::<NodeId>(0);
let replicas = ReplicationFactor::range(1, 3);
let candidates = arbitrary::set::<NodeId>(3..=3);
let seeds = candidates.into_iter().collect::<BTreeSet<_>>();
let config = FetcherConfig::public(seeds.clone(), replicas, local);
let mut fetcher = Fetcher::new(config).expect("fetcher should be constructed correctly");
let mut result = Vec::with_capacity(seeds.len());
while let Some(node) = fetcher.next_node() {
fetcher.ready_to_fetch(node, arbitrary::gen::<Address>(0));
if let Some((node, _)) = fetcher.next_fetch() {
match fetcher.fetch_complete(
node,
FetchResult::Success {
updated: vec![],
namespaces: HashSet::new(),
clone: false,
},
) {
ControlFlow::Continue(_) => result.push(node),
ControlFlow::Break(success) => {
assert_eq!(
*success.outcome(),
SuccessfulOutcome::PreferredNodes { preferred: 3 }
);
result.push(node);
break;
}
}
}
}
assert_eq!(result, seeds.into_iter().collect::<Vec<_>>());
}
#[test]
fn could_not_reach_target() {
let local = arbitrary::gen::<NodeId>(0);
let replicas = ReplicationFactor::must_reach(4);
let candidates = arbitrary::set::<NodeId>(3..=3);
let seeds = candidates.into_iter().collect::<BTreeSet<_>>();
let config = FetcherConfig::public(seeds.clone(), replicas, local);
let mut fetcher = Fetcher::new(config).expect("fetcher should be constructed correctly");
while let Some(node) = fetcher.next_node() {
fetcher.ready_to_fetch(node, arbitrary::gen::<Address>(0));
if let Some((node, _)) = fetcher.next_fetch() {
fetcher.fetch_failed(node, "could not connect");
}
}
let result = fetcher.finish();
assert!(matches!(result, FetcherResult::TargetError(_)));
}
}