This patch changes separates the business logic of fetching from the process of
fetching itself. It does this through a sans-IO approach, where a `Fetcher`
provides the necessary state to help drive a fetch forward, without performing
any of the IO itself.
What the `Fetcher` cares about is:
- What nodes are going to be attempted to be fetched from
- In what order should they be attempted
- When should the fetching process be considered finished
The `Fetcher` is then used in `radicle-cli` to drive forward the `sync::fetch`
function, allowing it to only care about the IO.
To keep expectations similar to the seed command after the changes in
the companion patch:
rad:z3gqcJUoA1n9HaHKufZs5FCSGazv5/patch/3be0e68b8948e65431a288f225454bafd93de34a
Adds the new `migrate` sub-command to migrate the COB database.
Checks for version when opening the COBs database and suggests the
command on out of date version.
Also adds the 2nd migration to the COBs cache, which updates the patch
JSON schema.
This is meant to record in Radicle that automated "jobs" have been run
on a repository. The job might be, for example, a CI run that builds
the software and runs its tests. The recording is done in a "job COB",
which can be in three states:
* fresh -- when the job has been triggered, but has not yet started running
* running -- when it's running
* finished -- when it's finished running
The finished state also records whether the job was successful or if
it failed.
A job COB records the git commit that it runs on, and the "run ID" of
the job. The run ID represents an external identifier, such as one
assigned by a remote CI system. The COB can optionally store a URL
related to the job, such as to the build log from CI.
Add the "rad job" subcommand to create and manage job COBs. For
production use, the COBs should be created and updated using
`heartwood` as a library instead of running "rad job", if at all
possible. However, "rad job list" lists the jobs that have been run on
the repository.
The COB type name is xyz.radicle.beta.job, until we've gained
experience with the COB and its implementation and are ready to
declare it stable.
Signed-off-by: Lars Wirzenius <liw@liw.fi>
When pushing to the default branch and updating the canonical head,
check to see if any merged patches are reverted due to this change.
If so, make sure the COB cache reflects this.
This adds an option to `rad init` to initialize a working copy with
an existing radicle repository in storage:
rad init --existing <rid>
This sets up the remote(s) and configures the repo for you.
If a peer had created any references but was also missing the
canonical branch, the `rad id update` command will fail when adding
that peer as a delegate.
Instead, missing delegates and missing default branches are tracked
during verification. If the number of missing delegates means the
`threshold` cannot be met, then the errors are returned.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
The patch creation flow would be partially interrupted when creating a
patch via a detached HEAD state. The creation of an upstream branch
would fail since there is no branch to set an upstream for. It would
result in the following error:
✓ Patch 6035d2f582afbe01ff23ea87528ae523d76875b6 opened
To rad://z42hL2jL4XNk6K8oHQaSWfMgCL7ji/z6MknSLrJoTcukLrE435hVNQT4JUhbvWLX4kUzqkEStBU8Vi
! [remote rejected] HEAD -> refs/patches (git: reference 'HEAD' is neither a local nor a remote branch.; class=Invalid (3))
error: failed to push some refs to 'rad://z42hL2jL4XNk6K8oHQaSWfMgCL7ji/z6MknSLrJoTcukLrE435hVNQT4JUhbvWLX4kUzqkEStBU8Vi'
To fix this, the reference that's resolved is checked to see if it a
local branch. If not, it will return early.
The behaviour is tested in the `rad-patch-detached-head` example.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
The rules for checking the quorum of the remote helper, when pushing to the
canonical branch, can prevent the operator from even attempting to try converge
to a state of agreeance.
Instead, the checking of the quorum rules are relaxed. It is still the case that
if the peer is pushing a diverging commit while in a working state they are
warned and the push does not go through. However, there are two cases where it
is fine to force push a change:
1. When pushing a new tip, as a delegate, there is still a canonical tip and the
new tip still converges with the history, i.e. it is equal to, behind, or
ahead of another delegate tip.
2. If there is no quorum in the current state, for example there were multiple
concurrent pushes, then the peer is free to continue to push to their default
branch in attempt to converge with the other peers.
The example tests ensure that these cases are covered, while also preserving the
old behaviour.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
If a delegate is in a diverged state from the canonical head, ensure
that they can easily reset to another delegate's head to converge
again.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
The previous implementation of using the `--allow` option in `rad id
update` would be an absolute list, rather than additive. That is, if
there were existing DIDs not passed in via `--allow` then they would
be remove.
Change this behaviour to check the existing value of the document's
allow list, and add to it if `--allow` is specified.
To enable the user to also remove a DID, the `--disallow` option is
also added.
The correct usage of these options is improved by checking the
validity of the `--visibility` option being used, the current state of
the repository's privacy, and the use of the `--allow` and
`--disallow` options.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
1. Adds a user-agent string to the node announcement. This lets us keep
track of what software version everyone is running. Especially useful
to see what percentage of the network has upgraded to a new release.
For old nodes, this defaults to the string `/radicle/`.
The user agent format is roughly based on BIP 0014.
2. Advertize protocol version in node announcement.
By advertizing the protocol version, nodes can decide to only connect to
peers with a compatible version. The default and current version is `1`.
We bundle these two changes as they both modify the node announcement
and node database.
3. Swap the order of the fields in announcements to make the message
extensible.
Previously, our advertized inventory was not always correct: when we
stopped seeding a repository, it was still advertised as part of our
inventory until node restart, because the `Storage` type doesn't have
access to seeding policies.
In this change, we remove the concept of inventory from `Storage` and
make the authoritative place for it the routing table in our database.
To make this work, we have to add our node to the database on profile
creation, to not violate the foreign key constraint on the routing
table. Hence, the tests are changed to include our alias which is now
always available.
This is a change to the `config.json` file which removes certain options
that are best not touched and changes some options to make them easier
to work with.
1. We change the default journaling mode to "wal" and remove the config
option.
2. We remove the option to set the peer connection "target", as it's not
a good idea to set this to a different value, as it affects the
network.
3. We combine seeding policy & scope, since there's no need to set the
scope when the policy is "block". In that case, we always want to
block "all" remotes. The new policy configuration has the following
schema:
{ seedingPolicy: { default: "block" | "allow", scope?: "all" | "followed" } }
The "scope" key is not used when "default" is set to "block".
4. We add an `extra` field to the node config for options that are not
recognized. We use this to warn the user.
We use the *staggered broadcast* technique when relaying gossip messages
to reduce message amplification. In our basic simulation, it brings
amplification down from 3.5 to 1.5.
The idea is simply to accumulate gossip messages and relay them after a fixed
interval of time. This has two effects:
1) Old messages that haven't been sent yet are replaced by newer ones,
so only one message is sent after the delay.
2) Because nodes have their own intervals, messages are not all sent at
the same time, which reduces the chance of messages crossing paths.
For now we only turn this on for inventory messages, since this is the
least likely to be noticed by users, and also the most problematic
currently.
Add a test to confirm that the identity can still be deserialized when an
unknown field is added.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
Relay now has three values: "always", "never" and "auto".
The first two operate like the previous `true`/`false`. The
new (default) value turns relaying on for nodes with public (external)
addresses and off for nodes without. This should reduce redundant traffic on
the network, that was causing the rate limiters to trigger.
* Try to use constants instead of strings
* Move crypto seed override out of radicle-crypto
* Use GIT_COMMITTER_DATE for commit overrides
* Use RAD_LOCAL_TIME as general time override
* Allow variables to be used in release mode
When a `Patch` or `Issue` is entirely removed from the Git repository,
then there is no update to be made -- instead it should also be
removed from the cache.
This covers the case where the original author of the COB deletes the
COB.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
* Add `--force` flag to force a fetch even if the node is not a seed.
* Don't fail if we can't connect to a node during a fetch.
* More accurately exit when we hit our replica count.
There are two areas where we can be more tolerant of delegate namespaces
being missing or not validating:
1. Calculating the canonical HEAD
2. Fetching from a remote
In 1. the protocol is tolerant in that if the local node does not have
the default branch for a delegate, it will still attempt to use any of
the delegates it does have to reach the threshold.
This is made safe by ensuring that if the threshold is being updated
then the node performing the update must have a threshold of delegates
locally in their storage. It also made safe by 2.
In 2. the protocol is tolerant by allowing delegates to be missing
from the serving side, as long as they can still meet a threshold of
delegates. This is further tolerant, when validating the received
data, a threshold of delegates are valid to consider the fetch
successful -- otherwise it will fail.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
Previously, we only kept tracked of unsynced nodes we were connected to.
Now we track all unsynced nodes, since announcements are relayed.
We also cap the replicas to exit earlier in case of very low replica
count.
If an ancestry check of an annotated tag was triggered then the fetch
from that peer would fail. This is due to `graph_ahead_behind` only
working for commit-ish objects.
To allow for annotated tags, and other object that peel to commits,
the check now peels the two objects to commits before getting their
`Oid`s. The call to `graph_ahead_behind` can take these two `Oid`s
instead.
A test for annotated tags is included as part of the `examples/git`
suite in the CLI tests.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
No tests for `rad patch delete` existed.
Add a test where two peers interact on a patch and one decides to
delete the patch. It ensures that the patch is still visible to the
peer that did not delete the patch.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
To make things a little more consistent, move the `rad self --address`
command to `rad node config --addresses`, since it's more about
externally-facing configuration of the node, rather than your local user
or profile.
Implement caching for the `Issue` and `Patch` COB. This is achieved by
having the `Transaction::initial` and `Transaction::commit` methods
take a cache which can update the newly created/updated object. The
`Store::remove` method also takes the cache for removing the object
from the cache, as well as the repository.
This meant that the `*Mut` types for the respective COBs are required
to carry a cache alongside the repository store. Identities will not
be cached, and so this is always set to `NoCache` -- which performs no
caching, and always succeeds.
To perform cache reading for issues and patches, the `cache::Issues`
and `cache::Patches` traits are introduced. They capture the minimal
amount of methods that are needed for the `rad issue` and `rad patch`
CLI commands.
The implementor for each of these traits is there respective `Cache`
types, which combines their backing repository store and the SQLite
database. They both provide convenience methods:
- `create` (and `draft` for patches)
- `get_mut`
- `remove`
which can be used instead of their repository store counterparts.
All uses of the repository stores (where needed) are replaced with
`Cache` types.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
Shows you updates on local repositories.
Included here is a basic implementation of a notification system. A new
database is created and updated after every fetch, with refs that have
changed.
The `rad inbox` command can then list items in the database and delete
them.
When using `network: "test"`, we shouldn't default to any public seed
node. This change makes sure that the "preferredSeeds" configuration var
is only given a value when `network` is `main`.
Fintan Halpenny
Lorenz Leutgeb
Lorenz Leutgeb
Sekhat Temporus
cloudhead
Lars Wirzenius
Alexis Sellier