Two concurrent fetches can occur from the same peer, while also being
for a different set of references. This was seen in the
`test_outdated_sigrefs` test. Alice would have an ongoing fetch from
Eve, and concurrently, Eve will create a new issue -- thus creating a
new `RefsAt` message -- and Alice would attempt to fetch this.
The previous logic would only check that Alice is fetching from Eve
and ignore the new updates -- meaning the test could fail. However, if
the `RefsAt` are checked then the other fetch can be recognised as
new, and queue it.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
Allow some margin of error when sending multiple inventory announcements
at a time. Previously, they might end up with the same timestamp, so
only the first announcement would be processed by nodes. Now, we ensure
that the timestamp is updated.
When a peer is performing a background and foreground fetch from two
different peers, the background fetch result was being used as the
result for the foreground fetch. This results in inconsistencies,
since the foreground peer has new updates from themselves.
The reason for this issue was the fetches that were queued were not
correctly being handled, and the other peer's results were being
returned instead.
This fixes the queueing and dequeueing by ensuring that the queued
fetch occurs, passing the correct result channel. This fetch will then
be subsequently dequeued correctly and the foreground fetch obtains
the correct result.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
This test is flakey and requires further investigation. Some extra
assertions are added to ensure that the expected behaviour is
happening upon fetching. It is unknown if the fetch queue is behaving
properly, and it's suspected that some existing fetch results might be
used for queued fetches.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
If a node had removed a seeding policy and another node fetched from
it, then it would perform the fetch.
Instead, check the policy during the authorization step and prevent
the fetch is the resulting policy is `Block`.
Note that the policy will be `Block` iff the `Block` policy was
specifically set or the `Block` is the default policy for the node.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
If a delegate is missing the default branch for a project certain
computations will fail, e.g. getting the canonical mainline branch.
One prevention for corrupting data is refusing to fetch a delegate
that does not have the default branch, when the repository is a
project. It is assumed that the repository is not a project if calling
`Doc::project` returns an error, since the error variants are not
found and parsing errors.
This is only checked in the case of a delegate since non-delegates are
safe to created COBs that don't require the default branch,
e.g. creating an issue.
Previously, if a delegate's namespace was fetched and its rad/sigrefs
was behind the current state, then it would fail to perform the fetch.
Instead, when the ancestry path is checked the outcome can be used
during the update:
1. If the tip is equal, then the update can be marked as skipped.
2. If the tip is behind, then the update can be rejected, iff the
policy does not allow it.
3. If the tip is ahead, then the update is a fast-forward.
4. If the tip is diverged, then the update results in a failure, iff
the policy does not allow it.
The protocol also prunes any delegates's updates when the `rad/sigrefs`
is behind and returns an error if they are diverged.
For any non-delegates, it will simply prune them.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
Wire up the `radicle-fetch` crate to the fetch logic in the
`radicle-node` worker.
Any required traits for storage, identity, tracking, and transport
used by `radicle-fetch`.
The worker adds a new module to run the `git upload-pack` process and
pipes the `stdout`/`stdin` to the respective channels.
The `Worker` type required adding a `FetchConfig` to easily allow the
configuration of fetches and passing the signer and tracking store
location through to the fetch `Handle`.
The channels code is adapted to have a new writer that always flushes
when a call to `write_all` is made. This is necessary since the
Gitoxide code never calls `flush`, and so it will hang since no data
is sent until `flush` is called.
Since `UserInfo` is used to set up a newly cloned repository, this
changed some of the SHA1 hashes output in the CLI test examples. It
was also necessary to update the CLI's clone code to be deterministic
in its output for tests.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
Make the validation logic customisable by extracting it out into a
separate trait that can be defined by other crates.
The Repository validation logic stays the same.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
Instead of returning the unsigned references, the validation of a
repository's remotes returns the set of validation errors. The caller
is then expected to handle these validation errors appropriately, for
example, by logging them and re-raising an error.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
Introduces a new COB type to store repository identity documents.
The reason for this change is to:
1. Simplify the code, as the identity document logic resembled a COB,
yet it had custom logic. This allows existing COB code to be used for
identities.
2. Make identity document update logic more flexible, since COB actions
can be added in the future.
3. Re-purpose existing tools around COBs to work on identities, eg. `rad
cob`.
4. Unify the concept of an identity change proposal, with regular identity
changes. This means we can remove the `id.proposal` COB in favor of
using the `id` COB itself.
Notes
-----
* Each repository has one Identity COB.
* The `Proposal` COB has been repurposed into the `Identity` COB.
* Identity documents are stored as *embeds* inside the Identity COB
actions.
* The action that contains new document versions is called `revision`,
just like the Patches COB.
* The namespaced `rad/id` ref is a symbolic reference to that Identity
COB.
* The canonical `rad/id` ref is a direct reference to the commit in the
Identity COB that contains the latest *accepted* revision of the
document.
* All commands for managing identities have been folded into `rad id`.
Hence `rad delegate` and `rad edit` are removed.
* The concept of "rebasing" an identity document is gone.
* The `rad id` output has been updated to match the style of other
commands.
* When a revision has enough signatures, it is automatically adopted as
the current identity, there is no longer the need for a "commit"
action.
* When an identity revision is proposed, and the current identity has a
threshold of `1`, that identity is automatically accepted due to the
above point.
* The idea of "verifying a peer's identity branch" no longer applies, as
COBs cannot be verified one history at a time.
* Since the root commit of the Identity COB does not have a "Resource"
to point to (it would have to point to itself, which is impossible),
we've made the resource id optional for COBs.
When a fetch is user-requested, a timeout can be supplied that is passed
down to the worker.
When a fetch is service-requested, a default timeout is used.
Only return once the connection is established or failed.
It's still not perfect, as several failure scenarios are not handled,
but it's an improvement.
As a starting point, we populate the address book (if empty) with some
bootstrap nodes which are trusted.
To prevent these nodes being connected to during tests (including e2e
tests), we create a new constructor for `Config` that is used in tests.
Note that a `cfg(test)` check is not enough given that e2e tests don't
have that set.
This is a fairly substantial change, which adds a new configuration file
to the user's `$RAD_HOME` that includes node configuration, including
the alias, and also makes node aliases required.
When running `rad auth`, the user is now prompted for an alias, which
defaults to `$USER`.
When running `rad self`, the alias is now shown.
If the user runs radicle without a config file, the defaults are loaded,
and `$USER` is used as the alias.
We fix the failing fetch tests by:
1. Not verifying our own refs, unless we're cloning, since we're
otherwise not fetching our own refs.
2. Always force-fetching `sigrefs` from remotes into the staging copy.
3. Making sure that sigref updates are fast-forward before transfering
the remote into the production copy.
Ensure that non-existent tracking relationships do not affect
fetching.
This is tested by first performing a clone, by `bob` from `alice`,
after `bob` tracks a remote that does not exist. `alice` creates an
issue which is then fetched by `bob` -- to ensure the fetching
scenario is also tested.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
Instead of returning the namespaces that we fetched with,
we return the remotes that were actually fetched from.
This can differ if some trusted peers were not available
on the remote node.
Previously, if that was the case, it would cause errors
since the remotes were looked up and that lookup failed.
With this change, we not only verify the signed refs before
the repository "transfer", but also validate that all refs in the
repo are signed and there is no discrepancy with the signed refs
file.
When a repository does not yet exist during a fetch, i.e. a clone,
then only delegates are being fetched.
Augment Namespaces::Many to hold the trusted and optional delegate peers
separately. Doing so allows the construction of the variant without
the repository existing, but tracking relationships existing.
This variant can then be used to build refspecs for both the trusted
peers and delegates when doing a cloning fetch.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
Previously, if a peer created a project which was replicated and then
moved to another setup, ultimately deleting that project, then they
would not be able to fetch it again since their remote namespace would
be excluded from the fetch.
This test confirms that a peer can remove their project and clone it
again from a peer that has replicated the project. This is simulated
by directly removing the project directory in the test and fetching
the project from the other peer.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
With the latest fetch updates, the test for fetching trusted remotes
can be enabled.
The caveat is that its expected behaviour is to fetch the delegate
upon a clone and fetch all trusted peers on a subsequent
fetch. Fetching all trusted peers is left as a follow-up task.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
The previous fetching logic had some pitfalls regarding its logic:
* Fetched 'rad' references could end up not being verifiable and leaving
the storage in a corrupt state
* Forced pushes would be prohibited, since they would be rejected by
the client side due to the refspecs being used.
The new approach is to stage a fetch in 2 steps. The first step is to
fetch the relevant `rad/id` and `rad/sigrefs` for verification. For
all remotes that are verified, fetch all references listed in their
respective `rad/sigrefs`, using the `+` marker in the refspec,
allowing to update for any force pushes -- note that this becomes a
safe operation since the `sigrefs` are signed by the remote that
created them.
To prevent unverifiable 'rad' references from polluting storage, all
fetch negotiations are done in a staging repository set up in a
temporary directory. All successful fetches are then transferred,
using the 'file://' protocol, from the temporary directory into the
radicle storage.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
To ensure the behaviour of ignoring references that are owned by the
local peer a test case is added.
The case has two peers: alice and bob, where alice creates a new
repository. bob fetches to be up-to-date with alice. alice, in turn,
creates an issue and fetches from bob -- which should be successful.
If the refspec:
^refs/namespaces/<alice>/*
was not in place, her fetch from bob would fail since the signed
references would not be a fast-forward.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
Adds a test for fetching from trusted delegates.
Currently marked as failing. When the fetch is initiated from a client
that does not have the repository then it will fetch all namespaces,
instead of the trusted set.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
The method `track_repo` only accepted the RID as the argument for
tracking.
Instead, allow passing Scope as an argument for tracking a
repository.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
Previously, the seeds information would only consist of the connected
NodeIds.
Improve on this by adding any disconnected and fetching seeds as
well. A seed can either be disconnected, connected, or fetching --
where fetching implies connected.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
Rust 1.67 was announced[0]. Update the necessary files for running
1.67 and fix the clippy suggestions for string interpolation.
[0]: https://blog.rust-lang.org/2023/01/26/Rust-1.67.0.html
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
Fintan Halpenny
cloudhead
Alexis Sellier
Alexis Sellier