This change updates to the latest radicle-git-ext, which required a
few changes within the radicle family of crates.
One set of changes is that the radicle-git-ext crate subsumes
git-commit, git-trailers, and git-ref-format -- so all those imports
go through radicle-git-ext.
The commit code requires that parent `Oid`s point to commits, which
made some tests fail. One of the reasons for these failures is that
the patch tests used fabricated `Oid`s and so the verification would
fail. This is fixed by allowing the test context code to create
`PatchRequest`s that create valid commits.
The other reason for failure was that the identity `Action` passed an
`Oid` that pointed to a blob. This is fixed by removing that code and
making a note of it in documentation.
The final major change was moving the Refspec type into git-ref-format
and removing the AsRefspecs trait. The trait was not required and
could its usage could be replaced by simpler code.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
When attempting to connect to a peer, and when succeeding, write the timestamp
to the database.
This will allow us to better select addresses and not retry peers to
soon.
On macOS, shutting down a socket returns an error if the socket is
already closed. We don't consider that a problem, as it just returns
`Ok(())` on Linux.
We fix the failing fetch tests by:
1. Not verifying our own refs, unless we're cloning, since we're
otherwise not fetching our own refs.
2. Always force-fetching `sigrefs` from remotes into the staging copy.
3. Making sure that sigref updates are fast-forward before transfering
the remote into the production copy.
Ensure that the production repository is validated after fetching and
performing deletions. This is to highlight any inconsistencies that
may show up between fetching and deleting references.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
Given the scenario of 3 peers, A, B, and C each having a project R. If
peer A has B's fork, but C does not, then when peer A attempts to
fetch from C the fetch will be rejected since Git will fail when a
refspec that is asked for does not exist on the remote side.
To avoid this, the fetch logic is changed so that the client first calls
ls-remote to the remote side, which results in the references that the
remote has which the client is interested in. The follow-up fetch then
uses these refs as the refspecs -- which should succeed unless there
was a race for a deletion on the remote side.
The rest of the verification logic stays the same, so the storage
should still be in a working state before transferring from the staged
repository to the production repository.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
Ensure that non-existent tracking relationships do not affect
fetching.
This is tested by first performing a clone, by `bob` from `alice`,
after `bob` tracks a remote that does not exist. `alice` creates an
issue which is then fetched by `bob` -- to ensure the fetching
scenario is also tested.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
We were allowing remotes to be constructed via a function that didn't
check its inputs.
This is now fixed by moving the public key into the `SignedRefs` struct.
It's currently possible for a channel to be blocked forever while
waiting for data, if the remote peer keeps the connection alive.
This can occupy a worker indefinitely.
With these timeouts, we ensure that workers exit if there is no
activity after a certain amount of time.
This command announces refs to peers and waits for them to be
in sync.
Adds a subscribe method to `Handle` so that we can get the events
from the seed to allow us to confirm that the configured seeds fetched
from us.
By having the keep-alive delta be the same as the ping timeout,
we essentially disconnected peers right after we sent them our
first ping.
This fixes that issue by setting the ping timeout to double the
keep-alive delta.
Instead of returning the namespaces that we fetched with,
we return the remotes that were actually fetched from.
This can differ if some trusted peers were not available
on the remote node.
Previously, if that was the case, it would cause errors
since the remotes were looked up and that lookup failed.
With this change, we not only verify the signed refs before
the repository "transfer", but also validate that all refs in the
repo are signed and there is no discrepancy with the signed refs
file.
If `git-fetch` returns a non-zero code on exit, we keep going in the
case of an existing repository.
The reason is that `git-fetch` will return an error code if any ref
is rejected during the fetch. But this can happen if someone
pushes conflicting `rad/sigrefs` branches for example, and the node
already has a copy of that branch.
Instead of parsing the worker requests and responses, which was causing issues
around packfile transmission, we use threads and pass the bytes
transparently.
Previously we were announcing inventories all the time, due to
relying on the timestamp changing on insertion.
We change this to only announce our inventory when it actually changes.
Previously, during the "transfer" stage of a fetch, we used `force:
false`, which didn't allow refs to be force-updated.
This change makes it possible to fetch force-updated refs, but ensures
that the sigrefs branch is never force-updated.
To improve the reliability and flexibility of the protocol, we introduce
multiplexing over peer connections. This involves a new `Frame` type
that carries a stream-id and payload.
Three stream types are made available:
1. Control
2. Gossip
3. Git
This change brings the following improvements:
* Removed need to queue fetch requests
* Removed need to queue gossip messages
* Removed need for `Fetch` and `FetchOk` messages
* Service doesn't need to know about inbound fetches
* Removed one round-trip for fetch negotiation
* Removed special `done` git packet
* Removed session logic and state around Git/Fetch protocols
* Removed code around upgrading/downgrading transport
* Worker in responder mode is able to process any number of fetches
* Connections support any number of concurrent fetches
We had to introduce a few extra things however to make it all work:
* A `VarInt` type for variable-length integers, since we want the frames
to be lightweight.
* A custom "tunnel" implementation, since we couldn't use the existing
one anymore.
Overall the change removes more complexity than it adds, while improving
the protocol along the way.
Support users with systemd integration by providing a systemd service
file. The file includes basic instructions for how to adapt to the
users context.
There are two scenarios it would be run in, as a system service, or as a
user service. The scenario as a user service is problematic.
`radicle-node` relies on ssh-agent or a clear text RAD_PASSPHRASE.
This fixes an issue that came up in tests where we would be reading
and writing to the database at the same time, and it would fail with
an error due to the lock being contended on.
The One variant caused many paint points for fetching logic, where it
was not necessary. It was only constructed in one place, which could
be replaced by using the variant that holds a set of keys.
Remove the Namespaces::One variant and rename Namespaces::Many to
Namespaces::Trusted.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
When a repository does not yet exist during a fetch, i.e. a clone,
then only delegates are being fetched.
Augment Namespaces::Many to hold the trusted and optional delegate peers
separately. Doing so allows the construction of the variant without
the repository existing, but tracking relationships existing.
This variant can then be used to build refspecs for both the trusted
peers and delegates when doing a cloning fetch.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
Previously, if a peer created a project which was replicated and then
moved to another setup, ultimately deleting that project, then they
would not be able to fetch it again since their remote namespace would
be excluded from the fetch.
This test confirms that a peer can remove their project and clone it
again from a peer that has replicated the project. This is simulated
by directly removing the project directory in the test and fetching
the project from the other peer.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
With the latest fetch updates, the test for fetching trusted remotes
can be enabled.
The caveat is that its expected behaviour is to fetch the delegate
upon a clone and fetch all trusted peers on a subsequent
fetch. Fetching all trusted peers is left as a follow-up task.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
The previous fetching logic had some pitfalls regarding its logic:
* Fetched 'rad' references could end up not being verifiable and leaving
the storage in a corrupt state
* Forced pushes would be prohibited, since they would be rejected by
the client side due to the refspecs being used.
The new approach is to stage a fetch in 2 steps. The first step is to
fetch the relevant `rad/id` and `rad/sigrefs` for verification. For
all remotes that are verified, fetch all references listed in their
respective `rad/sigrefs`, using the `+` marker in the refspec,
allowing to update for any force pushes -- note that this becomes a
safe operation since the `sigrefs` are signed by the remote that
created them.
To prevent unverifiable 'rad' references from polluting storage, all
fetch negotiations are done in a staging repository set up in a
temporary directory. All successful fetches are then transferred,
using the 'file://' protocol, from the temporary directory into the
radicle storage.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
The tracked nodes include nodes marked tracked in the db, and
the delegates and `Namespaces::All` if it is a new repo.
Signed-off-by: Han Xu <keepsimple@gmail.com>
First, we make sure to always attempt reconnection to persistent peers,
no matter the reason.
Second, we don't re-attempt connections when an inbound peer
disconnects. This minimizes the chance of connections crossing.
Third, we clean up `session::Error`, and don't use it to signal a
missing session, since it doesn't really belong there.
1. We were announcing inventory every 30 seconds. This is now changed
to 1 hour.
2. We were using a variable to track whether our inventory has been
announced to the network, but this var was set based on what we
tracked, not the inventory we had. This var has been removed.
Better handle case where the connection to the daemon fails, in the
responder scenario.
Instead of trying to read the `done` packet once, we loop until we've
read it. This handles a scenario where the uploader fails before the
git protocol is initiated, and the fetcher is still sending git packets.
This change ensures that on `Subscribe`, we only relay the latest
announcement of each node, per announcement category.
Previously, we would relay all announcement, while only the last one
was relevant.
If there is an unexpected error, eg. `BrokenPipe` in the upload process
process of the worker, we may have more than four bytes to read from the
remote socket. This change ensures that we don't attempt to read more
into a buffer of size `4`.
For whatever reason, we weren't using the deserializer for message
decoding. This meant that if a message came in two pieces, it would
cause decoding problems, since the first piece would be discarded.
It's possible that the daemon or stream closes the git connection
without reading the final `done` packet. If this is the case, we
try to read it after either of the connections is closed.
This may potentially mitigate the issue where the node then receives
the `done` packet while in gossip mode, and disconnects the remote
for misbehavior.
Due to never removing peers from the peer map in the wire
protocol module, it was possible for there to be multiple
connected peers with the same Node ID. Some might be in a
"connected" state, while others might be "disconnected".
This caused a bug where a peer, although connected would
be considered disconnected by the wire module, because
when searching for a peer with that Node ID, the disconnected
one would return first.
To fix this, we make sure to remove peers from the map once
the connection is dropped.
It's not a good idea to have fallible operations run after `FetchOk`
is received, since there's no easy way to communicate the error
to the remote at that point. What would then happen is that the fetch
just "hangs", since nothing is there to cancel it.
Hence, we compute namespaces earlier in the process, and store the
namespaces in the session. If there is a failure, the fetch is
never initiated.
It was possible to trigger multiple connection attempts to the same
peer, since we didn't track connection attempts that hadn't yet
returned to the service via `Service::attempted`.
With this change, we add an extra `Initial` state to sessions, so
that redundant connections are not attempted when one is already under
way.
The radicle-node was forced into using Scope::Trusted for its default
scope.
Allow the Scope to be specified at the command line using a
`--tracking-scope` option.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
Periodically, make sure we try to fetch inventory that is tracked
but missing from local storage.
We decide not to fetch immediately on `TrackRepo`, as this limits
the command's flexibility. For example, when `rad clone` is used,
we want to be able to fetch manually after tracking.
This avoids race conditions, especially in tests, where we try to
connect to the node right after spawning it, but since the
`Runtime::run` function is often called in a different thread, the
socket is not created in time.
It's possible for a peer to disconnect before responding with `FetchOk`.
In that case, we make sure to return to any pending fetch request with
an error.
To ensure the behaviour of ignoring references that are owned by the
local peer a test case is added.
The case has two peers: alice and bob, where alice creates a new
repository. bob fetches to be up-to-date with alice. alice, in turn,
creates an issue and fetches from bob -- which should be successful.
If the refspec:
^refs/namespaces/<alice>/*
was not in place, her fetch from bob would fail since the signed
references would not be a fast-forward.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
Adds a test for fetching from trusted delegates.
Currently marked as failing. When the fetch is initiated from a client
that does not have the repository then it will fetch all namespaces,
instead of the trusted set.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
To determine the correct Namespaces to use, the repo Policy and Scope
are consulted.
If the policy is Block then this is a failure state and is reported to
the other end.
Otherwise, if the policy is Track, then the scope is checked. For the
Trusted Scope then the Namespaces should consist of all tracked nodes
and delegates for the given repository. However, the repository might
not exist and so it is necessary for the whole repository to be
fetched.
For the All Scope, all Namespaces are fetched.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
The fetch implementation will potentially fetch the local operator's
remote, which is unnecessary and could result in deleted data.
Fix this by threading the local operator's public key and add an
ignore refspec to the fetch arguments, i.e.
^refs/namespaces/<local>/*
The implementation also strictly fetches all or one namespace. This is
improved upon by introducing a new variant for specifying a set of
namespaces.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
The method `track_repo` only accepted the RID as the argument for
tracking.
Instead, allow passing Scope as an argument for tracking a
repository.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
Instead of returning tuples for `repo_entry` and `node_entry`, return
the `Repo` and `Node` types, respectively.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
The tracking configuration has no way of consulting for a configured
scope.
This change allows the storing of this scope inside the tracking
`Config` -- currently unused.
The default Scope is chosen as Trusted, so that nodes do not
implicitly track any other node. If this behaviour is desired -- for
example, for seed nodes -- then All should be chosen.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
The DelegatesOnly scope is very limited and causes more confusion than
it solves.
The Trusted scope subsumes DelegatesOnly since it implies delegates
along with any tracked nodes. This feels more natural to use and so
DelegatesOnly is removed -- leaving only Trusted and All.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
Previously the alias field was simply a string which could possibly be
empty. Some checks were made to see if this string was empty.
Instead, make the logic more obvious by making the alias field
optional.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
The `routes_to` function can loop infinitely during test.
To prevent this from happening a number of tries variable is added. It
checks that the number of tries does not exceed 30, which gives the
function about 3 seconds to complete -- due to the call to
`thread::sleep` using 100ms.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
The MockStorage holds an inventory of repositories and their
identities but when asking for a MockRepository, none of this
information is transferred.
Add `id` and `doc` fields to MockRepository for defining some of its
functionality.
In addition, add a generator for a nonempty MockStorage to use in
tests that will expect the storage to be populated.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
A fetch is a client-server interaction and so there are two sides to
this interaction. These sides are generally referred to as receive --
the client side -- and upload -- the server side.
The `Fetch` type indicated whether it was doing a receive or upload by
using a field `initiated: bool`. However, this can be confusing when
you pair it with the `namespaces: Namespaces` field, since only the
receive side should be sending what refspecs it wants and the upload
side responds with those matching refspecs.
A better way to represent this is using an enum for which direction
the fetch is being considered, where the receive side contains the
Namespaces. To borrow from Noise, the variant names are 'Initiator'
for the client and 'Responder' for the server.
The resulting code then makes decisions based on what variant of the
enum was passed.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
It is possible for a pair of nodes to concurrently ask each other
to fetch some repo. These "crossing" requests are problematic,
because the nodes end up waiting for each other to reply, and
the reply never comes.
With this change, the node pair agrees on proceeding with one of
the fetches, while canceling the other.
Previous to this change, we were attempting to gossip after a successful
fetch, but before the session was switched back to the gossip protocol.
This meant that messages were queued for a short amount of time. With
this change, messages should be sent straight away.
The connect command was unimplemented for both Node and the control
socket.
This wires up the connect functionality in both places.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
The routing function for Node was not implemented and there were no
functions for getting any tracking information.
This implements routing, tracked_repos, and tracked_nodes. To
accomplish this it was necessary to add associated types to the
Handler trait so that implementations could differ in the types they
use for returning this data, i.e. the control socket needs to use a
channel while the Node needs to return serializable data.
Currently, the Node uses a `Vec` but it would be more beneficial to
return an Iterator directly to ensure we are not always hitting the
heap.
To allow for the tracking types to be used across radicle-node (the
crate) and in radicle::node (the module), it was necessary to move
the types to the common place, i.e. radicle::node. At the same time,
new structs were added to make it easier to refer to tracked repos vs
tracked nodes.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
Before this change, the identity document would be loaded for the
COB signer. This meant that a fork would be needed for that signer.
After this change, it's no longer necessary, since the identity doc
head is computed from available remotes.
While making those changes, it was also apparent that some of the
identity-related functions had bad names and error types, this
was fixed as well.
Previously, the seeds information would only consist of the connected
NodeIds.
Improve on this by adding any disconnected and fetching seeds as
well. A seed can either be disconnected, connected, or fetching --
where fetching implies connected.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
The Handle trait defines a FetchResult associated type. This type is
always instantiated to be the FetchResult enum in radicle::node.
Remove the associated type to simplify the trait.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
If a path was passed to `Home` in non-canonical form it would not use
the right path for any initialisation or use.
The aim is to use the `canonicalize` function for paths. However, this
may return an error if none of the paths exist. To ensure that the
path exists the `new` constructor for `Home` creates the directory and
canonicalizes the path.
Since `new` now initialises the home directory, it would be useful if
it also initialised all the necessary subdirectories -- which removes
the need for the `init` method.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
When a specific tracking policy isn't found, we allow setting a default.
To preserve existing behavior, the default is `Block`, ie. if a node
isn't tracked explicitly, it is blocked.
However, this change allows for using `Track` as the default, which
is a way to track everything without having to know the entities in
advance.
The last commit of a newly cloned repository should remain untouched by
making `Node::project()` use a repository fixture instead of a randomly
generated one.
Signed-off-by: Slack Coder <slackcoder@server.ky>
* Read git request line before starting daemon, so that we don't leave
git data in the socket read buffer in case of an error connecting to
the daemon.
* Send a response on the fetch channel even if we disconnect the remote.
* Don't send `done` packet if there's an error; it won't be expected.
We weren't updating the sessions map when a persistent peer
was re-attempted. It would then panic when `connected` was called,
because the state would be `Disconnected`.
Rust 1.67 was announced[0]. Update the necessary files for running
1.67 and fix the clippy suggestions for string interpolation.
[0]: https://blog.rust-lang.org/2023/01/26/Rust-1.67.0.html
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
Alexis Sellier
Fintan Halpenny
xphoniex
Slack Coder
Alexis Sellier
Han Xu
Adam Szkoda
Dr Maxim Orlovsky