This change ensures that on `Subscribe`, we only relay the latest
announcement of each node, per announcement category.
Previously, we would relay all announcement, while only the last one
was relevant.
If there is an unexpected error, eg. `BrokenPipe` in the upload process
process of the worker, we may have more than four bytes to read from the
remote socket. This change ensures that we don't attempt to read more
into a buffer of size `4`.
For whatever reason, we weren't using the deserializer for message
decoding. This meant that if a message came in two pieces, it would
cause decoding problems, since the first piece would be discarded.
It's possible that the daemon or stream closes the git connection
without reading the final `done` packet. If this is the case, we
try to read it after either of the connections is closed.
This may potentially mitigate the issue where the node then receives
the `done` packet while in gossip mode, and disconnects the remote
for misbehavior.
Due to never removing peers from the peer map in the wire
protocol module, it was possible for there to be multiple
connected peers with the same Node ID. Some might be in a
"connected" state, while others might be "disconnected".
This caused a bug where a peer, although connected would
be considered disconnected by the wire module, because
when searching for a peer with that Node ID, the disconnected
one would return first.
To fix this, we make sure to remove peers from the map once
the connection is dropped.
It's not a good idea to have fallible operations run after `FetchOk`
is received, since there's no easy way to communicate the error
to the remote at that point. What would then happen is that the fetch
just "hangs", since nothing is there to cancel it.
Hence, we compute namespaces earlier in the process, and store the
namespaces in the session. If there is a failure, the fetch is
never initiated.
It was possible to trigger multiple connection attempts to the same
peer, since we didn't track connection attempts that hadn't yet
returned to the service via `Service::attempted`.
With this change, we add an extra `Initial` state to sessions, so
that redundant connections are not attempted when one is already under
way.
The radicle-node was forced into using Scope::Trusted for its default
scope.
Allow the Scope to be specified at the command line using a
`--tracking-scope` option.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
Periodically, make sure we try to fetch inventory that is tracked
but missing from local storage.
We decide not to fetch immediately on `TrackRepo`, as this limits
the command's flexibility. For example, when `rad clone` is used,
we want to be able to fetch manually after tracking.
This avoids race conditions, especially in tests, where we try to
connect to the node right after spawning it, but since the
`Runtime::run` function is often called in a different thread, the
socket is not created in time.
It's possible for a peer to disconnect before responding with `FetchOk`.
In that case, we make sure to return to any pending fetch request with
an error.
To ensure the behaviour of ignoring references that are owned by the
local peer a test case is added.
The case has two peers: alice and bob, where alice creates a new
repository. bob fetches to be up-to-date with alice. alice, in turn,
creates an issue and fetches from bob -- which should be successful.
If the refspec:
^refs/namespaces/<alice>/*
was not in place, her fetch from bob would fail since the signed
references would not be a fast-forward.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
Adds a test for fetching from trusted delegates.
Currently marked as failing. When the fetch is initiated from a client
that does not have the repository then it will fetch all namespaces,
instead of the trusted set.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
To determine the correct Namespaces to use, the repo Policy and Scope
are consulted.
If the policy is Block then this is a failure state and is reported to
the other end.
Otherwise, if the policy is Track, then the scope is checked. For the
Trusted Scope then the Namespaces should consist of all tracked nodes
and delegates for the given repository. However, the repository might
not exist and so it is necessary for the whole repository to be
fetched.
For the All Scope, all Namespaces are fetched.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
The fetch implementation will potentially fetch the local operator's
remote, which is unnecessary and could result in deleted data.
Fix this by threading the local operator's public key and add an
ignore refspec to the fetch arguments, i.e.
^refs/namespaces/<local>/*
The implementation also strictly fetches all or one namespace. This is
improved upon by introducing a new variant for specifying a set of
namespaces.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
The method `track_repo` only accepted the RID as the argument for
tracking.
Instead, allow passing Scope as an argument for tracking a
repository.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
Instead of returning tuples for `repo_entry` and `node_entry`, return
the `Repo` and `Node` types, respectively.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
The tracking configuration has no way of consulting for a configured
scope.
This change allows the storing of this scope inside the tracking
`Config` -- currently unused.
The default Scope is chosen as Trusted, so that nodes do not
implicitly track any other node. If this behaviour is desired -- for
example, for seed nodes -- then All should be chosen.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
The DelegatesOnly scope is very limited and causes more confusion than
it solves.
The Trusted scope subsumes DelegatesOnly since it implies delegates
along with any tracked nodes. This feels more natural to use and so
DelegatesOnly is removed -- leaving only Trusted and All.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
Previously the alias field was simply a string which could possibly be
empty. Some checks were made to see if this string was empty.
Instead, make the logic more obvious by making the alias field
optional.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
The `routes_to` function can loop infinitely during test.
To prevent this from happening a number of tries variable is added. It
checks that the number of tries does not exceed 30, which gives the
function about 3 seconds to complete -- due to the call to
`thread::sleep` using 100ms.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
The MockStorage holds an inventory of repositories and their
identities but when asking for a MockRepository, none of this
information is transferred.
Add `id` and `doc` fields to MockRepository for defining some of its
functionality.
In addition, add a generator for a nonempty MockStorage to use in
tests that will expect the storage to be populated.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
A fetch is a client-server interaction and so there are two sides to
this interaction. These sides are generally referred to as receive --
the client side -- and upload -- the server side.
The `Fetch` type indicated whether it was doing a receive or upload by
using a field `initiated: bool`. However, this can be confusing when
you pair it with the `namespaces: Namespaces` field, since only the
receive side should be sending what refspecs it wants and the upload
side responds with those matching refspecs.
A better way to represent this is using an enum for which direction
the fetch is being considered, where the receive side contains the
Namespaces. To borrow from Noise, the variant names are 'Initiator'
for the client and 'Responder' for the server.
The resulting code then makes decisions based on what variant of the
enum was passed.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
It is possible for a pair of nodes to concurrently ask each other
to fetch some repo. These "crossing" requests are problematic,
because the nodes end up waiting for each other to reply, and
the reply never comes.
With this change, the node pair agrees on proceeding with one of
the fetches, while canceling the other.
Previous to this change, we were attempting to gossip after a successful
fetch, but before the session was switched back to the gossip protocol.
This meant that messages were queued for a short amount of time. With
this change, messages should be sent straight away.
The connect command was unimplemented for both Node and the control
socket.
This wires up the connect functionality in both places.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
The routing function for Node was not implemented and there were no
functions for getting any tracking information.
This implements routing, tracked_repos, and tracked_nodes. To
accomplish this it was necessary to add associated types to the
Handler trait so that implementations could differ in the types they
use for returning this data, i.e. the control socket needs to use a
channel while the Node needs to return serializable data.
Currently, the Node uses a `Vec` but it would be more beneficial to
return an Iterator directly to ensure we are not always hitting the
heap.
To allow for the tracking types to be used across radicle-node (the
crate) and in radicle::node (the module), it was necessary to move
the types to the common place, i.e. radicle::node. At the same time,
new structs were added to make it easier to refer to tracked repos vs
tracked nodes.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
Before this change, the identity document would be loaded for the
COB signer. This meant that a fork would be needed for that signer.
After this change, it's no longer necessary, since the identity doc
head is computed from available remotes.
While making those changes, it was also apparent that some of the
identity-related functions had bad names and error types, this
was fixed as well.
Previously, the seeds information would only consist of the connected
NodeIds.
Improve on this by adding any disconnected and fetching seeds as
well. A seed can either be disconnected, connected, or fetching --
where fetching implies connected.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
The Handle trait defines a FetchResult associated type. This type is
always instantiated to be the FetchResult enum in radicle::node.
Remove the associated type to simplify the trait.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
If a path was passed to `Home` in non-canonical form it would not use
the right path for any initialisation or use.
The aim is to use the `canonicalize` function for paths. However, this
may return an error if none of the paths exist. To ensure that the
path exists the `new` constructor for `Home` creates the directory and
canonicalizes the path.
Since `new` now initialises the home directory, it would be useful if
it also initialised all the necessary subdirectories -- which removes
the need for the `init` method.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
When a specific tracking policy isn't found, we allow setting a default.
To preserve existing behavior, the default is `Block`, ie. if a node
isn't tracked explicitly, it is blocked.
However, this change allows for using `Track` as the default, which
is a way to track everything without having to know the entities in
advance.
The last commit of a newly cloned repository should remain untouched by
making `Node::project()` use a repository fixture instead of a randomly
generated one.
Signed-off-by: Slack Coder <slackcoder@server.ky>
* Read git request line before starting daemon, so that we don't leave
git data in the socket read buffer in case of an error connecting to
the daemon.
* Send a response on the fetch channel even if we disconnect the remote.
* Don't send `done` packet if there's an error; it won't be expected.
We weren't updating the sessions map when a persistent peer
was re-attempted. It would then panic when `connected` was called,
because the state would be `Disconnected`.
Rust 1.67 was announced[0]. Update the necessary files for running
1.67 and fix the clippy suggestions for string interpolation.
[0]: https://blog.rust-lang.org/2023/01/26/Rust-1.67.0.html
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
This change makes sure that nodes don't gossip announcements back to
the announcers.
Since the tests were relying on this mechanism to populate the routing
table, we had to change a few more things, such as adding the inventory
to the node's routing table.
Signed-off-by: Alexis Sellier <self@cloudhead.io>
The tests were flaky because the routes are sampled at a moment
when gossip could still be happening. Instead, we check the
local inventory directly to build the expected routing table.
Signed-off-by: Alexis Sellier <self@cloudhead.io>
By allowing the node to run using a `MockSigner`, we speed up tests
significantly, since PBKDF2 doesn't need to run.
Signed-off-by: Alexis Sellier <self@cloudhead.io>
Instead of only being interested in messages from "now", we request
messages from one hour in the past up to now. This lets nodes catch
up on state more quickly, without having to wait for new announcements.
Signed-off-by: Alexis Sellier <self@cloudhead.io>
We shouldn't be marking attempted peers as "connected". We should wait
for the connection to be properly established.
Signed-off-by: Alexis Sellier <alexis@radicle.xyz>
There are few unrelated changes in this commit that I wasn't able to
split out, mostly due to laziness, so I'll list them here:
* The node startup logic is moved out of the main function, and into a
dedicated functionin `client.rs`.
* There is a tentative worker implementation that is untested, and
handles git replication both ways. It's likely lacking in various ways
but I wanted to get it in there anyway.
* `Address` was moved from `radicle-node` to `radicle`, because it's
needed to send `Connect` commands from the node handle.
* The `netservices` dependency was updated.
* Finally, there is a first end-to-end node test, which currently
doesn't use NoiseXK, but uses the new reactor in `netservices`.
Signed-off-by: Alexis Sellier <self@cloudhead.io>
This is a large change that introduces transport encryption via
a Noise_XK handshake. The inner `Service` now works directly
with `NodeId` and doesn't need to interact with socket addresses.
The transport logic takes place in `wire::transport`, and the
established sessions are passed down to `Service`.
Note that the i/o implementation in the `poll-reactor` crate still
needs testing, and there is still work to be done to integrate the
Git transport as part of this architecture.
Signed-off-by: Alexis Sellier <alexis@radicle.xyz>
Use the BoundedVec to limit the supported number of refs in
RefAnnouncement messages. The size (235) is the maximum supported by
message wire size.
To reduce the complexity of the changes, BoundedVec is used again
instead of using a new bounded Refs or BTreeMap. This requires
defining new Decode/Encode logic for Ref's element types, and
workarounds for arbitrary instances.
Signed-off-by: Slack Coder <slackcoder@server.ky>
Allow setting a test peer's local time to avoid slight race conditions in
tests. The race condition's happen because peers created within a test may
unexpectedly have slightly different local times.
To support this, a struct is used as argument to peer::config(..). The
linter 'cargo clippy' will complain if instead of using a struct, the local
time is passed as an additional parameter in config()'s function call.
Signed-off-by: Slack Coder <slackcoder@server.ky>
The tracking command was tracking a project when it should
have been tracking peers.
We also add the ability to pass an alias when tracking from
the CLI.
Signed-off-by: Alexis Sellier <alexis@radicle.xyz>
Sometimes it is more efficient to perform multiple operations at once
and store those in the collaborative object graph together.
Change the contents to be a non-empty sequence of byte sequences,
i.e. NonEmpty<Vec<u8>>. These are stored as blobs in a git tree in
order by their index in the non-empty sequence. They are, in turn,
retrieved in order of their original index.
Change the conversion of an `EntryWithClock` to return a non-empty
sequence of `Op`s, which are in turn used to apply many changes in
each `from_history` implementation.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
The latter is a fork of the former, which adds const generics
and a bunch of other quality of life improvements that aren't
in the original crate due to it being unmaintained.
Signed-off-by: Alexis Sellier <alexis@radicle.xyz>
Borrow documentation from Rusts standard library to document BoundedVec's
functions, and add documentation tests where appropriate.
And make it's deref trait return a slice instead of a Vec, which is more
common in rust.
Signed-off-by: Slack Coder <slackcoder@server.ky>
Restrict message size decoding by limiting Message's vector size.
Introduce the BoundedVec type to do this on the type level for
convenience and clarity.
Signed-off-by: Slack Coder <slackcoder@server.ky>
Send the node's configured external addresses instead of its listenning
addresses. External addresses are ones which the service is reachable
from the outside world, the listening addresses can be the wrong context
(local area network, or 0.0.0.0).
Signed-off-by: Slack Coder <slackcoder@server.ky>
This was causing an issue in rust-analyzer, and probably would fail
in other ways via cargo. In any case it wasn't a good idea.
Signed-off-by: Alexis Sellier <alexis@radicle.xyz>
Avoid wasting persistent memory by cleaning up stale routing entries
when the routing table is considered constrained. Limit pruning to
improve resiliency in the face of long intervals between service
downtimes.
Signed-off-by: Slack Coder <slackcoder@server.ky>
Keep the node's understanding of project hosting location current by
updating how Inventory Announcement messages are processed.
The Inventory Announcements contain a node's entire listing of projects.
Replace the routing table information for the announcer with those from the
message.
Signed-off-by: Slack Coder <slackcoder@server.ky>
Address a FIXME in service.process_inventory() by using the announcer's
time instead of the service's.
Signed-off-by: Slack Coder <slackcoder@server.ky>
Fix a problem from a recent commit which caused tests in the tests
module not to run.
The list of modules in 'lib.rs' should use the 'tests' module's new
location.
Signed-off-by: Slack Coder <slackcoder@server.ky>
Avoid redundantly emitting Node Announcements about it's publicly
available addresses when it has none.
Signed-off-by: Slack Coder <slackcoder@server.ky>
Include the node's IP address in the Node's announcements. Source it
from the service's configuration which can assumed to be populated with
default values when available.
Signed-off-by: Slack Coder <slackcoder@server.ky>
Try having a number of outbound connections to other Radicle nodes at any
given time. The number has been defined earlier as TARGET_OUTBOUND_PEERS.
All network nodes are treated equally, there is no preference.
Attempts to keep connected happen at set intervals (IDLE_INTERVAL) or when
a node disconnects.
Signed-off-by: Slack Coder <slackcoder@server.ky>
In interactive (user) contexts, use `try_sign` instead of `sign`.
We also revisit the `Profile` by adding a `signer` method that can
fail, eg. if we couldn't connect to ssh-agent.
Finally, we replace `UnsafeSigner` with `MemorySigner`, which makes
use of `ssh::Keystore`.
Signed-off-by: Alexis Sellier <alexis@radicle.xyz>
To support ssh-agent, add a signing method that can fail.
Also cleanup usage so that we always accept signer references.
Signed-off-by: Alexis Sellier <alexis@radicle.xyz>
Han Xu
Alexis Sellier
xphoniex
Alexis Sellier
Fintan Halpenny
Dr Maxim Orlovsky
Slack Coder
Dr Maxim Orlovsky
Adam Szkoda
Dr Maxim Orlovsky