This prepares the crate versions for a new release. The version bumps were
determined by the `cargo-semver-checks` tool.
However, there were some issues with the `radicle-fetch` crate – where when
attempting to check it, it was unable to compile due to the previous version
expecting the kind of `Doc` with a generic parameter. I believe this is due to
the version for `radicle` being specified as simply `0`. This prompted changing
the version specifications to use `major.minor` for all the internal crates.
This, for example, would mean that `radicle-fetch` that relies on `0.15` can use
`>=0.15.0,<0.16.0` – allowing any patch versions but must require a minor
version bump.
This change is to help separate the fact the namespacing by a `NodeId` is a
Radicle specific concern. The identifier can, in theory, be any kind of path
component, as long it is valid in Git.
The motivation for the change is to help separate the idea of a device's
`NodeId` vs an agent being an author – for future agent repository work.
The motivation of this change is to move away from tying signing to be
specifically for ed25519. The reason being that the protocol will want move
towards two different kinds of signing – the node signing artifacts, and an
author signing artifacts.
This change captures the former, node signing, by introducing a `Device` type
that is the `NodeId` – a PublicKey underneath the hood – and a signing
mechanism. This allows the replacement of the `Signer` trait being used – which
always assumed a `PublicKey`. Instead, the `Device` is constructed with
`NodeId`.
In `radicle-cob`, a signer is expected to implement
`signature::Signer<ExtendedSignature>`, and everywhere in `radicle`,
`radicle-node`, `radicle-cli`, and `radicle-remote-helper` is expected the
signer is expected to implement `signature::Signer<Signature>`.
A `Device` implements both of these but only requires
`signature::Signer<Signature>` to do so – since an `ExtendedSignature` is
essentially `(PublicKey, Signature)`, and the `NodeId` of the `Device` can be
used.
This declares the minimum supported Rust version, independently of the
`rust-toolchain.toml` file, which specifies the Rust version and
toolchain components the crate/workspace should be built with. The
MSRV affects other crates that depend on anything in this workspace,
but the toolchain file does not seem to affect them. This means the
MSRV is useful for those who build or develop dependents.
Signed-off-by: Lars Wirzenius <liw@liw.fi>
This introduces rate limits for the `ChannelReader` to limit DDoS attacks and
attempts to upload repositories that are larger than a node is will to permit.
The limiter sets the total number of bytes it is will to accept in a single
exchange, defaulting to 500MB. This means that initial fetches will prevent
large repositories, but is plenty for new packfile data to be sent in subsequent
fetch exchanges.
The limit can be configured within the node's config file, under the limits.
If an upload-pack is of a considerable size, the writer thread will be busy
writing bytes to the receiving side. During this period, the receiving side will
not be sending any bytes.
To ensure that the reader does not exit while the writing thread is performing
writes, the reading side will check when the last time a write was performed. If
it has not reached the timeout, then it can continue attempting to read.
Otherwise, it will break out of the loop; killing the upload-pack process safely.
Update the rust toolchain version to 1.85.
Note that this is not the latest, however, it is the latest available in nixpgs,
so this version is good enough for now.
Instead of collecting the results and returning the collection as an
iterator, introduce `FollowPolicies` and `SeedPolicies` types, which
implement `Iterator`.
Note that a call in `service` changed due to the lifetime borrow. It
simply `collect`s into a `Vec` for the time being.
Adds the new `migrate` sub-command to migrate the COB database.
Checks for version when opening the COBs database and suggests the
command on out of date version.
Also adds the 2nd migration to the COBs cache, which updates the patch
JSON schema.
The `no quorum found` error can be opaque and lead to a lot of confusion. It is
better to give more information for this error so that it leads to easier
debugging, since it can be quite a common warning/error.
Instead of a single error, there are two cases provided: `NoCandidates` and
`Diverging`, to capture the two different variants of errors that can occur.
`Display` implementations are provided for both to provide more information.
The failure to set a HEAD ref, due to quourum, during a fetch should not fail
the fetch entirely.
Instead, log the success and failure, or return an error in other cases.
In fact, any failure to set the head could have resulted in notifications not
being sent, nor caching to occur.
The `missing_delegate_default_branch` test was flakey and would fail checking if
Bob had the `rid` in storage.
This can be improved by waiting to ensure that Bob has a `RefsFetched` event
before performing the check.
When fetching, the validation logic would prune a delegate if it was missing the
default branch. Originally, this was done to prevent any broken state, but later
it was realised that this can be too restrictive and would not allow nodes to
recover into an improved state.
Remove this check so thtat nodes can still fetch from each other when the
default branch is missing.
An e2e test is added to show that nodes can still fetch from each other and
still receive other references, even when a delegate is missing their default
branch.
The aim of this change is to make the `Doc` type more safe to use by approaching
the design via [Parse don't validate][[0]] approach.
The problem with the previous approach was that all field were `pub` and thus a
`Doc<Verified>` could easily be mutated and serialized. Granted, the code that
used the serialization would tend to verify the `Doc` first, however, this
approach *ensures* that only a verified `Doc` can be serialized. It also meant
that trying to add new data that would follow the parse approach would require
more generic parameters on top of the existing `PhantomData` parameter, i.e. we
need to do something like: `Doc<RawField, V> -> Doc<ValidField, V>`.
The new approach splits the type into two separate types: `RawDoc` and `Doc`.
The former is allowed to be mutated at will, and uses types that are less
strict. The latter is the valid type that can only be constructed by validating
a `RawDoc` (or the `initial` constructor). The `Doc` type's fields can then only
be accessed by read-only methods.
Solves the problems above by only allowing mutations to `RawDoc`, as well as,
new fields being added to `RawDoc` which are then validated via
`RawDoc::verified`.
[0]: https://lexi-lambda.github.io/blog/2019/11/05/parse-don-t-validate/
We ensure that a `rad/root` ref is included in the signed refs file
under `rad/sigrefs` for all remotes. This prevents a certain kind of
"grafting" attack where signed refs can be copied between repositories,
by having the peer sign over the identity root together with the data refs.
When verifying signed refs, we ensure that the ref is present and points
to an identity branch root that matches the repository identity containing
the signed refs.
Alternatives: lots of alternatives were considered, but this one doesn't
introduce any changes to the signing. The `rad/id/root` name was
considered but is invalid due to `rad/id`.
Though refs of private repositories are not announced publicly, they can
sometimes be relayed to nodes that are not in the allow list.
We fix this by always checking the visibility of a repository before
sending a refs announcement of it to a peer.
Discovered-by: Adrian Duke <adrian.duke@gmail.com>
In the `is_authorized` helper, the logic checks the policy and the visibility of
the repository.
If the policy is set to block, the function can return before getting the
repository and the identity document. This improves the check, since the
repository and identity document may be missing if the repository is blocked, so
it would return a different error other than the expected unauthorized error.
This change simply logs an error if there is a problem loading a COB
from storage, instead of bailing on the cache update process.
We also ensure that objects that can't be loaded from storage are
removed from the cache.
Otherwise, previously, when an error occurred before `logger::init()`
then no message about the error would be shown, because `log` ignores
messages generated before the logger is initialized. E.g. when
`$RAD_HOME/config.json` is missing, invoking `radicle-node`
directly (i.e. not via `rad node start`) would fail without any
explanation being logged nor printed. There also are some other
possible errors that can occur before `logger::init()` where no message
would've been shown.
The fix is to detect if the logger is enabled, which it won't be when it
hasn't been initialized, and if not then fallback to printing directly
to stderr.
To show the lower-level source of an error, like previously, and to
avoid now needing more conditionals with more format strings for all
possibilities of `err.source().is_some()` and `log_enabled!()`, the
"alternate" form (`{:#}`) of formatting `anyhow::Error` is now used.
This also introduces a change in behavior such that the entire chain of
source errors will now be shown, instead of only the first in the chain,
which seems more desirable for errors that cause fatal exiting of
`radicle-node`. Note that this form still formats as only a single
line, like previously.
The prefix "Error: " is used for the new fallback printing, because in
this case it's not a log message (though, it might be written to the log
file), and because that prefix is consistent with how Rust errors that
cause immediate termination are usually printed directly. For the
opposite case, the "Fatal: " prefix serves to distinguish it as being
fatal among the many other various log messages.
While it would be possible to instead return `anyhow::Result<()>` from
`main`, to achieve the printing of early errors (and exiting with
failure code), that would cause undesirable duplication, for non-early
errors that occur after `logger::init()`, of the error message where it
would be written to both the log file and to stderr which often is the
same as the log file.
Signed-off-by: Derick Eddington <kcired@pm.me>
Ensure that when a node initialises multiple repositories, those
repositories can successfully be fetched by another peer once they
connect.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
Fix "signals" thread to continue upon other signals.
Otherwise, previously, when one of the non-shutdown signals was
received, that would cause the "signals" thread to finish, but then a
subsequent signal that should cause shutdown would not because the
thread no longer existed to do so. E.g. when SIGHUP or SIGWINCH was
received first and ignored and then SIGTERM or SIGINT was received next,
the program should still be shutdown, but it wouldn't be. This is now
fixed by simply looping to continue handling subsequent signals.
Logging is now done for:
- Receiving of SIGHUP, at log level `debug`, because the default action
for that signal would be to terminate the process but that is not done
by `radicle_signals`. Someone sending that signal might want to debug
why the process isn't being terminated.
- Disconnection of the signal-notifications channel, at log level
`warn`, because, even though that should be impossible, if it somehow
occurs then a warning is warranted.
Logging is not done for receiving of SIGWINCH, because the default
action for that signal would be to ignore it, which is what
`radicle-node` does.
A wildcard pattern is not used in the match arms, so that any future
changes to the `Signal` variants will require reviewing their handling
in the "signals" thread.
Signed-off-by: Derick Eddington <kcired@pm.me>
In the current design it's possible for one peer to fill the fetch queue
so that fetches from other peers are delayed.
To improve fairness, we move to a queue per peer. We then try to dequeue
from all peers in a random order for good measure.
We were comparing the available peers with the target, so the comparison
would always succeed. We now compare it with the missing count, and use
debug logging since it's not something of concern yet.
To avoid buffering large amounts of data in the process, we set the
worker channel size to `64`. Keep in mind that this is one
`ChannelEvent`, not one byte. `ChannelEvent::Data` can already contain
an arbitrary amount of data via its `Vec<u8>`, but in practice the
maximum is 8192, due to the use of `io::copy`.
We also remove an unused function, and move the flushing to the
`ChannelFlushWriter`.
If two peers are persistent for each other and experience
disconnections, it's possible that an outbound connection is turned into
an inbound one. But if we don't set that correctly, we'll eventually run
into a crash where an already connected peer re-connects, since the
disconnection logic checks for the link direction.
The fetch queue could sometimes not drain due to skipped fetches not
signaling that to the dequeue loop.
We now return a `bool` from `try_fetch` and keep trying to dequeue
fetches until one of them actually is initiated.
Additionally, we try to dequeue fetches at a regular interval
(`IDLE_INTERVAL`) to ensure that the queue is drained.
There are cases where the service state doesn't match the state
of the underlying wire protocol. We remedy this by transitioning peers
to a "connected" state when a message is received, if they are in a
"connecting" state.
The long term solution to this will likely be to merge the service and
protocol layer so that there are no inconsistencies.
The other case happens when a persistent peer is in "disconnected" state
and attempts an inbound connection to us.
Fintan Halpenny
Lars Wirzenius
Lorenz Leutgeb
cloudhead
Sebastian Martinez
Derick Eddington
Alexis Sellier