This introduces rate limits for the `ChannelReader` to limit DDoS attacks and
attempts to upload repositories that are larger than a node is will to permit.
The limiter sets the total number of bytes it is will to accept in a single
exchange, defaulting to 500MB. This means that initial fetches will prevent
large repositories, but is plenty for new packfile data to be sent in subsequent
fetch exchanges.
The limit can be configured within the node's config file, under the limits.
If an upload-pack is of a considerable size, the writer thread will be busy
writing bytes to the receiving side. During this period, the receiving side will
not be sending any bytes.
To ensure that the reader does not exit while the writing thread is performing
writes, the reading side will check when the last time a write was performed. If
it has not reached the timeout, then it can continue attempting to read.
Otherwise, it will break out of the loop; killing the upload-pack process safely.
Update the rust toolchain version to 1.85.
Note that this is not the latest, however, it is the latest available in nixpgs,
so this version is good enough for now.
Instead of collecting the results and returning the collection as an
iterator, introduce `FollowPolicies` and `SeedPolicies` types, which
implement `Iterator`.
Note that a call in `service` changed due to the lifetime borrow. It
simply `collect`s into a `Vec` for the time being.
Adds the new `migrate` sub-command to migrate the COB database.
Checks for version when opening the COBs database and suggests the
command on out of date version.
Also adds the 2nd migration to the COBs cache, which updates the patch
JSON schema.
The `no quorum found` error can be opaque and lead to a lot of confusion. It is
better to give more information for this error so that it leads to easier
debugging, since it can be quite a common warning/error.
Instead of a single error, there are two cases provided: `NoCandidates` and
`Diverging`, to capture the two different variants of errors that can occur.
`Display` implementations are provided for both to provide more information.
The failure to set a HEAD ref, due to quourum, during a fetch should not fail
the fetch entirely.
Instead, log the success and failure, or return an error in other cases.
In fact, any failure to set the head could have resulted in notifications not
being sent, nor caching to occur.
The `missing_delegate_default_branch` test was flakey and would fail checking if
Bob had the `rid` in storage.
This can be improved by waiting to ensure that Bob has a `RefsFetched` event
before performing the check.
When fetching, the validation logic would prune a delegate if it was missing the
default branch. Originally, this was done to prevent any broken state, but later
it was realised that this can be too restrictive and would not allow nodes to
recover into an improved state.
Remove this check so thtat nodes can still fetch from each other when the
default branch is missing.
An e2e test is added to show that nodes can still fetch from each other and
still receive other references, even when a delegate is missing their default
branch.
The aim of this change is to make the `Doc` type more safe to use by approaching
the design via [Parse don't validate][[0]] approach.
The problem with the previous approach was that all field were `pub` and thus a
`Doc<Verified>` could easily be mutated and serialized. Granted, the code that
used the serialization would tend to verify the `Doc` first, however, this
approach *ensures* that only a verified `Doc` can be serialized. It also meant
that trying to add new data that would follow the parse approach would require
more generic parameters on top of the existing `PhantomData` parameter, i.e. we
need to do something like: `Doc<RawField, V> -> Doc<ValidField, V>`.
The new approach splits the type into two separate types: `RawDoc` and `Doc`.
The former is allowed to be mutated at will, and uses types that are less
strict. The latter is the valid type that can only be constructed by validating
a `RawDoc` (or the `initial` constructor). The `Doc` type's fields can then only
be accessed by read-only methods.
Solves the problems above by only allowing mutations to `RawDoc`, as well as,
new fields being added to `RawDoc` which are then validated via
`RawDoc::verified`.
[0]: https://lexi-lambda.github.io/blog/2019/11/05/parse-don-t-validate/
We ensure that a `rad/root` ref is included in the signed refs file
under `rad/sigrefs` for all remotes. This prevents a certain kind of
"grafting" attack where signed refs can be copied between repositories,
by having the peer sign over the identity root together with the data refs.
When verifying signed refs, we ensure that the ref is present and points
to an identity branch root that matches the repository identity containing
the signed refs.
Alternatives: lots of alternatives were considered, but this one doesn't
introduce any changes to the signing. The `rad/id/root` name was
considered but is invalid due to `rad/id`.
Though refs of private repositories are not announced publicly, they can
sometimes be relayed to nodes that are not in the allow list.
We fix this by always checking the visibility of a repository before
sending a refs announcement of it to a peer.
Discovered-by: Adrian Duke <adrian.duke@gmail.com>
In the `is_authorized` helper, the logic checks the policy and the visibility of
the repository.
If the policy is set to block, the function can return before getting the
repository and the identity document. This improves the check, since the
repository and identity document may be missing if the repository is blocked, so
it would return a different error other than the expected unauthorized error.
This change simply logs an error if there is a problem loading a COB
from storage, instead of bailing on the cache update process.
We also ensure that objects that can't be loaded from storage are
removed from the cache.
Otherwise, previously, when an error occurred before `logger::init()`
then no message about the error would be shown, because `log` ignores
messages generated before the logger is initialized. E.g. when
`$RAD_HOME/config.json` is missing, invoking `radicle-node`
directly (i.e. not via `rad node start`) would fail without any
explanation being logged nor printed. There also are some other
possible errors that can occur before `logger::init()` where no message
would've been shown.
The fix is to detect if the logger is enabled, which it won't be when it
hasn't been initialized, and if not then fallback to printing directly
to stderr.
To show the lower-level source of an error, like previously, and to
avoid now needing more conditionals with more format strings for all
possibilities of `err.source().is_some()` and `log_enabled!()`, the
"alternate" form (`{:#}`) of formatting `anyhow::Error` is now used.
This also introduces a change in behavior such that the entire chain of
source errors will now be shown, instead of only the first in the chain,
which seems more desirable for errors that cause fatal exiting of
`radicle-node`. Note that this form still formats as only a single
line, like previously.
The prefix "Error: " is used for the new fallback printing, because in
this case it's not a log message (though, it might be written to the log
file), and because that prefix is consistent with how Rust errors that
cause immediate termination are usually printed directly. For the
opposite case, the "Fatal: " prefix serves to distinguish it as being
fatal among the many other various log messages.
While it would be possible to instead return `anyhow::Result<()>` from
`main`, to achieve the printing of early errors (and exiting with
failure code), that would cause undesirable duplication, for non-early
errors that occur after `logger::init()`, of the error message where it
would be written to both the log file and to stderr which often is the
same as the log file.
Signed-off-by: Derick Eddington <kcired@pm.me>
Ensure that when a node initialises multiple repositories, those
repositories can successfully be fetched by another peer once they
connect.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
Fix "signals" thread to continue upon other signals.
Otherwise, previously, when one of the non-shutdown signals was
received, that would cause the "signals" thread to finish, but then a
subsequent signal that should cause shutdown would not because the
thread no longer existed to do so. E.g. when SIGHUP or SIGWINCH was
received first and ignored and then SIGTERM or SIGINT was received next,
the program should still be shutdown, but it wouldn't be. This is now
fixed by simply looping to continue handling subsequent signals.
Logging is now done for:
- Receiving of SIGHUP, at log level `debug`, because the default action
for that signal would be to terminate the process but that is not done
by `radicle_signals`. Someone sending that signal might want to debug
why the process isn't being terminated.
- Disconnection of the signal-notifications channel, at log level
`warn`, because, even though that should be impossible, if it somehow
occurs then a warning is warranted.
Logging is not done for receiving of SIGWINCH, because the default
action for that signal would be to ignore it, which is what
`radicle-node` does.
A wildcard pattern is not used in the match arms, so that any future
changes to the `Signal` variants will require reviewing their handling
in the "signals" thread.
Signed-off-by: Derick Eddington <kcired@pm.me>
In the current design it's possible for one peer to fill the fetch queue
so that fetches from other peers are delayed.
To improve fairness, we move to a queue per peer. We then try to dequeue
from all peers in a random order for good measure.
We were comparing the available peers with the target, so the comparison
would always succeed. We now compare it with the missing count, and use
debug logging since it's not something of concern yet.
To avoid buffering large amounts of data in the process, we set the
worker channel size to `64`. Keep in mind that this is one
`ChannelEvent`, not one byte. `ChannelEvent::Data` can already contain
an arbitrary amount of data via its `Vec<u8>`, but in practice the
maximum is 8192, due to the use of `io::copy`.
We also remove an unused function, and move the flushing to the
`ChannelFlushWriter`.
If two peers are persistent for each other and experience
disconnections, it's possible that an outbound connection is turned into
an inbound one. But if we don't set that correctly, we'll eventually run
into a crash where an already connected peer re-connects, since the
disconnection logic checks for the link direction.
The fetch queue could sometimes not drain due to skipped fetches not
signaling that to the dequeue loop.
We now return a `bool` from `try_fetch` and keep trying to dequeue
fetches until one of them actually is initiated.
Additionally, we try to dequeue fetches at a regular interval
(`IDLE_INTERVAL`) to ensure that the queue is drained.
There are cases where the service state doesn't match the state
of the underlying wire protocol. We remedy this by transitioning peers
to a "connected" state when a message is received, if they are in a
"connecting" state.
The long term solution to this will likely be to merge the service and
protocol layer so that there are no inconsistencies.
The other case happens when a persistent peer is in "disconnected" state
and attempts an inbound connection to us.
1. Adds a user-agent string to the node announcement. This lets us keep
track of what software version everyone is running. Especially useful
to see what percentage of the network has upgraded to a new release.
For old nodes, this defaults to the string `/radicle/`.
The user agent format is roughly based on BIP 0014.
2. Advertize protocol version in node announcement.
By advertizing the protocol version, nodes can decide to only connect to
peers with a compatible version. The default and current version is `1`.
We bundle these two changes as they both modify the node announcement
and node database.
3. Swap the order of the fields in announcements to make the message
extensible.
This prevents potential memory leaks. We also ensure that sends fail
instead of blocking, in case the channels are full.
Additionally, we add some metrics to report on channel size.
Previously, our advertized inventory was not always correct: when we
stopped seeding a repository, it was still advertised as part of our
inventory until node restart, because the `Storage` type doesn't have
access to seeding policies.
In this change, we remove the concept of inventory from `Storage` and
make the authoritative place for it the routing table in our database.
To make this work, we have to add our node to the database on profile
creation, to not violate the foreign key constraint on the routing
table. Hence, the tests are changed to include our alias which is now
always available.
Lorenz Leutgeb
Fintan Halpenny
cloudhead
Derick Eddington
Alexis Sellier