The motivation of this change is to move away from tying signing to be
specifically for ed25519. The reason being that the protocol will want move
towards two different kinds of signing – the node signing artifacts, and an
author signing artifacts.
This change captures the former, node signing, by introducing a `Device` type
that is the `NodeId` – a PublicKey underneath the hood – and a signing
mechanism. This allows the replacement of the `Signer` trait being used – which
always assumed a `PublicKey`. Instead, the `Device` is constructed with
`NodeId`.
In `radicle-cob`, a signer is expected to implement
`signature::Signer<ExtendedSignature>`, and everywhere in `radicle`,
`radicle-node`, `radicle-cli`, and `radicle-remote-helper` is expected the
signer is expected to implement `signature::Signer<Signature>`.
A `Device` implements both of these but only requires
`signature::Signer<Signature>` to do so – since an `ExtendedSignature` is
essentially `(PublicKey, Signature)`, and the `NodeId` of the `Device` can be
used.
This introduces rate limits for the `ChannelReader` to limit DDoS attacks and
attempts to upload repositories that are larger than a node is will to permit.
The limiter sets the total number of bytes it is will to accept in a single
exchange, defaulting to 500MB. This means that initial fetches will prevent
large repositories, but is plenty for new packfile data to be sent in subsequent
fetch exchanges.
The limit can be configured within the node's config file, under the limits.
Instead of collecting the results and returning the collection as an
iterator, introduce `FollowPolicies` and `SeedPolicies` types, which
implement `Iterator`.
Note that a call in `service` changed due to the lifetime borrow. It
simply `collect`s into a `Vec` for the time being.
The aim of this change is to make the `Doc` type more safe to use by approaching
the design via [Parse don't validate][[0]] approach.
The problem with the previous approach was that all field were `pub` and thus a
`Doc<Verified>` could easily be mutated and serialized. Granted, the code that
used the serialization would tend to verify the `Doc` first, however, this
approach *ensures* that only a verified `Doc` can be serialized. It also meant
that trying to add new data that would follow the parse approach would require
more generic parameters on top of the existing `PhantomData` parameter, i.e. we
need to do something like: `Doc<RawField, V> -> Doc<ValidField, V>`.
The new approach splits the type into two separate types: `RawDoc` and `Doc`.
The former is allowed to be mutated at will, and uses types that are less
strict. The latter is the valid type that can only be constructed by validating
a `RawDoc` (or the `initial` constructor). The `Doc` type's fields can then only
be accessed by read-only methods.
Solves the problems above by only allowing mutations to `RawDoc`, as well as,
new fields being added to `RawDoc` which are then validated via
`RawDoc::verified`.
[0]: https://lexi-lambda.github.io/blog/2019/11/05/parse-don-t-validate/
Though refs of private repositories are not announced publicly, they can
sometimes be relayed to nodes that are not in the allow list.
We fix this by always checking the visibility of a repository before
sending a refs announcement of it to a peer.
Discovered-by: Adrian Duke <adrian.duke@gmail.com>
In the current design it's possible for one peer to fill the fetch queue
so that fetches from other peers are delayed.
To improve fairness, we move to a queue per peer. We then try to dequeue
from all peers in a random order for good measure.
We were comparing the available peers with the target, so the comparison
would always succeed. We now compare it with the missing count, and use
debug logging since it's not something of concern yet.
If two peers are persistent for each other and experience
disconnections, it's possible that an outbound connection is turned into
an inbound one. But if we don't set that correctly, we'll eventually run
into a crash where an already connected peer re-connects, since the
disconnection logic checks for the link direction.
The fetch queue could sometimes not drain due to skipped fetches not
signaling that to the dequeue loop.
We now return a `bool` from `try_fetch` and keep trying to dequeue
fetches until one of them actually is initiated.
Additionally, we try to dequeue fetches at a regular interval
(`IDLE_INTERVAL`) to ensure that the queue is drained.
There are cases where the service state doesn't match the state
of the underlying wire protocol. We remedy this by transitioning peers
to a "connected" state when a message is received, if they are in a
"connecting" state.
The long term solution to this will likely be to merge the service and
protocol layer so that there are no inconsistencies.
The other case happens when a persistent peer is in "disconnected" state
and attempts an inbound connection to us.
1. Adds a user-agent string to the node announcement. This lets us keep
track of what software version everyone is running. Especially useful
to see what percentage of the network has upgraded to a new release.
For old nodes, this defaults to the string `/radicle/`.
The user agent format is roughly based on BIP 0014.
2. Advertize protocol version in node announcement.
By advertizing the protocol version, nodes can decide to only connect to
peers with a compatible version. The default and current version is `1`.
We bundle these two changes as they both modify the node announcement
and node database.
3. Swap the order of the fields in announcements to make the message
extensible.
This prevents potential memory leaks. We also ensure that sends fail
instead of blocking, in case the channels are full.
Additionally, we add some metrics to report on channel size.
Previously, our advertized inventory was not always correct: when we
stopped seeding a repository, it was still advertised as part of our
inventory until node restart, because the `Storage` type doesn't have
access to seeding policies.
In this change, we remove the concept of inventory from `Storage` and
make the authoritative place for it the routing table in our database.
To make this work, we have to add our node to the database on profile
creation, to not violate the foreign key constraint on the routing
table. Hence, the tests are changed to include our alias which is now
always available.
This is a change to the `config.json` file which removes certain options
that are best not touched and changes some options to make them easier
to work with.
1. We change the default journaling mode to "wal" and remove the config
option.
2. We remove the option to set the peer connection "target", as it's not
a good idea to set this to a different value, as it affects the
network.
3. We combine seeding policy & scope, since there's no need to set the
scope when the policy is "block". In that case, we always want to
block "all" remotes. The new policy configuration has the following
schema:
{ seedingPolicy: { default: "block" | "allow", scope?: "all" | "followed" } }
The "scope" key is not used when "default" is set to "block".
4. We add an `extra` field to the node config for options that are not
recognized. We use this to warn the user.
We use the *staggered broadcast* technique when relaying gossip messages
to reduce message amplification. In our basic simulation, it brings
amplification down from 3.5 to 1.5.
The idea is simply to accumulate gossip messages and relay them after a fixed
interval of time. This has two effects:
1) Old messages that haven't been sent yet are replaced by newer ones,
so only one message is sent after the delay.
2) Because nodes have their own intervals, messages are not all sent at
the same time, which reduces the chance of messages crossing paths.
For now we only turn this on for inventory messages, since this is the
least likely to be noticed by users, and also the most problematic
currently.
In the case of an out-of-sync local node, we were showing the wrong tip.
Additionally, mark your tip as unannounced if it isn't announced. This
isn't fool-proof, but can be useful to debug certain issues.
From empirical testing, the `FETCH_TIMEOUT` can be set to a lower value of 3s.
For a repository that is about 300MiB, the process will successfully send all
packfile bytes and the references will be written.
The `--timeout` option is also used in the `git-upload-pack` process for good
measure.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
* Remove `rad sync --force` flag
* Remove `rad clone --private` flag
These flags are no longer necessary. Simplify specifying which seed
to fetch from with `--seed` is enough.
*Breaking change* The `tor` configuration field is no longer
supported. Users must remove that field from their config.
Allows for a mixed mode, where regular addresses bypass the Tor proxy,
while onion addresses go through it.
When `onion.mode` is set to "forward", Tor connections are fowarded to
the global proxy, and if it isn't set, to the OS's DNS resolution.
For mixed mode, a global proxy simply isn't set, so that IP/DNS is not
proxied.
The previous banning implementation didn't work, since it included port
numbers in the address value.
Here, we keep a separate table with (node, ip) pairs and update this
table when a node connects or is banned.
After empirically testing HardenedBSD[[0]], which has a total blob size of
42GiB, and its largest checkout is 1.26GiB (using `git-sizer`[[1]]), it was
found that letting the background process fetch the repository with a timeout of
1 hour is enough (a very generous estimate).
This deemed sufficient, since cases smaller than this will report back that they
are done before this time. For cases larger than this, more research will be
needed.
[0]: https://git.hardenedbsd.org/hardenedbsd/HardenedBSD
[1]: https://github.com/github/git-sizer
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
Emit the `UploadPack` events from the `upload_pack` method.
A wrapper type, `Reporter`, is introduced for implementing a `Write` instance
that will parse the given bytes as `gix_protocol::RemoteProgress`, if possible,
and emit it via the `Emitter`, and then write the bytes to the underlying
writer. This allows any subscribers to see the progress of an upload-pack, i.e.
`Compressing objects`, `Counting object`, `Enumerating objects`.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
Instead of using `DEFAULT_CHANNEL_TIMEOUT`, the timeout is dynamically set
per-fetch by the caller.
This passed via the `Io::Fetch` variant -- when initiating a fetch, and the
`Control::Open` variant when responding to a fetch.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
Instead of resetting the number of attempts on connection, reset it when
the connection is stable, ie. after some time has passed.
This prevents connection retries from happening in succession when a
peer is disconnected during the initial handshake, as is the case with
invalid subscribe timestamps.
If our inventory hasn't updated, don't issue a new inventory
announcement, simply use the previous message.
This avoids identical inventories propagating over the network and using
up resources.
Relay now has three values: "always", "never" and "auto".
The first two operate like the previous `true`/`false`. The
new (default) value turns relaying on for nodes with public (external)
addresses and off for nodes without. This should reduce redundant traffic on
the network, that was causing the rate limiters to trigger.
Many (most?) system clocks are not monotonic, and even though we're
getting a `LocalTime`, which *is* monotonic when using `now()`, it is
constructed from a `SystemTime` which isn't.
Lorenz Leutgeb
Fintan Halpenny
cloudhead
Alexis Sellier