We use the *staggered broadcast* technique when relaying gossip messages
to reduce message amplification. In our basic simulation, it brings
amplification down from 3.5 to 1.5.
The idea is simply to accumulate gossip messages and relay them after a fixed
interval of time. This has two effects:
1) Old messages that haven't been sent yet are replaced by newer ones,
so only one message is sent after the delay.
2) Because nodes have their own intervals, messages are not all sent at
the same time, which reduces the chance of messages crossing paths.
For now we only turn this on for inventory messages, since this is the
least likely to be noticed by users, and also the most problematic
currently.
In the case of an out-of-sync local node, we were showing the wrong tip.
Additionally, mark your tip as unannounced if it isn't announced. This
isn't fool-proof, but can be useful to debug certain issues.
Introduce an `UploadPack` type for keeping track of relevant data from
upload-pack events and displaying progress to the terminal.
It keeps track of the number of remotes that are being uploaded to as well as
the throughput of transmitted data.
The `Progress` events are logged rather than being displayed since they don't
provide any useful details to the user.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
* Remove `rad sync --force` flag
* Remove `rad clone --private` flag
These flags are no longer necessary. Simplify specifying which seed
to fetch from with `--seed` is enough.
*Breaking change* The `tor` configuration field is no longer
supported. Users must remove that field from their config.
Allows for a mixed mode, where regular addresses bypass the Tor proxy,
while onion addresses go through it.
When `onion.mode` is set to "forward", Tor connections are fowarded to
the global proxy, and if it isn't set, to the OS's DNS resolution.
For mixed mode, a global proxy simply isn't set, so that IP/DNS is not
proxied.
To ensure that the `rad id` command does not modify the project payload without
it being able to be deserialized into the `Project` type, the command performs
this check throught the `project` method, and returns an error if this fails.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
As a response to Issue # c4134924cf2ce84444c6f7235a27651ad49311b5.
A no-op command in rad inbox defaults to show, but the inbox argument parser
only permits --repo in instances where the operation is explicity declared.
Here, the check for an explicit operation is removed so one is never required.
To improve the user experience of `rad init`, the events listener waits for
upload-pack messages indefinitely. The spinner reports back the upload progress
over time.
This process can be cancelled via Ctrl+C without affecting the announcement,
since that is happening in the background.
To also improve the experience, the remote NIDs that have successfully synced
are reported to the terminal.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
The timeout for a `UnixStream` will interrupt a `read` even when more data is
available. The preference would be that the timeout applies to each read, where
the timeout is reset again once a read is successful. This better supports
streaming semantics expected for the `Handle::subscribe` method.
To achieve this, the `LineIter` type holds the `UnixStream` and timeout
`Duration`. The `Iterator` implementation can then use `read_line` to progress
the stream one line at a time, and the timeout only applies to each read.
Co-authored-by: Alexis Sellier <alexis@radicle.xyz>
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
Add a test to confirm that the identity can still be deserialized when an
unknown field is added.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
The rate-limitter gets hit on connect, due to historical gossip messages
being exchanged. This results in a high initial burst that needs to be
accounted for.
Relay now has three values: "always", "never" and "auto".
The first two operate like the previous `true`/`false`. The
new (default) value turns relaying on for nodes with public (external)
addresses and off for nodes without. This should reduce redundant traffic on
the network, that was causing the rate limiters to trigger.
With the increase in traffic on the network after launch, the old
defaults don't make sense anymore.
Longer term, we should look into throttling outgoing messages as well,
prioritizing original messages over relayed messages.
* Try to use constants instead of strings
* Move crypto seed override out of radicle-crypto
* Use GIT_COMMITTER_DATE for commit overrides
* Use RAD_LOCAL_TIME as general time override
* Allow variables to be used in release mode
If multiple ids are specified for `rad inbox clear`, it would report
that only 1 item was cleared, when in fact multiple were.
This was found to be due to the SQL code returning a change count of
`1`. This is likely due to the way the statement is being reset each
time. The `sqlite` library does not seem to easily support the usage
of `IN` in a `WHERE` clause. So instead, the `count` is aggregated in
the loop and returned instead.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
When a `Patch` or `Issue` is entirely removed from the Git repository,
then there is no update to be made -- instead it should also be
removed from the cache.
This covers the case where the original author of the COB deletes the
COB.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
* Add `--force` flag to force a fetch even if the node is not a seed.
* Don't fail if we can't connect to a node during a fetch.
* More accurately exit when we hit our replica count.
When `--private` is passed to `clone`, we force a fetch from nodes that
aren't known seeds.
When it isn't specified, we skip nodes that aren't known seeds, even if
they are in the preferred seeds list. Previously, we would still fetch
from them, which would cause errors.
Implement this function on `RemoteRepository`, and move `repositories`
to the trait level.
This function will be used to quickly get the `RefsAt` of all repository
remotes.
Add an example of changing a public repository to a private
repository, noting that any replicated data will be public, but any
new changes will not be replicated.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
There are two areas where we can be more tolerant of delegate namespaces
being missing or not validating:
1. Calculating the canonical HEAD
2. Fetching from a remote
In 1. the protocol is tolerant in that if the local node does not have
the default branch for a delegate, it will still attempt to use any of
the delegates it does have to reach the threshold.
This is made safe by ensuring that if the threshold is being updated
then the node performing the update must have a threshold of delegates
locally in their storage. It also made safe by 2.
In 2. the protocol is tolerant by allowing delegates to be missing
from the serving side, as long as they can still meet a threshold of
delegates. This is further tolerant, when validating the received
data, a threshold of delegates are valid to consider the fetch
successful -- otherwise it will fail.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
Previously, we only kept tracked of unsynced nodes we were connected to.
Now we track all unsynced nodes, since announcements are relayed.
We also cap the replicas to exit earlier in case of very low replica
count.
We distinguish between the cases:
a) There are no seeds at all.
b) There are no seeds to announce to despite the user wanting a higher
replication factor, for example.
Instead of waiting for *all* configured/preferred seeds to be synced,
we wait for at least one.
Note that the previous behavior can still be used by setting
`--replicas` appropriately.
The inventory operation of the `rad sync` command does not require an
RID. Localise resolving the RID in the match arms that require it.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
Instead of re-loading the full inventory on `rad init`, we simply add
the new repository to the in-memory cache.
We also keep track of whether a fetch was a full clone, to know when
to update our routing table.
When running `rad patch list`, show the set of delegate reviews for
each patch, with the following meaning for the symbols:
- ✔: delegate has accepted the patch
- ✗: delegate has rejected the patch
- -: delegate has not made a verdict on the patch
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
If the user passes a <path> argument to init, it is incorrect and a
bit confusing to reference the "current directory" or "this directory"
e.g. "run rad . in current directory" when the repo was not created in the
current directory and was created with a path argument instead.
Use the path provided by the user in help text, otherwise reference
the current directory.
If `rad id` is called with no arguments, or just `--title` and
`--description`, then it will fail because there were no changes made
to the identity.
Check if the `proposal` document is the same the `current` document,
and if so supply a hint and a better error.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
When the `rad inbox` command would come across a reference name that
it did not recognise as a `Cob` or `Branch`, then it would fail with
an `Unknown` error.
Instead, add an `Unknown` variant to `NotificationKind` which can be
displayed in the `inbox` CLI, by using the `--show-unknown`
flag. Otherwise, it will be skipped and the inbox will work as usual.
While refactoring, the `TypedId` type is used for the `Cob` variant to
simplify some of its uses.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
Help the user know which revision they currently have checked out when
running `rad patch checkout`.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
If an ancestry check of an annotated tag was triggered then the fetch
from that peer would fail. This is due to `graph_ahead_behind` only
working for commit-ish objects.
To allow for annotated tags, and other object that peel to commits,
the check now peels the two objects to commits before getting their
`Oid`s. The call to `graph_ahead_behind` can take these two `Oid`s
instead.
A test for annotated tags is included as part of the `examples/git`
suite in the CLI tests.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
If the name is available to show for the repository -- it may not be
if it hasn't been replicated yet -- then add it to the output of the
`rad seed` command.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
The documentation for `rad inbox show` and `rad inbox clear` were
missing.
Add documentation into the help message for both.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
This writes out information that may be useful when debugging Radicle
remotely on a user's machine. Things like versions of programs, and
values of relevant environment variables.
Signed-off-by: Lars Wirzenius <liw@liw.fi>
No tests for `rad patch delete` existed.
Add a test where two peers interact on a patch and one decides to
delete the patch. It ensures that the patch is still visible to the
peer that did not delete the patch.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
If any replies to patches were made, it was not possible to see them
in the output of `rad patch show`.
Add the replies to show output, but only if the `--verbose` flag is
set.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
The option to reply to a comment was undocumented in the `rad patch
comment` command.
Add the option to the "Comment options" section of the help message.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
To make things a little more consistent, move the `rad self --address`
command to `rad node config --addresses`, since it's more about
externally-facing configuration of the node, rather than your local user
or profile.
In order to bootstrap the COB cache, and also in case of issues,
supply a command for caching a set of patches/issues, or a single
patch/issue.
This is necessary since reads are not read-thru, it may appear that
COBs are missing when in fact they haven't been cached when first
moving over to the cache.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
Implement caching for the `Issue` and `Patch` COB. This is achieved by
having the `Transaction::initial` and `Transaction::commit` methods
take a cache which can update the newly created/updated object. The
`Store::remove` method also takes the cache for removing the object
from the cache, as well as the repository.
This meant that the `*Mut` types for the respective COBs are required
to carry a cache alongside the repository store. Identities will not
be cached, and so this is always set to `NoCache` -- which performs no
caching, and always succeeds.
To perform cache reading for issues and patches, the `cache::Issues`
and `cache::Patches` traits are introduced. They capture the minimal
amount of methods that are needed for the `rad issue` and `rad patch`
CLI commands.
The implementor for each of these traits is there respective `Cache`
types, which combines their backing repository store and the SQLite
database. They both provide convenience methods:
- `create` (and `draft` for patches)
- `get_mut`
- `remove`
which can be used instead of their repository store counterparts.
All uses of the repository stores (where needed) are replaced with
`Cache` types.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
Previously, authorization was only checked for ops after the root op.
We also avoid creating empty actions for `assign` and `label`, which
was needed for non-authorized users to be able to create issues/patches.
This makes the default policy `block` with scope `all`. The good thing
about this is that you don't have to manage individual nodes in the
beginning. You simply seed the repos you want, and everything is
followed within those repos. This matches user expectations, as `rad
seed` does what you expect it to do.
For open-policy seed nodes, it also means that simply changing `block`
to `allow` from the default config is all you need.
This leaves the non-default `followed` scope as a more "advanced"
setting, since it is much more cumbersome: it forces you to follow a
user before you can see their issues/patches, which is not something
most users expect coming from old-shcool forges.
Shows you updates on local repositories.
Included here is a basic implementation of a notification system. A new
database is created and updated after every fetch, with refs that have
changed.
The `rad inbox` command can then list items in the database and delete
them.
When using `network: "test"`, we shouldn't default to any public seed
node. This change makes sure that the "preferredSeeds" configuration var
is only given a value when `network` is `main`.
The command now works with both a replica target *and* a seeds target.
This is especially useful to eg. ensure that a preferred seed has been
synced, while still trying to hit a higher replica count.
Introduce pinned repositories to the `profile::Config`, i.e.
`config.json`.
This is used for pinning repositories in the httpd `/projects` handle.
To allow returning all repositories, an `all` query parameter is also
included.
This is non-backwards compatible change, since we introduce the `show`
query parameter which, if not supplied, will default to `pinned`.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
In the case where we don't find any seeds, display a better error
message.
We also don't bail out on error, since the announcement step might still
be worth trying.
Previously, COB change commits would not distinguish between their
parents. This means that they would try to load source code commits
(eg. for patch COBs) and then fail since those commits wouldn't have a
manifest or `Rad-Resource`.
We introduce a backwards compatible change here which is that parent
oids that are not COB commits are accompanied by a commit trailer called
`Rad-Related`. This is for all related content that should be pulled in
but is not part of the change graph proper.
When building the change graph, we simply ignore commits that are marked
as "related".
We were using the two pretty interchangeably. "Project" should only be
used to refer to the repository payload. "Repository" should be used
when referring to the resource that is fetched, synced, cloned, checked
out etc.
Add a test to show the behaviour of setting up 3 delegates, through two
seed nodes -- ensuring that commands are still operational after
accepting a delegate that the node is missing.
The two seeds are setup with two different scopes:
- `Scope::All`
- `Scope::Followed`
This is to show that the network has eventual consistency once there
is a way to fetch any other node's fork in the network.
The setup of the test is:
- Alice adds Bob as a delegate
- Bob adds Eve as a delegate
- Alice accepts Bob's changes
Alice is able to get Eve's fork through the `Scope::All` seed. The
`Scope::Followed` seed will then be able get Eve's fork once any other
node syncs with it, in this case Eve.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
If the expected replica count had exceeded the amount of seeds already
synced with the following message would appear:
Not connected to any seeds for {rid}.
This is not true, since there are seeds that were already synced.
This is fixed by calculating the `synced` and `connected` nodes,
without the local peer in the set. This means that the check for
`synced.len() >= connected.len()` will get a hit and the function will
correctly return early.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
We cannot guarantee that the `rad sync status` will run before all the
nodes are synced.
Remove the call to `rad sync status` and explain that at least 1 of
the nodes will be synced.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
cloudhead
Lorenz Leutgeb
Lars Wirzenius
Fintan Halpenny
RadsammyT
Alexis Sellier
Sebastian Martinez
Michael Raitza
josibake
Zlatan Todoric
Christopher Fredén