The rate-limitter gets hit on connect, due to historical gossip messages
being exchanged. This results in a high initial burst that needs to be
accounted for.
Relay now has three values: "always", "never" and "auto".
The first two operate like the previous `true`/`false`. The
new (default) value turns relaying on for nodes with public (external)
addresses and off for nodes without. This should reduce redundant traffic on
the network, that was causing the rate limiters to trigger.
With the increase in traffic on the network after launch, the old
defaults don't make sense anymore.
Longer term, we should look into throttling outgoing messages as well,
prioritizing original messages over relayed messages.
* Try to use constants instead of strings
* Move crypto seed override out of radicle-crypto
* Use GIT_COMMITTER_DATE for commit overrides
* Use RAD_LOCAL_TIME as general time override
* Allow variables to be used in release mode
In radicle/src/test.rs, we have a helper type for a Node, which
creates a temporary directory using tempfile::tempdir, which creates a
TempDir. When the TempDir is dropped, it deletes its associated
directory. Before that happens, the Node has already stored the path.
Right after that the TempDir is dropped, and the directory deleted.
When the test code later opens the repository, using Storage::open,
it recreates the directory. As a result, there is nothing that would
delete the directory when the test finishes.
This means tests that use the helper Node type leave temporary files
behind.
Fix this by storing the TempDir in the helper Node, not just its path.
This means the TempDir is only dropped at the end of the test its used
in, and no temporary files are left behind.
Signed-off-by: Lars Wirzenius <liw@liw.fi>
Implement a new build pipeline using `podman` and `zig` that is
reproducible and can be run entirely on linux.
We also simplify the versioning system, defaulting to the output of `git
describe` when there are no exact tag matches.
If multiple ids are specified for `rad inbox clear`, it would report
that only 1 item was cleared, when in fact multiple were.
This was found to be due to the SQL code returning a change count of
`1`. This is likely due to the way the statement is being reset each
time. The `sqlite` library does not seem to easily support the usage
of `IN` in a `WHERE` clause. So instead, the `count` is aggregated in
the loop and returned instead.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
Implement this function on `RemoteRepository`, and move `repositories`
to the trait level.
This function will be used to quickly get the `RefsAt` of all repository
remotes.
There are two areas where we can be more tolerant of delegate namespaces
being missing or not validating:
1. Calculating the canonical HEAD
2. Fetching from a remote
In 1. the protocol is tolerant in that if the local node does not have
the default branch for a delegate, it will still attempt to use any of
the delegates it does have to reach the threshold.
This is made safe by ensuring that if the threshold is being updated
then the node performing the update must have a threshold of delegates
locally in their storage. It also made safe by 2.
In 2. the protocol is tolerant by allowing delegates to be missing
from the serving side, as long as they can still meet a threshold of
delegates. This is further tolerant, when validating the received
data, a threshold of delegates are valid to consider the fetch
successful -- otherwise it will fail.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
When deciding whether or not to fetch a repo based on a refs
announcement, we need to know whether our signed refs are stale or not,
compared to the ones in the announcement. This can be quite expensive as
it requires multiple reads from Git's ODB, which often results in
packfile loading.
To remedy this, we introduce a refs cache in our node database that is
consulted when an announcement is received.
We also remove the logic that generates `info` messages, since they are
rarely needed and introduced more complexity.
Additionally, we revert commit 3ad2b4431f,
which introduced logic to minimze wants and haves with additional logic,
since it potentially also increases the load on the ODB.
Instead of dropping the existing connection(s), we drop the same
connection on both sides, by using a simple rule that yields the same
result whether the node sees the connection as inbound or outbound.
The trick is to use something that both nodes agree on, for instance
whos public key is greater than the other, and use that information
to close the same connection on both sides.
Instead of re-loading the full inventory on `rad init`, we simply add
the new repository to the in-memory cache.
We also keep track of whether a fetch was a full clone, to know when
to update our routing table.
For large seeds with many repos, it's expensive to traverse the
inventory everytime the node connects to a new peer, for example.
To improve the situation, we cache the inventory per `Storage` instance,
and add a `refresh` method to refresh the inventory.
This method is called when the node is told to synchronize its
inventory.
This change limits the amount of `wants` and `haves` data that is sent
to the serving side of a fetch.
When a `RefsAt` announcement is sent, the fetching peer can calculate
the difference between the `rad/sigrefs` they're aware of -- if it
exists -- and the newly advertised `rad/sigrefs`. This means they can
efficiently ask for the `wants` and `haves` of references that have
changed -- saving some data sent over to the serving side.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
When the `rad inbox` command would come across a reference name that
it did not recognise as a `Cob` or `Branch`, then it would fail with
an `Unknown` error.
Instead, add an `Unknown` variant to `NotificationKind` which can be
displayed in the `inbox` CLI, by using the `--show-unknown`
flag. Otherwise, it will be skipped and the inbox will work as usual.
While refactoring, the `TypedId` type is used for the `Cob` variant to
simplify some of its uses.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
For larger fetches, a timeout for the channels and socket control are
required. Bump the timeout from 9s to 30s.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
If any replies to patches were made, it was not possible to see them
in the output of `rad patch show`.
Add the replies to show output, but only if the `--verbose` flag is
set.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
Implement caching for the `Issue` and `Patch` COB. This is achieved by
having the `Transaction::initial` and `Transaction::commit` methods
take a cache which can update the newly created/updated object. The
`Store::remove` method also takes the cache for removing the object
from the cache, as well as the repository.
This meant that the `*Mut` types for the respective COBs are required
to carry a cache alongside the repository store. Identities will not
be cached, and so this is always set to `NoCache` -- which performs no
caching, and always succeeds.
To perform cache reading for issues and patches, the `cache::Issues`
and `cache::Patches` traits are introduced. They capture the minimal
amount of methods that are needed for the `rad issue` and `rad patch`
CLI commands.
The implementor for each of these traits is there respective `Cache`
types, which combines their backing repository store and the SQLite
database. They both provide convenience methods:
- `create` (and `draft` for patches)
- `get_mut`
- `remove`
which can be used instead of their repository store counterparts.
All uses of the repository stores (where needed) are replaced with
`Cache` types.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
Add the cob::cache module which describes an SQLite store and schema
for caching COBs. It also provides in-memory and no-cache types for
testing purposes.
The intention is for the store to be used alongside the existing COB
types so that a write-thru cache can be implemented and used for
efficient reads.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
Add derivations of `Serialize` and `Deserialize` for the `Issue` and
`Thread` cob types.
It's necessary to circumvent `Infallible` for the case of the
`Comment` type. This is because `Infallible` does not have these
traits implemented, and there does not seem to be a plan to do so.
Instead, we can use a custom `Infallible` equivalent, which is dubbed
`Never` here.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
Ensure that MockRepository can be used where the COB storage traits
are required. Currently, the implementations of the methods themselves
are not required -- they are left as `todo!()`.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
The standard in the codebase for boolean methods is to prefix them
with `is_`.
Change `resolved` to `is_resolved`.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
The combination of a COB's `TypeName` and `ObjectId` is useful for
parsing a Radicle COB refname, i.e. `refs/cobs/<typename>/<object
id>`.
Introduce `TypedId` for this and provide helper functions for parsing
from the git refname types, as well as being able to tell what kind of
COB it is.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
Previously, authorization was only checked for ops after the root op.
We also avoid creating empty actions for `assign` and `label`, which
was needed for non-authorized users to be able to create issues/patches.
This makes the default policy `block` with scope `all`. The good thing
about this is that you don't have to manage individual nodes in the
beginning. You simply seed the repos you want, and everything is
followed within those repos. This matches user expectations, as `rad
seed` does what you expect it to do.
For open-policy seed nodes, it also means that simply changing `block`
to `allow` from the default config is all you need.
This leaves the non-default `followed` scope as a more "advanced"
setting, since it is much more cumbersome: it forces you to follow a
user before you can see their issues/patches, which is not something
most users expect coming from old-shcool forges.
Shows you updates on local repositories.
Included here is a basic implementation of a notification system. A new
database is created and updated after every fetch, with refs that have
changed.
The `rad inbox` command can then list items in the database and delete
them.
Most of these changes should be self-explanatory.
* Atomic fetch was no longer used
* The default policy when initializing should be 'all', as it is with
`rad init`.
Dequeued fetches were always fetching all refs, instead of fetching the
specific refs that were initially requested. We fix that by storing the
refs to fetch in the fetch queue.
We were not properly guarding against the fact that we could receive
multiple sync events from the same node. Normally though, this wouldn't
happen, as further announcements would be stale.
Instead of using tuples and lots of arrays, this commit tries to use
`BTreeMap`s to provide more context to review, issues, patch and other
comments.
Also the returned object by the httpd is more aligned with what is
needed by the consumer, so less recomputation is needed for display.
When using `network: "test"`, we shouldn't default to any public seed
node. This change makes sure that the "preferredSeeds" configuration var
is only given a value when `network` is `main`.
The command now works with both a replica target *and* a seeds target.
This is especially useful to eg. ensure that a preferred seed has been
synced, while still trying to hit a higher replica count.
Introduce pinned repositories to the `profile::Config`, i.e.
`config.json`.
This is used for pinning repositories in the httpd `/projects` handle.
To allow returning all repositories, an `all` query parameter is also
included.
This is non-backwards compatible change, since we introduce the `show`
query parameter which, if not supplied, will default to `pinned`.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
When evaluating a COB, expose to the `apply` function the operations
concurrent to the current one being applied. These are essentially
"related" nodes in the graph, that are neither ancestors or descendants.
This will let us for example have different failure modes depending on
whether there were conflicting operations, as well as expose conflicts
to the user.
Previously, COB change commits would not distinguish between their
parents. This means that they would try to load source code commits
(eg. for patch COBs) and then fail since those commits wouldn't have a
manifest or `Rad-Resource`.
We introduce a backwards compatible change here which is that parent
oids that are not COB commits are accompanied by a commit trailer called
`Rad-Related`. This is for all related content that should be pulled in
but is not part of the change graph proper.
When building the change graph, we simply ignore commits that are marked
as "related".
We were using the two pretty interchangeably. "Project" should only be
used to refer to the repository payload. "Repository" should be used
when referring to the resource that is fetched, synced, cloned, checked
out etc.
It is possible that there can be errors while configuring a Radicle
`Repository` during `rad::init`, for example:
- configuring working copy remote fails
- signing references fails
- setting the identity head fails
- setting the project head fails
If any of these occur, a best effort is made to remove the
`Repository` and the working copy remote, so that the caller may fix
any issues and retry.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
Add a `Config` constructor for testing seed nodes that uses
`usize::MAX` for capacity.
This allows test scenarios to not drop messages due to rate limiting
-- all nodes are on the same IP address and rate limiting is performed
per IP address.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
To allow `rad sync` to break early from the call to `announce`,
control flow is introduced for the callback.
In `rad sync`, the number of synced nodes is checked against the
`replicas` -- if specified -- and sends a `Break`.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
Instead of having a timeline of a patch be global and sorted by
timestamp, organise it via revisions. Each revision has an entry in
the timeline, sorted by timestamp. Under each revision, a series of
updates are inlined, also sorted by their timestamps. Currently, these
updates can either be reviews or merges.
For example:
```
├────────────────────────────────────────────────────────────────────────────┤
│ ● opened by fintohaps (you) 1 week ago │
│ └── ✓ accepted by fintohaps (you) 1 week ago │
│ ↑ updated to 5f2a123e3f2aa60f2e03815df283563495b73085 (a1c3c0f) 1 week ago │
│ └── ✓ accepted by fintohaps (you) 1 week ago │
│ └── ✓ merged by cloudhead (z6Mkt67…v4N1tRk) now │
╰────────────────────────────────────────────────────────────────────────────╯
```
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
By default, this command now:
* Follows the peer if not already followed
* Syncs refs of the peer, if it wasn't followed before
* Fetches refs into working copy
This functionality can be disabled via `--no-sync` and `--no-fetch`.
Shell out to `git`, avoiding using `git2`. This is to avoid an
issue where somewhere within the fetch there is an attempt to lookup a
`rad/sigrefs` object, which says that the object is missing. It is
suspected that this is due to the object being localised in the same
packfile as other objects we are fetching. Since the `rad/sigrefs`
object is never needed nor used in the working copy, this will always
result in the object missing. This seems to only be an issue with
`libgit2`/`git2` and not `git` itself.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
To make `CodeLocation` reusable across different components, require
that a commit's `Oid` is part of the struct.
An `Oid` is necessary since the `path` can refer to a path at any
point in time throughout the history of the repository. By pinning the
commit that is being referred to, the path will refer to the path
within the associated tree [of the commit].
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
`Infallible` is a more appropriate type in comparsion to `()` for
comments that do not require a `CodeLocation`.
That is because if the type for `location` is `Option<()>`, this means
that `Some(())` can be constructed, however, we never intend to
construct that variant. On the other hand, `Infallible` can never be
constructed, and so, only `None` can be used.
This requires updates to some functions to use the generic `T` and
also handle the different kinds of `Comment`s -- `Infallible` and
`CodeLocation`.
X-Co-Authored-By: Fintan Halpenny <fintan.halpenny@gmail.com>
To be able to use log setup across the different components of
heartwood, expose a `logger` module from `radicle`. This also includes
a `test` logger.
These modules are guarded behind a `logger` feature flag, so that
other crates are not bringing any uneeded dependencies.
This allows logging to be setup in `radicle-cli-test` so that when a
test fails, the log output is present for debugging purposes.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
This test is flakey and requires further investigation. Some extra
assertions are added to ensure that the expected behaviour is
happening upon fetching. It is unknown if the fetch queue is behaving
properly, and it's suspected that some existing fetch results might be
used for queued fetches.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
Previously, we might ban a node if we're having trouble connecting to
it, but this could be due to internet connectivity issues.
We introduce a penalty score for nodes, that increases if there are
connection issues or the node misbehaves, and decreases on successful
connections.
When connected to new nodes, we sort them so that low-penalty nodes are
attempted first.
We create a system for migrating the our databases, as well as a first
migration, for adding a `penalty` field to the `nodes` table, which we
are going to need soon, as an example.
Previously we panicked if a revision wasn't found after an update. But
it can happen that the update goes through yet a concurrent update
prevents the revision from being valid.
To partially deal with this, we don't try to lookup the revision after
an update, we simply return the ID. This means a subsequent lookup can
fail, and the error can be handled there.
Align the naming with the field itself and Patch::assignees.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
This allows the display of any edits done to comments,
replies and issue descriptions.
This commit doesn't remove the body property
which is the latest comment body but adds a edits property.
Also adds an author property to the `Edit` struct,
which allows us to see the author of an edit action.
Update to git2-0.18.1, which also requires updating radicle-git-ext
and radicle-surf.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
Instead of having a routing database, and an addresses database,
consolidate that in a single database called `node.db`.
We also simplify database access, by providing a single type for opening
the database, and use traits to access the various functions.
We create a new `seed::Store` that has seed-related methods split out of
the `address::Store`.
Note that tracking is still in its own database currently, as it isn't
ephemeral state. This makes it easy to delete the new `node.db` without
risk of losing any important information.
To avoid renaming all database methods, we create a `Stores` wrapper
that allows us to access routing, addresses, seeds etc. without type
ambiguity.
This is a breaking change and requires `addresses.db` and `routing.db`
to be deleted.
Allow the caller of `rad sync` to `--sort-by` the NID, alias, or
status of the seeds.
This prevents random ordering, due to timestamps, in CLI testing.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
If the local peer does not have any `rad/sigrefs` then there is no
risk of any data corruption. In this case the repository can be
removed entirely.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
The `remove` method is no longer being used, with `clean` taking its
place.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
Add a `clean` method to remove the remotes of a repository that are
not the local peer nor the delegates.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
Distinguish between repos that are simply tracked, vs. repos that are
cloned (and thus forked). By default, we don't show repos that don't
have a local fork.
Add the `--name` option to `rad patch checkout`. This allows the user
to name the branch they are checking out, rather than use the
`patch/<short-id>` format.
This change includes returning early for setting the upstream branch,
if the upstream already existed.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
Currently, if refs are updated while the node is stopped, these refs are
not immediately announced when the node starts, due to the node being
unaware of the updates.
To remedy this, on startup, we compare the refs we have in storage with
the refs we announced to the network. Since only our own refs could have
changed while the node was stopped, this change only applies to owned
refs. If we detect that storage has changed while the node was stopped,
we create ref announcements for the updated repos and store them in our
gossip database. These announcements will be included with other
historical gossip announcements when nodes connect to us.
To keep track of what was announced, we store our local node's sync
status in the database. This is also useful when using the `rad sync status`
command, as it tells us what oid our local node is at, and whether it
is aware of our local changes yet.
In addition to this, we make a small fix: previously, inventory and ref
announcements originating from our local node were not stored in the
announcement database. We ensure that they are via the new `announce`
method.
This command shows the sync status of a repository by consulting the
database via the node.
To make it work, we call into the address store to update the
`repo-sync-status` table when receiving ref announcements that are
relevant to us.
An informational message for the protocol is useful for sending
messages to a node that is not required to be relayed.
This message is introduced by starting with an ACK message for a set
of references being synced. This fixes an issue for announcing to a
node that is already synced, since it can now ACK that it received the
message and replies that it is synced.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
cloudhead
Lars Wirzenius
Fintan Halpenny
Sebastian Martinez
Alexis Sellier