Instead of re-loading the full inventory on `rad init`, we simply add
the new repository to the in-memory cache.
We also keep track of whether a fetch was a full clone, to know when
to update our routing table.
For large seeds with many repos, it's expensive to traverse the
inventory everytime the node connects to a new peer, for example.
To improve the situation, we cache the inventory per `Storage` instance,
and add a `refresh` method to refresh the inventory.
This method is called when the node is told to synchronize its
inventory.
This change limits the amount of `wants` and `haves` data that is sent
to the serving side of a fetch.
When a `RefsAt` announcement is sent, the fetching peer can calculate
the difference between the `rad/sigrefs` they're aware of -- if it
exists -- and the newly advertised `rad/sigrefs`. This means they can
efficiently ask for the `wants` and `haves` of references that have
changed -- saving some data sent over to the serving side.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
When the `rad inbox` command would come across a reference name that
it did not recognise as a `Cob` or `Branch`, then it would fail with
an `Unknown` error.
Instead, add an `Unknown` variant to `NotificationKind` which can be
displayed in the `inbox` CLI, by using the `--show-unknown`
flag. Otherwise, it will be skipped and the inbox will work as usual.
While refactoring, the `TypedId` type is used for the `Cob` variant to
simplify some of its uses.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
For larger fetches, a timeout for the channels and socket control are
required. Bump the timeout from 9s to 30s.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
If any replies to patches were made, it was not possible to see them
in the output of `rad patch show`.
Add the replies to show output, but only if the `--verbose` flag is
set.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
Implement caching for the `Issue` and `Patch` COB. This is achieved by
having the `Transaction::initial` and `Transaction::commit` methods
take a cache which can update the newly created/updated object. The
`Store::remove` method also takes the cache for removing the object
from the cache, as well as the repository.
This meant that the `*Mut` types for the respective COBs are required
to carry a cache alongside the repository store. Identities will not
be cached, and so this is always set to `NoCache` -- which performs no
caching, and always succeeds.
To perform cache reading for issues and patches, the `cache::Issues`
and `cache::Patches` traits are introduced. They capture the minimal
amount of methods that are needed for the `rad issue` and `rad patch`
CLI commands.
The implementor for each of these traits is there respective `Cache`
types, which combines their backing repository store and the SQLite
database. They both provide convenience methods:
- `create` (and `draft` for patches)
- `get_mut`
- `remove`
which can be used instead of their repository store counterparts.
All uses of the repository stores (where needed) are replaced with
`Cache` types.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
Add the cob::cache module which describes an SQLite store and schema
for caching COBs. It also provides in-memory and no-cache types for
testing purposes.
The intention is for the store to be used alongside the existing COB
types so that a write-thru cache can be implemented and used for
efficient reads.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
Add derivations of `Serialize` and `Deserialize` for the `Issue` and
`Thread` cob types.
It's necessary to circumvent `Infallible` for the case of the
`Comment` type. This is because `Infallible` does not have these
traits implemented, and there does not seem to be a plan to do so.
Instead, we can use a custom `Infallible` equivalent, which is dubbed
`Never` here.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
Ensure that MockRepository can be used where the COB storage traits
are required. Currently, the implementations of the methods themselves
are not required -- they are left as `todo!()`.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
The standard in the codebase for boolean methods is to prefix them
with `is_`.
Change `resolved` to `is_resolved`.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
The combination of a COB's `TypeName` and `ObjectId` is useful for
parsing a Radicle COB refname, i.e. `refs/cobs/<typename>/<object
id>`.
Introduce `TypedId` for this and provide helper functions for parsing
from the git refname types, as well as being able to tell what kind of
COB it is.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
Previously, authorization was only checked for ops after the root op.
We also avoid creating empty actions for `assign` and `label`, which
was needed for non-authorized users to be able to create issues/patches.
This makes the default policy `block` with scope `all`. The good thing
about this is that you don't have to manage individual nodes in the
beginning. You simply seed the repos you want, and everything is
followed within those repos. This matches user expectations, as `rad
seed` does what you expect it to do.
For open-policy seed nodes, it also means that simply changing `block`
to `allow` from the default config is all you need.
This leaves the non-default `followed` scope as a more "advanced"
setting, since it is much more cumbersome: it forces you to follow a
user before you can see their issues/patches, which is not something
most users expect coming from old-shcool forges.
Shows you updates on local repositories.
Included here is a basic implementation of a notification system. A new
database is created and updated after every fetch, with refs that have
changed.
The `rad inbox` command can then list items in the database and delete
them.
Most of these changes should be self-explanatory.
* Atomic fetch was no longer used
* The default policy when initializing should be 'all', as it is with
`rad init`.
Dequeued fetches were always fetching all refs, instead of fetching the
specific refs that were initially requested. We fix that by storing the
refs to fetch in the fetch queue.
We were not properly guarding against the fact that we could receive
multiple sync events from the same node. Normally though, this wouldn't
happen, as further announcements would be stale.
Instead of using tuples and lots of arrays, this commit tries to use
`BTreeMap`s to provide more context to review, issues, patch and other
comments.
Also the returned object by the httpd is more aligned with what is
needed by the consumer, so less recomputation is needed for display.
When using `network: "test"`, we shouldn't default to any public seed
node. This change makes sure that the "preferredSeeds" configuration var
is only given a value when `network` is `main`.
The command now works with both a replica target *and* a seeds target.
This is especially useful to eg. ensure that a preferred seed has been
synced, while still trying to hit a higher replica count.
Introduce pinned repositories to the `profile::Config`, i.e.
`config.json`.
This is used for pinning repositories in the httpd `/projects` handle.
To allow returning all repositories, an `all` query parameter is also
included.
This is non-backwards compatible change, since we introduce the `show`
query parameter which, if not supplied, will default to `pinned`.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
When evaluating a COB, expose to the `apply` function the operations
concurrent to the current one being applied. These are essentially
"related" nodes in the graph, that are neither ancestors or descendants.
This will let us for example have different failure modes depending on
whether there were conflicting operations, as well as expose conflicts
to the user.
Previously, COB change commits would not distinguish between their
parents. This means that they would try to load source code commits
(eg. for patch COBs) and then fail since those commits wouldn't have a
manifest or `Rad-Resource`.
We introduce a backwards compatible change here which is that parent
oids that are not COB commits are accompanied by a commit trailer called
`Rad-Related`. This is for all related content that should be pulled in
but is not part of the change graph proper.
When building the change graph, we simply ignore commits that are marked
as "related".
We were using the two pretty interchangeably. "Project" should only be
used to refer to the repository payload. "Repository" should be used
when referring to the resource that is fetched, synced, cloned, checked
out etc.
It is possible that there can be errors while configuring a Radicle
`Repository` during `rad::init`, for example:
- configuring working copy remote fails
- signing references fails
- setting the identity head fails
- setting the project head fails
If any of these occur, a best effort is made to remove the
`Repository` and the working copy remote, so that the caller may fix
any issues and retry.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
Add a `Config` constructor for testing seed nodes that uses
`usize::MAX` for capacity.
This allows test scenarios to not drop messages due to rate limiting
-- all nodes are on the same IP address and rate limiting is performed
per IP address.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
To allow `rad sync` to break early from the call to `announce`,
control flow is introduced for the callback.
In `rad sync`, the number of synced nodes is checked against the
`replicas` -- if specified -- and sends a `Break`.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
Instead of having a timeline of a patch be global and sorted by
timestamp, organise it via revisions. Each revision has an entry in
the timeline, sorted by timestamp. Under each revision, a series of
updates are inlined, also sorted by their timestamps. Currently, these
updates can either be reviews or merges.
For example:
```
├────────────────────────────────────────────────────────────────────────────┤
│ ● opened by fintohaps (you) 1 week ago │
│ └── ✓ accepted by fintohaps (you) 1 week ago │
│ ↑ updated to 5f2a123e3f2aa60f2e03815df283563495b73085 (a1c3c0f) 1 week ago │
│ └── ✓ accepted by fintohaps (you) 1 week ago │
│ └── ✓ merged by cloudhead (z6Mkt67…v4N1tRk) now │
╰────────────────────────────────────────────────────────────────────────────╯
```
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
By default, this command now:
* Follows the peer if not already followed
* Syncs refs of the peer, if it wasn't followed before
* Fetches refs into working copy
This functionality can be disabled via `--no-sync` and `--no-fetch`.
Shell out to `git`, avoiding using `git2`. This is to avoid an
issue where somewhere within the fetch there is an attempt to lookup a
`rad/sigrefs` object, which says that the object is missing. It is
suspected that this is due to the object being localised in the same
packfile as other objects we are fetching. Since the `rad/sigrefs`
object is never needed nor used in the working copy, this will always
result in the object missing. This seems to only be an issue with
`libgit2`/`git2` and not `git` itself.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
To make `CodeLocation` reusable across different components, require
that a commit's `Oid` is part of the struct.
An `Oid` is necessary since the `path` can refer to a path at any
point in time throughout the history of the repository. By pinning the
commit that is being referred to, the path will refer to the path
within the associated tree [of the commit].
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
`Infallible` is a more appropriate type in comparsion to `()` for
comments that do not require a `CodeLocation`.
That is because if the type for `location` is `Option<()>`, this means
that `Some(())` can be constructed, however, we never intend to
construct that variant. On the other hand, `Infallible` can never be
constructed, and so, only `None` can be used.
This requires updates to some functions to use the generic `T` and
also handle the different kinds of `Comment`s -- `Infallible` and
`CodeLocation`.
X-Co-Authored-By: Fintan Halpenny <fintan.halpenny@gmail.com>
To be able to use log setup across the different components of
heartwood, expose a `logger` module from `radicle`. This also includes
a `test` logger.
These modules are guarded behind a `logger` feature flag, so that
other crates are not bringing any uneeded dependencies.
This allows logging to be setup in `radicle-cli-test` so that when a
test fails, the log output is present for debugging purposes.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
This test is flakey and requires further investigation. Some extra
assertions are added to ensure that the expected behaviour is
happening upon fetching. It is unknown if the fetch queue is behaving
properly, and it's suspected that some existing fetch results might be
used for queued fetches.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
Previously, we might ban a node if we're having trouble connecting to
it, but this could be due to internet connectivity issues.
We introduce a penalty score for nodes, that increases if there are
connection issues or the node misbehaves, and decreases on successful
connections.
When connected to new nodes, we sort them so that low-penalty nodes are
attempted first.
We create a system for migrating the our databases, as well as a first
migration, for adding a `penalty` field to the `nodes` table, which we
are going to need soon, as an example.
Previously we panicked if a revision wasn't found after an update. But
it can happen that the update goes through yet a concurrent update
prevents the revision from being valid.
To partially deal with this, we don't try to lookup the revision after
an update, we simply return the ID. This means a subsequent lookup can
fail, and the error can be handled there.
Align the naming with the field itself and Patch::assignees.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
This allows the display of any edits done to comments,
replies and issue descriptions.
This commit doesn't remove the body property
which is the latest comment body but adds a edits property.
Also adds an author property to the `Edit` struct,
which allows us to see the author of an edit action.
Update to git2-0.18.1, which also requires updating radicle-git-ext
and radicle-surf.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
Instead of having a routing database, and an addresses database,
consolidate that in a single database called `node.db`.
We also simplify database access, by providing a single type for opening
the database, and use traits to access the various functions.
We create a new `seed::Store` that has seed-related methods split out of
the `address::Store`.
Note that tracking is still in its own database currently, as it isn't
ephemeral state. This makes it easy to delete the new `node.db` without
risk of losing any important information.
To avoid renaming all database methods, we create a `Stores` wrapper
that allows us to access routing, addresses, seeds etc. without type
ambiguity.
This is a breaking change and requires `addresses.db` and `routing.db`
to be deleted.
Allow the caller of `rad sync` to `--sort-by` the NID, alias, or
status of the seeds.
This prevents random ordering, due to timestamps, in CLI testing.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
If the local peer does not have any `rad/sigrefs` then there is no
risk of any data corruption. In this case the repository can be
removed entirely.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
The `remove` method is no longer being used, with `clean` taking its
place.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
cloudhead
Sebastian Martinez
Alexis Sellier
Fintan Halpenny