Implement caching for the `Issue` and `Patch` COB. This is achieved by
having the `Transaction::initial` and `Transaction::commit` methods
take a cache which can update the newly created/updated object. The
`Store::remove` method also takes the cache for removing the object
from the cache, as well as the repository.
This meant that the `*Mut` types for the respective COBs are required
to carry a cache alongside the repository store. Identities will not
be cached, and so this is always set to `NoCache` -- which performs no
caching, and always succeeds.
To perform cache reading for issues and patches, the `cache::Issues`
and `cache::Patches` traits are introduced. They capture the minimal
amount of methods that are needed for the `rad issue` and `rad patch`
CLI commands.
The implementor for each of these traits is there respective `Cache`
types, which combines their backing repository store and the SQLite
database. They both provide convenience methods:
- `create` (and `draft` for patches)
- `get_mut`
- `remove`
which can be used instead of their repository store counterparts.
All uses of the repository stores (where needed) are replaced with
`Cache` types.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
1. Simplify latency calculation to be more predictable, otherwise,
setting a latency is not super useful.
2. Turn a hard error into a log. It's not worth fixing it now though, as
it isn't a problem for the tests.
This is going to be useful for testing message amplification in the
network protocol.
We also reset the state of a few things everytime the simulator is run.
Most of these changes should be self-explanatory.
* Atomic fetch was no longer used
* The default policy when initializing should be 'all', as it is with
`rad init`.
Dequeued fetches were always fetching all refs, instead of fetching the
specific refs that were initially requested. We fix that by storing the
refs to fetch in the fetch queue.
When using `network: "test"`, we shouldn't default to any public seed
node. This change makes sure that the "preferredSeeds" configuration var
is only given a value when `network` is `main`.
Introduce pinned repositories to the `profile::Config`, i.e.
`config.json`.
This is used for pinning repositories in the httpd `/projects` handle.
To allow returning all repositories, an `all` query parameter is also
included.
This is non-backwards compatible change, since we introduce the `show`
query parameter which, if not supplied, will default to `pinned`.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
To be able to use log setup across the different components of
heartwood, expose a `logger` module from `radicle`. This also includes
a `test` logger.
These modules are guarded behind a `logger` feature flag, so that
other crates are not bringing any uneeded dependencies.
This allows logging to be setup in `radicle-cli-test` so that when a
test fails, the log output is present for debugging purposes.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
Some test flakes would occur due to the `converge` function looping
forever. It was noticed that in these cases the log output would look
like:
test: Node z6MkqTKLVnFdVF24PQiDoQffbTPT7U5igghS3XWuFJLfah1P has
{(Id(rad:z39WuDs9hrqmCr9shjoz6dn2Abqyz),
PublicKey(z6MkemMxfWhRQFKy4UPX7FsN8GjLAZ9T65rbmDsUn7fchioP)),
(Id(rad:z39WuDs9hrqmCr9shjoz6dn2Abqyz),
PublicKey(z6MkpdPJp7naAurUjCpwef1BDD5jfPhAvwfinaGd7pVh227j)),
(Id(rad:z39WuDs9hrqmCr9shjoz6dn2Abqyz),
PublicKey(z6MkqTKLVnFdVF24PQiDoQffbTPT7U5igghS3XWuFJLfah1P))}
with further investigation, the variable `all_routes` was found to be
smaller than the calculated routes.
Instead of checking `==`, instead check that
`routes.is_superset(&all_routes)`, which means that `routes` must be
equal to `all_routes`, but allows it have extra entries.
Signed-off-by: Fintan Halpenny <Fintan Halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
Nix Flakes[[0]] are a more modular way of expressing nix derivations.
The `flake.nix` file describes:
- How to build the project, or components of a projet, via `nix build`
- What "checks" can run for a project, which can be executed via
`nix flake check`
- What binaries can be run, e.g. `nix run .#rad` to execute the CLI
This approach is preferable since a lot of the Nix ecosystem is using
Flakes -- despite it still being marked as experimental.
One interesting benefit, that needs confirming, is that a check can be
run from a remote Git source. So if someone creates a patch on
Radicle, someone else with the `nix` binary can execute:
nix flake check git+https://app.radicle.xyz/nodes/seed.radicle.xyz/rad:z3gqcJUoA1n9HaHKufZs5FCSGazv5?refs=patches/<patch id>
This means that anyone with `nix` can check patches without having to
fetch the patch itself.
Another obvious benefit will be that it allows the Radicle binaries to
enter the Nix ecosystem.
The big caveat to both of the above is that, currently, the tests are
failing while running `nix flake check`. They are currently disabled
with `doCheck = false`. They are potentially failing due to some
strange sandboxing issues which are being looked into.
[0]: https://nixos.wiki/wiki/Flakes
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
Since forking creates your `rad/sigrefs`, which then needs to be backed
up forever, we avoid forking on `rad clone`. This should make little or
no difference for users, as the "fork" will be created on `git push`.
Instead of having a routing database, and an addresses database,
consolidate that in a single database called `node.db`.
We also simplify database access, by providing a single type for opening
the database, and use traits to access the various functions.
We create a new `seed::Store` that has seed-related methods split out of
the `address::Store`.
Note that tracking is still in its own database currently, as it isn't
ephemeral state. This makes it easy to delete the new `node.db` without
risk of losing any important information.
To avoid renaming all database methods, we create a `Stores` wrapper
that allows us to access routing, addresses, seeds etc. without type
ambiguity.
This is a breaking change and requires `addresses.db` and `routing.db`
to be deleted.
Currently, if refs are updated while the node is stopped, these refs are
not immediately announced when the node starts, due to the node being
unaware of the updates.
To remedy this, on startup, we compare the refs we have in storage with
the refs we announced to the network. Since only our own refs could have
changed while the node was stopped, this change only applies to owned
refs. If we detect that storage has changed while the node was stopped,
we create ref announcements for the updated repos and store them in our
gossip database. These announcements will be included with other
historical gossip announcements when nodes connect to us.
To keep track of what was announced, we store our local node's sync
status in the database. This is also useful when using the `rad sync status`
command, as it tells us what oid our local node is at, and whether it
is aware of our local changes yet.
In addition to this, we make a small fix: previously, inventory and ref
announcements originating from our local node were not stored in the
announcement database. We ensure that they are via the new `announce`
method.
An informational message for the protocol is useful for sending
messages to a node that is not required to be relayed.
This message is introduced by starting with an ACK message for a set
of references being synced. This fixes an issue for announcing to a
node that is already synced, since it can now ACK that it received the
message and replies that it is synced.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
It can be useful to return the `RefsAt` that was announced for this
node, for example, to compare it to node events.
Change the `Handle` method to return `RefsAt` so that when it is
called it returns the local node's `RefsAt`.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
Previously, we would process any valid inventory or refs announcement.
This could cause problems for keeping sync statuses, as we could get
a refs announcement from a node that isn't in our database.
In practice, there is no good reason why we should receive a refs or
inventory announcement before the announcer's node announcement, unless
nodes are not following the protocol.
Implementing this change meant that many of the tests needed updating,
as it wasn't assumed that a node announcement was needed before
receiving inventories etc. from a peer.
We also had to implement a basic form of node address banning to make
the tests pass after this change.
The RefsAnnouncement previous announced its complete set of
`rad/sigrefs`. This was useful in that verification can occur on the
receiveing side immediately by verifying the refs and proceeding with
the fetch -- note that it did nothing with the ref information it
received.
This can be minimised to only advertising the namespace, i.e. the
NID, and the new SHA of the `rad/sigrefs`. This is represented by a
new message type, `RefsAt`, which contains the NID and the Oid.
For checking if an announcement is stale or fresh, the receiving
side can check the existence of the SHA in its Git repository.
The tests are appropriately modified to use NIDs and Oids.
To prepare for a follow-up change, the announcement data is also
threaded through for the fetch protocol to focus fetching only the
advertised set of Oids.
N.B.: because the refs aren't communicated, the verification happens
when fetching since the data needs to be fetched before
verifying. This is a trade-off with minimising the message payload for
verifying earlier.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
The fetch V2 improvements introduced a `UserInfo` type that is used to
set the Git config's name and email when cloning a repository. This
`UserInfo` can also be used when creating a new `Repository`.
The `Storage` type now takes a `UserInfo` so that it can be passed to
`Repository::create`, for creating a new repository.
For testing, this requires adding a new `fixtures::user` so that
testing `Storage` instances can be easily constructed.
It also means that the `radicle-cli` tests all changed with new SHAs
since the name and email are used as part of the hashing input.
N.b. the order of the timelines while testing the identity cob
changed. It's not clear why that is.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
Instead of storing received gossip messages in an in-memory `BTreeMap`,
we persist them to the database (`addresses.db`). There are two reasons
for this:
1. Node restarts do not wipe the state, meaning that it is much less
likely that some gossip message will get lost before reaching all
nodes.
2. This will allow us in the next commit to know how far behind we are
after a restart, to be able to request the correct time range of
gossip messages that we missed while offline.
Note that we now share the addresses.db file between two connections and
stores, which we don't do anywhere else. Having multiple connections to
the same database is fine, but the way we're having to do it is a bit
awkward. In a future patch, we could pull out the database schema from the
`address` module and rename the database to make things clearer.
The git-daemon code is no longer needed since git-upload-pack is used
in its place.
Remove the git-daemon code from the runtime and worker.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
Wire up the `radicle-fetch` crate to the fetch logic in the
`radicle-node` worker.
Any required traits for storage, identity, tracking, and transport
used by `radicle-fetch`.
The worker adds a new module to run the `git upload-pack` process and
pipes the `stdout`/`stdin` to the respective channels.
The `Worker` type required adding a `FetchConfig` to easily allow the
configuration of fetches and passing the signer and tracking store
location through to the fetch `Handle`.
The channels code is adapted to have a new writer that always flushes
when a call to `write_all` is made. This is necessary since the
Gitoxide code never calls `flush`, and so it will hang since no data
is sent until `flush` is called.
Since `UserInfo` is used to set up a newly cloned repository, this
changed some of the SHA1 hashes output in the CLI test examples. It
was also necessary to update the CLI's clone code to be deterministic
in its output for tests.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
Make the validation logic customisable by extracting it out into a
separate trait that can be defined by other crates.
The Repository validation logic stays the same.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
This change allows users to track/untrack peers and repos without a
running node. This fixes a bug when running `rad init` without the node
running, where the repo would be tracked with a different scope than
what was specified on the CLI.
Introduces a new COB type to store repository identity documents.
The reason for this change is to:
1. Simplify the code, as the identity document logic resembled a COB,
yet it had custom logic. This allows existing COB code to be used for
identities.
2. Make identity document update logic more flexible, since COB actions
can be added in the future.
3. Re-purpose existing tools around COBs to work on identities, eg. `rad
cob`.
4. Unify the concept of an identity change proposal, with regular identity
changes. This means we can remove the `id.proposal` COB in favor of
using the `id` COB itself.
Notes
-----
* Each repository has one Identity COB.
* The `Proposal` COB has been repurposed into the `Identity` COB.
* Identity documents are stored as *embeds* inside the Identity COB
actions.
* The action that contains new document versions is called `revision`,
just like the Patches COB.
* The namespaced `rad/id` ref is a symbolic reference to that Identity
COB.
* The canonical `rad/id` ref is a direct reference to the commit in the
Identity COB that contains the latest *accepted* revision of the
document.
* All commands for managing identities have been folded into `rad id`.
Hence `rad delegate` and `rad edit` are removed.
* The concept of "rebasing" an identity document is gone.
* The `rad id` output has been updated to match the style of other
commands.
* When a revision has enough signatures, it is automatically adopted as
the current identity, there is no longer the need for a "commit"
action.
* When an identity revision is proposed, and the current identity has a
threshold of `1`, that identity is automatically accepted due to the
above point.
* The idea of "verifying a peer's identity branch" no longer applies, as
COBs cannot be verified one history at a time.
* Since the root commit of the Identity COB does not have a "Resource"
to point to (it would have to point to itself, which is impossible),
we've made the resource id optional for COBs.
We were being too strict by requiring that no routes existed that
weren't passed into the function. This isn't really useful for testing,
since we're more interested in checking that certain routes are there,
rather than testing that *only* those routes are there.
When a fetch is user-requested, a timeout can be supplied that is passed
down to the worker.
When a fetch is service-requested, a default timeout is used.
Private repos are implemented by extending the identity document with a
`visibility` attribute, that can either be `"public"` (default) or
`"private"`.
In case of `private` visibility, only the delegates are allowed to view
the repo, as well as any DIDs added to the allow list.
To implement repo visibility, we simply block fetches from and
announcements to peers for whom the repo should remain invisible.
Private repos are also not announced in the `inventory` message, since
the full list of peers that *may* have the repo is retrievable from the
repo identity. This could cause errors if eg. a peer who is allowed to
view the repo doesn't actually have it. However this is an ok trade-off
for now to keep the complexity low. For repos to truly be private, it's
important that the RIDs don't leak either.
Finally, we modify `radicle-httpd` for now to only list public repos.
Eventually, we would want to change this depending on whether an allowed
peer is authenticated with the service or not.
---
It's also worth mentioning why this approach was taken, vs. end-to-end
encryption. The reasons are as follows:
1. Nodes that do not have access to a private repo will generally not want to
replicate encrypted data that they cannot examine or use.
2. The chosen solution is trivial, while encrypting git objects isn't.
3. Performance of the chosen solution is much better, there is no
overhead.
4. Privacy of the chosen solution is better: RIDs are never leaked, and
neither is the existence of a private repo, nor who has access to it.
There is one downside: Paying for storage of private repos is no better
in terms of privacy than what GitHub offers. Hosting providers will have
access to your private repos, if this solution is used.
Add `deny.toml` with a few exceptions, and update certain dependencies
to get rid of duplicates.
Running `cargo deny check` will yield some warnings still, but no
errors.
This change allows for files (blobs) to be embedded into COB entries.
This allows for things like image attachments in issue comments for example.
For now, we only enable this in the `issue` COB.
The way it works is that relevant COB actions carry metadata about which
files are attached to them; and we store those files as blobs inside the
COB entry's *tree* object, under an `embeds/` folder.
The `Embed<T>` type is used for the above, and either carries actual
content, or carries a content-id. Retrieving the actual content is as
simple as asking the repository for a blob with that content-id.
As a possible future extension, MIME types could be stored alongside the
files in a "metadata" file. This could help clients display the content
appropriately.
This adds functionality so that on `rad sync` and `rad clone`, we
connect to known seeds if necessary, before attempting to fetch.
In passing, some related changes were made:
* The `rad sync` command's arguments were reworked:
`--replicas <count>` can be used to specify a replica count when
fetching
* The `rad patch` command no longer has a `--fetch` option
* It's now possible to tell the node not to automatically connect to
peers in the background
* `AddressBook` is being used underneath `Seeds` to allow for shuffled
iteration
Only return once the connection is established or failed.
It's still not perfect, as several failure scenarios are not handled,
but it's an improvement.
As a starting point, we populate the address book (if empty) with some
bootstrap nodes which are trusted.
To prevent these nodes being connected to during tests (including e2e
tests), we create a new constructor for `Config` that is used in tests.
Note that a `cfg(test)` check is not enough given that e2e tests don't
have that set.
cloudhead
Fintan Halpenny
Fintan Halpenny
Alexis Sellier