This change is to help separate the fact the namespacing by a `NodeId` is a
Radicle specific concern. The identifier can, in theory, be any kind of path
component, as long it is valid in Git.
The motivation for the change is to help separate the idea of a device's
`NodeId` vs an agent being an author – for future agent repository work.
The motivation of this change is to move away from tying signing to be
specifically for ed25519. The reason being that the protocol will want move
towards two different kinds of signing – the node signing artifacts, and an
author signing artifacts.
This change captures the former, node signing, by introducing a `Device` type
that is the `NodeId` – a PublicKey underneath the hood – and a signing
mechanism. This allows the replacement of the `Signer` trait being used – which
always assumed a `PublicKey`. Instead, the `Device` is constructed with
`NodeId`.
In `radicle-cob`, a signer is expected to implement
`signature::Signer<ExtendedSignature>`, and everywhere in `radicle`,
`radicle-node`, `radicle-cli`, and `radicle-remote-helper` is expected the
signer is expected to implement `signature::Signer<Signature>`.
A `Device` implements both of these but only requires
`signature::Signer<Signature>` to do so – since an `ExtendedSignature` is
essentially `(PublicKey, Signature)`, and the `NodeId` of the `Device` can be
used.
In fresh environments, without a global git config, the `user.name` and
`user.email` will not be set for the fixture repository. This causes issues when
tests are run and attempt to call `git2::Signature::now` – it will fail to read
a user name and email for the commit.
To remedy this, the name and email are set for the repository when initialising
it.
The aim of this change is to make the `Doc` type more safe to use by approaching
the design via [Parse don't validate][[0]] approach.
The problem with the previous approach was that all field were `pub` and thus a
`Doc<Verified>` could easily be mutated and serialized. Granted, the code that
used the serialization would tend to verify the `Doc` first, however, this
approach *ensures* that only a verified `Doc` can be serialized. It also meant
that trying to add new data that would follow the parse approach would require
more generic parameters on top of the existing `PhantomData` parameter, i.e. we
need to do something like: `Doc<RawField, V> -> Doc<ValidField, V>`.
The new approach splits the type into two separate types: `RawDoc` and `Doc`.
The former is allowed to be mutated at will, and uses types that are less
strict. The latter is the valid type that can only be constructed by validating
a `RawDoc` (or the `initial` constructor). The `Doc` type's fields can then only
be accessed by read-only methods.
Solves the problems above by only allowing mutations to `RawDoc`, as well as,
new fields being added to `RawDoc` which are then validated via
`RawDoc::verified`.
[0]: https://lexi-lambda.github.io/blog/2019/11/05/parse-don-t-validate/
We ensure that a `rad/root` ref is included in the signed refs file
under `rad/sigrefs` for all remotes. This prevents a certain kind of
"grafting" attack where signed refs can be copied between repositories,
by having the peer sign over the identity root together with the data refs.
When verifying signed refs, we ensure that the ref is present and points
to an identity branch root that matches the repository identity containing
the signed refs.
Alternatives: lots of alternatives were considered, but this one doesn't
introduce any changes to the signing. The `rad/id/root` name was
considered but is invalid due to `rad/id`.
1. Adds a user-agent string to the node announcement. This lets us keep
track of what software version everyone is running. Especially useful
to see what percentage of the network has upgraded to a new release.
For old nodes, this defaults to the string `/radicle/`.
The user agent format is roughly based on BIP 0014.
2. Advertize protocol version in node announcement.
By advertizing the protocol version, nodes can decide to only connect to
peers with a compatible version. The default and current version is `1`.
We bundle these two changes as they both modify the node announcement
and node database.
3. Swap the order of the fields in announcements to make the message
extensible.
Previously, our advertized inventory was not always correct: when we
stopped seeding a repository, it was still advertised as part of our
inventory until node restart, because the `Storage` type doesn't have
access to seeding policies.
In this change, we remove the concept of inventory from `Storage` and
make the authoritative place for it the routing table in our database.
To make this work, we have to add our node to the database on profile
creation, to not violate the foreign key constraint on the routing
table. Hence, the tests are changed to include our alias which is now
always available.
Implement this function on `RemoteRepository`, and move `repositories`
to the trait level.
This function will be used to quickly get the `RefsAt` of all repository
remotes.
Instead of re-loading the full inventory on `rad init`, we simply add
the new repository to the in-memory cache.
We also keep track of whether a fetch was a full clone, to know when
to update our routing table.
For large seeds with many repos, it's expensive to traverse the
inventory everytime the node connects to a new peer, for example.
To improve the situation, we cache the inventory per `Storage` instance,
and add a `refresh` method to refresh the inventory.
This method is called when the node is told to synchronize its
inventory.
Ensure that MockRepository can be used where the COB storage traits
are required. Currently, the implementations of the methods themselves
are not required -- they are left as `todo!()`.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
Dequeued fetches were always fetching all refs, instead of fetching the
specific refs that were initially requested. We fix that by storing the
refs to fetch in the fetch queue.
The `remove` method is no longer being used, with `clean` taking its
place.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
Add a `clean` method to remove the remotes of a repository that are
not the local peer nor the delegates.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
Currently, if refs are updated while the node is stopped, these refs are
not immediately announced when the node starts, due to the node being
unaware of the updates.
To remedy this, on startup, we compare the refs we have in storage with
the refs we announced to the network. Since only our own refs could have
changed while the node was stopped, this change only applies to owned
refs. If we detect that storage has changed while the node was stopped,
we create ref announcements for the updated repos and store them in our
gossip database. These announcements will be included with other
historical gossip announcements when nodes connect to us.
To keep track of what was announced, we store our local node's sync
status in the database. This is also useful when using the `rad sync status`
command, as it tells us what oid our local node is at, and whether it
is aware of our local changes yet.
In addition to this, we make a small fix: previously, inventory and ref
announcements originating from our local node were not stored in the
announcement database. We ensure that they are via the new `announce`
method.
It can be useful to be able to check if a repository contains a given
commit. Add a `contains` method to the `ReadRepository` trait to
handle this.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
The RefsAnnouncement previous announced its complete set of
`rad/sigrefs`. This was useful in that verification can occur on the
receiveing side immediately by verifying the refs and proceeding with
the fetch -- note that it did nothing with the ref information it
received.
This can be minimised to only advertising the namespace, i.e. the
NID, and the new SHA of the `rad/sigrefs`. This is represented by a
new message type, `RefsAt`, which contains the NID and the Oid.
For checking if an announcement is stale or fresh, the receiving
side can check the existence of the SHA in its Git repository.
The tests are appropriately modified to use NIDs and Oids.
To prepare for a follow-up change, the announcement data is also
threaded through for the fetch protocol to focus fetching only the
advertised set of Oids.
N.B.: because the refs aren't communicated, the verification happens
when fetching since the data needs to be fetched before
verifying. This is a trade-off with minimising the message payload for
verifying earlier.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
The minimal error for the `ReadRepository::reference_oid` method is
`git::raw::Error`.
Change the error type from `git::ext::Error` to `git::raw::Error`.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
The fetch V2 improvements introduced a `UserInfo` type that is used to
set the Git config's name and email when cloning a repository. This
`UserInfo` can also be used when creating a new `Repository`.
The `Storage` type now takes a `UserInfo` so that it can be passed to
`Repository::create`, for creating a new repository.
For testing, this requires adding a new `fixtures::user` so that
testing `Storage` instances can be easily constructed.
It also means that the `radicle-cli` tests all changed with new SHAs
since the name and email are used as part of the hashing input.
N.b. the order of the timelines while testing the identity cob
changed. It's not clear why that is.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
Add the ability to remove a repository, given its RID.
This is useful in the case of attempting to clone a repository, and if
that process fails -- for example, if the repository is private --
then it's necessary to clean up the initially created repository.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
Make the validation logic customisable by extracting it out into a
separate trait that can be defined by other crates.
The Repository validation logic stays the same.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
Instead of returning the unsigned references, the validation of a
repository's remotes returns the set of validation errors. The caller
is then expected to handle these validation errors appropriately, for
example, by logging them and re-raising an error.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
Add a type to capture user information that can be used for Git
configuration and signatures.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
Introduces a new COB type to store repository identity documents.
The reason for this change is to:
1. Simplify the code, as the identity document logic resembled a COB,
yet it had custom logic. This allows existing COB code to be used for
identities.
2. Make identity document update logic more flexible, since COB actions
can be added in the future.
3. Re-purpose existing tools around COBs to work on identities, eg. `rad
cob`.
4. Unify the concept of an identity change proposal, with regular identity
changes. This means we can remove the `id.proposal` COB in favor of
using the `id` COB itself.
Notes
-----
* Each repository has one Identity COB.
* The `Proposal` COB has been repurposed into the `Identity` COB.
* Identity documents are stored as *embeds* inside the Identity COB
actions.
* The action that contains new document versions is called `revision`,
just like the Patches COB.
* The namespaced `rad/id` ref is a symbolic reference to that Identity
COB.
* The canonical `rad/id` ref is a direct reference to the commit in the
Identity COB that contains the latest *accepted* revision of the
document.
* All commands for managing identities have been folded into `rad id`.
Hence `rad delegate` and `rad edit` are removed.
* The concept of "rebasing" an identity document is gone.
* The `rad id` output has been updated to match the style of other
commands.
* When a revision has enough signatures, it is automatically adopted as
the current identity, there is no longer the need for a "commit"
action.
* When an identity revision is proposed, and the current identity has a
threshold of `1`, that identity is automatically accepted due to the
above point.
* The idea of "verifying a peer's identity branch" no longer applies, as
COBs cannot be verified one history at a time.
* Since the root commit of the Identity COB does not have a "Resource"
to point to (it would have to point to itself, which is impossible),
we've made the resource id optional for COBs.
We employ various tricks to ensure that tests are reproducible.
In this case, having stable commit ids allows us to test specific
traversal orders of a COB history in cases of concurrent updates.
We employ a better algorithm for determining which commit should be
canonical when there are multiple possible options.
The algorithm still doesn't work in certain cases when there are
multiple merge commits as candidates, with shared parents.
We leave some (ignored) failing tests to be resolved in the future.
Private repos are implemented by extending the identity document with a
`visibility` attribute, that can either be `"public"` (default) or
`"private"`.
In case of `private` visibility, only the delegates are allowed to view
the repo, as well as any DIDs added to the allow list.
To implement repo visibility, we simply block fetches from and
announcements to peers for whom the repo should remain invisible.
Private repos are also not announced in the `inventory` message, since
the full list of peers that *may* have the repo is retrievable from the
repo identity. This could cause errors if eg. a peer who is allowed to
view the repo doesn't actually have it. However this is an ok trade-off
for now to keep the complexity low. For repos to truly be private, it's
important that the RIDs don't leak either.
Finally, we modify `radicle-httpd` for now to only list public repos.
Eventually, we would want to change this depending on whether an allowed
peer is authenticated with the service or not.
---
It's also worth mentioning why this approach was taken, vs. end-to-end
encryption. The reasons are as follows:
1. Nodes that do not have access to a private repo will generally not want to
replicate encrypted data that they cannot examine or use.
2. The chosen solution is trivial, while encrypting git objects isn't.
3. Performance of the chosen solution is much better, there is no
overhead.
4. Privacy of the chosen solution is better: RIDs are never leaked, and
neither is the existence of a private repo, nor who has access to it.
There is one downside: Paying for storage of private repos is no better
in terms of privacy than what GitHub offers. Hosting providers will have
access to your private repos, if this solution is used.
Instead of having two types, we simply have one, called `Entry`, which
we rename from `Change`.
There was no real benefit to having two almost identical types, but lots
of added complexity. This patch simplifies the crate by removing one of
the types.
Add `deny.toml` with a few exceptions, and update certain dependencies
to get rid of duplicates.
Running `cargo deny check` will yield some warnings still, but no
errors.
This change allows for files (blobs) to be embedded into COB entries.
This allows for things like image attachments in issue comments for example.
For now, we only enable this in the `issue` COB.
The way it works is that relevant COB actions carry metadata about which
files are attached to them; and we store those files as blobs inside the
COB entry's *tree* object, under an `embeds/` folder.
The `Embed<T>` type is used for the above, and either carries actual
content, or carries a content-id. Retrieving the actual content is as
simple as asking the repository for a blob with that content-id.
As a possible future extension, MIME types could be stored alongside the
files in a "metadata" file. This could help clients display the content
appropriately.
This new trait is used when only signing refs is needed, and not general
write access to a repository.
We also implement `ReadRepository` and `SignRepository` for
`DraftStore`.
This is a fairly substantial change, which adds a new configuration file
to the user's `$RAD_HOME` that includes node configuration, including
the alias, and also makes node aliases required.
When running `rad auth`, the user is now prompted for an alias, which
defaults to `$USER`.
When running `rad self`, the alias is now shown.
If the user runs radicle without a config file, the defaults are loaded,
and `$USER` is used as the alias.
There were a few subtle issues with the apply logic of certain COBs
that should be fixed now.
Note that the underlying store guarantees exactly-once delivery, and so
it does not make sense to test idempotence at the patch level.
When applying a `Merge` action, we verify that the commit is in the
history of the default branch of that user.
Since this is not the case when issuing a `rad merge`, and since that
command is no longer needed with the push-based merge, we remove the
`rad merge` command completely.
Fintan Halpenny
cloudhead
Alexis Sellier