Introduces a new COB type to store repository identity documents.
The reason for this change is to:
1. Simplify the code, as the identity document logic resembled a COB,
yet it had custom logic. This allows existing COB code to be used for
identities.
2. Make identity document update logic more flexible, since COB actions
can be added in the future.
3. Re-purpose existing tools around COBs to work on identities, eg. `rad
cob`.
4. Unify the concept of an identity change proposal, with regular identity
changes. This means we can remove the `id.proposal` COB in favor of
using the `id` COB itself.
Notes
-----
* Each repository has one Identity COB.
* The `Proposal` COB has been repurposed into the `Identity` COB.
* Identity documents are stored as *embeds* inside the Identity COB
actions.
* The action that contains new document versions is called `revision`,
just like the Patches COB.
* The namespaced `rad/id` ref is a symbolic reference to that Identity
COB.
* The canonical `rad/id` ref is a direct reference to the commit in the
Identity COB that contains the latest *accepted* revision of the
document.
* All commands for managing identities have been folded into `rad id`.
Hence `rad delegate` and `rad edit` are removed.
* The concept of "rebasing" an identity document is gone.
* The `rad id` output has been updated to match the style of other
commands.
* When a revision has enough signatures, it is automatically adopted as
the current identity, there is no longer the need for a "commit"
action.
* When an identity revision is proposed, and the current identity has a
threshold of `1`, that identity is automatically accepted due to the
above point.
* The idea of "verifying a peer's identity branch" no longer applies, as
COBs cannot be verified one history at a time.
* Since the root commit of the Identity COB does not have a "Resource"
to point to (it would have to point to itself, which is impossible),
we've made the resource id optional for COBs.
We were being too strict by requiring that no routes existed that
weren't passed into the function. This isn't really useful for testing,
since we're more interested in checking that certain routes are there,
rather than testing that *only* those routes are there.
When a fetch is user-requested, a timeout can be supplied that is passed
down to the worker.
When a fetch is service-requested, a default timeout is used.
Private repos are implemented by extending the identity document with a
`visibility` attribute, that can either be `"public"` (default) or
`"private"`.
In case of `private` visibility, only the delegates are allowed to view
the repo, as well as any DIDs added to the allow list.
To implement repo visibility, we simply block fetches from and
announcements to peers for whom the repo should remain invisible.
Private repos are also not announced in the `inventory` message, since
the full list of peers that *may* have the repo is retrievable from the
repo identity. This could cause errors if eg. a peer who is allowed to
view the repo doesn't actually have it. However this is an ok trade-off
for now to keep the complexity low. For repos to truly be private, it's
important that the RIDs don't leak either.
Finally, we modify `radicle-httpd` for now to only list public repos.
Eventually, we would want to change this depending on whether an allowed
peer is authenticated with the service or not.
---
It's also worth mentioning why this approach was taken, vs. end-to-end
encryption. The reasons are as follows:
1. Nodes that do not have access to a private repo will generally not want to
replicate encrypted data that they cannot examine or use.
2. The chosen solution is trivial, while encrypting git objects isn't.
3. Performance of the chosen solution is much better, there is no
overhead.
4. Privacy of the chosen solution is better: RIDs are never leaked, and
neither is the existence of a private repo, nor who has access to it.
There is one downside: Paying for storage of private repos is no better
in terms of privacy than what GitHub offers. Hosting providers will have
access to your private repos, if this solution is used.
Add `deny.toml` with a few exceptions, and update certain dependencies
to get rid of duplicates.
Running `cargo deny check` will yield some warnings still, but no
errors.
This change allows for files (blobs) to be embedded into COB entries.
This allows for things like image attachments in issue comments for example.
For now, we only enable this in the `issue` COB.
The way it works is that relevant COB actions carry metadata about which
files are attached to them; and we store those files as blobs inside the
COB entry's *tree* object, under an `embeds/` folder.
The `Embed<T>` type is used for the above, and either carries actual
content, or carries a content-id. Retrieving the actual content is as
simple as asking the repository for a blob with that content-id.
As a possible future extension, MIME types could be stored alongside the
files in a "metadata" file. This could help clients display the content
appropriately.
This adds functionality so that on `rad sync` and `rad clone`, we
connect to known seeds if necessary, before attempting to fetch.
In passing, some related changes were made:
* The `rad sync` command's arguments were reworked:
`--replicas <count>` can be used to specify a replica count when
fetching
* The `rad patch` command no longer has a `--fetch` option
* It's now possible to tell the node not to automatically connect to
peers in the background
* `AddressBook` is being used underneath `Seeds` to allow for shuffled
iteration
Only return once the connection is established or failed.
It's still not perfect, as several failure scenarios are not handled,
but it's an improvement.
As a starting point, we populate the address book (if empty) with some
bootstrap nodes which are trusted.
To prevent these nodes being connected to during tests (including e2e
tests), we create a new constructor for `Config` that is used in tests.
Note that a `cfg(test)` check is not enough given that e2e tests don't
have that set.
It turns out that the CRDT formed by the union of Git DAGs
is enough to guarantee everything we need for COBs.
This changes the following things:
* COB operations no longer need to be commutative
* COB histories are traversed in the same deterministic order on all
replicas
* It's now possible to implement RSMs on top of COBs, eg. scripting
* Lamport clocks have been removed
* `radicle-crdt` is no longer a dependency of `radicle`
* COBs are no longer instances of `Semilattice`
* The `Ops` type was removed in favor of having `Op` contain multiple
actions
This new trait is used when only signing refs is needed, and not general
write access to a repository.
We also implement `ReadRepository` and `SignRepository` for
`DraftStore`.
The tracking database can be opened in read-only or read-write
mode. Knowing which is being used is significant in concurrent
scenarios, e.g. it's safer to have a read-only handle on multiple
threads.
Provide static type information by signifying what kind of database
handle is being worked with, read or read-write.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
This is a fairly substantial change, which adds a new configuration file
to the user's `$RAD_HOME` that includes node configuration, including
the alias, and also makes node aliases required.
When running `rad auth`, the user is now prompted for an alias, which
defaults to `$USER`.
When running `rad self`, the alias is now shown.
If the user runs radicle without a config file, the defaults are loaded,
and `$USER` is used as the alias.
* Add `--home` flag
* Organize the information better
Now looks like this:
DID did🔑z6MknSLrJoTcukLrE435hVNQT4JUhbvWLX4kUzqkEStBU8Vi
└╴Node ID (NID) z6MknSLrJoTcukLrE435hVNQT4JUhbvWLX4kUzqkEStBU8Vi
SSH running (823)
├╴Key (hash) SHA256:UIedaL6Cxm6OUErh9GQUzzglSk7VpQlVTI1TAFB/HWA
└╴Key (full) ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIHahWSBEpuT1ESZbynOmBNkLBSnR32Ar4woZqSV2YNH1
Home [..]
├╴Storage [..]/storage
├╴Keys [..]/keys
└╴Node [..]/node
For the purposes of debugging running nodes, it's useful to know which
threads are doing what. This allows tools like `htop` to reveal which
threads are running.
We consolidate the setup done by `rad checkout`, `rad clone` and `rad
remote add`, so that they all check for node aliases, and are all
capable of setting up tracking branches etc.
Most of the test flakiness was caused by the git-daemon not being able
to bind to a port. It seems like using an RNG was not a good enough
solution.
After this change, tests seem to be passing without issue.
Previously, we only sent the announcement when there were external
addresses. But it contains other information like the node alias,
therefore we always send it.
Since the proof-of-work can be expensive to compute everytime, we add
the ability to load the local node announcement from the file system.
We also change the proof-of-work parameters so that they are relaxed
when running in debug mode, instead of when running tests.
Finally, we remove the boolean validation of the PoW. Instead, we store
the work per-node, in the database, to be used in the future in case of
DoS attacks or network congestion.
Previously, only one connection to the node control socket could be
handled at a time. This usually worked fine, but if one connection hangs
for whatever reason, no other process can access the node.
When attempting to connect to a peer, and when succeeding, write the timestamp
to the database.
This will allow us to better select addresses and not retry peers to
soon.
We were allowing remotes to be constructed via a function that didn't
check its inputs.
This is now fixed by moving the public key into the `SignedRefs` struct.
This command announces refs to peers and waits for them to be
in sync.
Adds a subscribe method to `Handle` so that we can get the events
from the seed to allow us to confirm that the configured seeds fetched
from us.
Instead of returning the namespaces that we fetched with,
we return the remotes that were actually fetched from.
This can differ if some trusted peers were not available
on the remote node.
Previously, if that was the case, it would cause errors
since the remotes were looked up and that lookup failed.
With this change, we not only verify the signed refs before
the repository "transfer", but also validate that all refs in the
repo are signed and there is no discrepancy with the signed refs
file.
To improve the reliability and flexibility of the protocol, we introduce
multiplexing over peer connections. This involves a new `Frame` type
that carries a stream-id and payload.
Three stream types are made available:
1. Control
2. Gossip
3. Git
This change brings the following improvements:
* Removed need to queue fetch requests
* Removed need to queue gossip messages
* Removed need for `Fetch` and `FetchOk` messages
* Service doesn't need to know about inbound fetches
* Removed one round-trip for fetch negotiation
* Removed special `done` git packet
* Removed session logic and state around Git/Fetch protocols
* Removed code around upgrading/downgrading transport
* Worker in responder mode is able to process any number of fetches
* Connections support any number of concurrent fetches
We had to introduce a few extra things however to make it all work:
* A `VarInt` type for variable-length integers, since we want the frames
to be lightweight.
* A custom "tunnel" implementation, since we couldn't use the existing
one anymore.
Overall the change removes more complexity than it adds, while improving
the protocol along the way.
The One variant caused many paint points for fetching logic, where it
was not necessary. It was only constructed in one place, which could
be replaced by using the variant that holds a set of keys.
Remove the Namespaces::One variant and rename Namespaces::Many to
Namespaces::Trusted.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
When a repository does not yet exist during a fetch, i.e. a clone,
then only delegates are being fetched.
Augment Namespaces::Many to hold the trusted and optional delegate peers
separately. Doing so allows the construction of the variant without
the repository existing, but tracking relationships existing.
This variant can then be used to build refspecs for both the trusted
peers and delegates when doing a cloning fetch.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
The tracked nodes include nodes marked tracked in the db, and
the delegates and `Namespaces::All` if it is a new repo.
Signed-off-by: Han Xu <keepsimple@gmail.com>
First, we make sure to always attempt reconnection to persistent peers,
no matter the reason.
Second, we don't re-attempt connections when an inbound peer
disconnects. This minimizes the chance of connections crossing.
Third, we clean up `session::Error`, and don't use it to signal a
missing session, since it doesn't really belong there.
It was possible to trigger multiple connection attempts to the same
peer, since we didn't track connection attempts that hadn't yet
returned to the service via `Service::attempted`.
With this change, we add an extra `Initial` state to sessions, so
that redundant connections are not attempted when one is already under
way.
To ensure the behaviour of ignoring references that are owned by the
local peer a test case is added.
The case has two peers: alice and bob, where alice creates a new
repository. bob fetches to be up-to-date with alice. alice, in turn,
creates an issue and fetches from bob -- which should be successful.
If the refspec:
^refs/namespaces/<alice>/*
was not in place, her fetch from bob would fail since the signed
references would not be a fast-forward.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
To determine the correct Namespaces to use, the repo Policy and Scope
are consulted.
If the policy is Block then this is a failure state and is reported to
the other end.
Otherwise, if the policy is Track, then the scope is checked. For the
Trusted Scope then the Namespaces should consist of all tracked nodes
and delegates for the given repository. However, the repository might
not exist and so it is necessary for the whole repository to be
fetched.
For the All Scope, all Namespaces are fetched.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
The fetch implementation will potentially fetch the local operator's
remote, which is unnecessary and could result in deleted data.
Fix this by threading the local operator's public key and add an
ignore refspec to the fetch arguments, i.e.
^refs/namespaces/<local>/*
The implementation also strictly fetches all or one namespace. This is
improved upon by introducing a new variant for specifying a set of
namespaces.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
The method `track_repo` only accepted the RID as the argument for
tracking.
Instead, allow passing Scope as an argument for tracking a
repository.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
The tracking configuration has no way of consulting for a configured
scope.
This change allows the storing of this scope inside the tracking
`Config` -- currently unused.
The default Scope is chosen as Trusted, so that nodes do not
implicitly track any other node. If this behaviour is desired -- for
example, for seed nodes -- then All should be chosen.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
Previously the alias field was simply a string which could possibly be
empty. Some checks were made to see if this string was empty.
Instead, make the logic more obvious by making the alias field
optional.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
The `routes_to` function can loop infinitely during test.
To prevent this from happening a number of tries variable is added. It
checks that the number of tries does not exceed 30, which gives the
function about 3 seconds to complete -- due to the call to
`thread::sleep` using 100ms.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
The MockStorage holds an inventory of repositories and their
identities but when asking for a MockRepository, none of this
information is transferred.
Add `id` and `doc` fields to MockRepository for defining some of its
functionality.
In addition, add a generator for a nonempty MockStorage to use in
tests that will expect the storage to be populated.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
A fetch is a client-server interaction and so there are two sides to
this interaction. These sides are generally referred to as receive --
the client side -- and upload -- the server side.
The `Fetch` type indicated whether it was doing a receive or upload by
using a field `initiated: bool`. However, this can be confusing when
you pair it with the `namespaces: Namespaces` field, since only the
receive side should be sending what refspecs it wants and the upload
side responds with those matching refspecs.
A better way to represent this is using an enum for which direction
the fetch is being considered, where the receive side contains the
Namespaces. To borrow from Noise, the variant names are 'Initiator'
for the client and 'Responder' for the server.
The resulting code then makes decisions based on what variant of the
enum was passed.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
The routing function for Node was not implemented and there were no
functions for getting any tracking information.
This implements routing, tracked_repos, and tracked_nodes. To
accomplish this it was necessary to add associated types to the
Handler trait so that implementations could differ in the types they
use for returning this data, i.e. the control socket needs to use a
channel while the Node needs to return serializable data.
Currently, the Node uses a `Vec` but it would be more beneficial to
return an Iterator directly to ensure we are not always hitting the
heap.
To allow for the tracking types to be used across radicle-node (the
crate) and in radicle::node (the module), it was necessary to move
the types to the common place, i.e. radicle::node. At the same time,
new structs were added to make it easier to refer to tracked repos vs
tracked nodes.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
Previously, the seeds information would only consist of the connected
NodeIds.
Improve on this by adding any disconnected and fetching seeds as
well. A seed can either be disconnected, connected, or fetching --
where fetching implies connected.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
The Handle trait defines a FetchResult associated type. This type is
always instantiated to be the FetchResult enum in radicle::node.
Remove the associated type to simplify the trait.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
If a path was passed to `Home` in non-canonical form it would not use
the right path for any initialisation or use.
The aim is to use the `canonicalize` function for paths. However, this
may return an error if none of the paths exist. To ensure that the
path exists the `new` constructor for `Home` creates the directory and
canonicalizes the path.
Since `new` now initialises the home directory, it would be useful if
it also initialised all the necessary subdirectories -- which removes
the need for the `init` method.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
When a specific tracking policy isn't found, we allow setting a default.
To preserve existing behavior, the default is `Block`, ie. if a node
isn't tracked explicitly, it is blocked.
However, this change allows for using `Track` as the default, which
is a way to track everything without having to know the entities in
advance.
The last commit of a newly cloned repository should remain untouched by
making `Node::project()` use a repository fixture instead of a randomly
generated one.
Signed-off-by: Slack Coder <slackcoder@server.ky>
* Read git request line before starting daemon, so that we don't leave
git data in the socket read buffer in case of an error connecting to
the daemon.
* Send a response on the fetch channel even if we disconnect the remote.
* Don't send `done` packet if there's an error; it won't be expected.
We weren't updating the sessions map when a persistent peer
was re-attempted. It would then panic when `connected` was called,
because the state would be `Disconnected`.
Rust 1.67 was announced[0]. Update the necessary files for running
1.67 and fix the clippy suggestions for string interpolation.
[0]: https://blog.rust-lang.org/2023/01/26/Rust-1.67.0.html
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
This change makes sure that nodes don't gossip announcements back to
the announcers.
Since the tests were relying on this mechanism to populate the routing
table, we had to change a few more things, such as adding the inventory
to the node's routing table.
Signed-off-by: Alexis Sellier <self@cloudhead.io>
There are few unrelated changes in this commit that I wasn't able to
split out, mostly due to laziness, so I'll list them here:
* The node startup logic is moved out of the main function, and into a
dedicated functionin `client.rs`.
* There is a tentative worker implementation that is untested, and
handles git replication both ways. It's likely lacking in various ways
but I wanted to get it in there anyway.
* `Address` was moved from `radicle-node` to `radicle`, because it's
needed to send `Connect` commands from the node handle.
* The `netservices` dependency was updated.
* Finally, there is a first end-to-end node test, which currently
doesn't use NoiseXK, but uses the new reactor in `netservices`.
Signed-off-by: Alexis Sellier <self@cloudhead.io>
cloudhead
Alexis Sellier
Fintan Halpenny
xphoniex
Alexis Sellier
Han Xu
Slack Coder
Dr. Maxim Orlovsky