This is going to be useful for testing message amplification in the
network protocol.
We also reset the state of a few things everytime the simulator is run.
Shows you updates on local repositories.
Included here is a basic implementation of a notification system. A new
database is created and updated after every fetch, with refs that have
changed.
The `rad inbox` command can then list items in the database and delete
them.
Most of these changes should be self-explanatory.
* Atomic fetch was no longer used
* The default policy when initializing should be 'all', as it is with
`rad init`.
Dequeued fetches were always fetching all refs, instead of fetching the
specific refs that were initially requested. We fix that by storing the
refs to fetch in the fetch queue.
When using `network: "test"`, we shouldn't default to any public seed
node. This change makes sure that the "preferredSeeds" configuration var
is only given a value when `network` is `main`.
The previous change to the wire protocol introduced a bug: outbound
peers that failed to connect fully would not notify the service on
disconnect. This change fixes it.
* Make sure we cleanup pending peer states when disconnecting before
handshake.
* Make sure we don't panic if a peer disconnects before being fully
connected.
Introduce pinned repositories to the `profile::Config`, i.e.
`config.json`.
This is used for pinning repositories in the httpd `/projects` handle.
To allow returning all repositories, an `all` query parameter is also
included.
This is non-backwards compatible change, since we introduce the `show`
query parameter which, if not supplied, will default to `pinned`.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
When a DNS error occured, the `dial` call failed and we didn't get a
connection attempt, just a disconnect. We fix that by always calling
back into the service when we attempt a connection.
There was an issue with the old version due to the use of `RawFd` as
peer IDs, since they are not unique for the lifetime of a process.
With this change, we use the new `ResourceId` type, though these are not
immediately available, as they are generated when the resource is
registered.
Allow some margin of error when sending multiple inventory announcements
at a time. Previously, they might end up with the same timestamp, so
only the first announcement would be processed by nodes. Now, we ensure
that the timestamp is updated.
To be able to use log setup across the different components of
heartwood, expose a `logger` module from `radicle`. This also includes
a `test` logger.
These modules are guarded behind a `logger` feature flag, so that
other crates are not bringing any uneeded dependencies.
This allows logging to be setup in `radicle-cli-test` so that when a
test fails, the log output is present for debugging purposes.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
When a peer is performing a background and foreground fetch from two
different peers, the background fetch result was being used as the
result for the foreground fetch. This results in inconsistencies,
since the foreground peer has new updates from themselves.
The reason for this issue was the fetches that were queued were not
correctly being handled, and the other peer's results were being
returned instead.
This fixes the queueing and dequeueing by ensuring that the queued
fetch occurs, passing the correct result channel. This fetch will then
be subsequently dequeued correctly and the foreground fetch obtains
the correct result.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
This test is flakey and requires further investigation. Some extra
assertions are added to ensure that the expected behaviour is
happening upon fetching. It is unknown if the fetch queue is behaving
properly, and it's suspected that some existing fetch results might be
used for queued fetches.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
Previously, we might ban a node if we're having trouble connecting to
it, but this could be due to internet connectivity issues.
We introduce a penalty score for nodes, that increases if there are
connection issues or the node misbehaves, and decreases on successful
connections.
When connected to new nodes, we sort them so that low-penalty nodes are
attempted first.
Some test flakes would occur due to the `converge` function looping
forever. It was noticed that in these cases the log output would look
like:
test: Node z6MkqTKLVnFdVF24PQiDoQffbTPT7U5igghS3XWuFJLfah1P has
{(Id(rad:z39WuDs9hrqmCr9shjoz6dn2Abqyz),
PublicKey(z6MkemMxfWhRQFKy4UPX7FsN8GjLAZ9T65rbmDsUn7fchioP)),
(Id(rad:z39WuDs9hrqmCr9shjoz6dn2Abqyz),
PublicKey(z6MkpdPJp7naAurUjCpwef1BDD5jfPhAvwfinaGd7pVh227j)),
(Id(rad:z39WuDs9hrqmCr9shjoz6dn2Abqyz),
PublicKey(z6MkqTKLVnFdVF24PQiDoQffbTPT7U5igghS3XWuFJLfah1P))}
with further investigation, the variable `all_routes` was found to be
smaller than the calculated routes.
Instead of checking `==`, instead check that
`routes.is_superset(&all_routes)`, which means that `routes` must be
equal to `all_routes`, but allows it have extra entries.
Signed-off-by: Fintan Halpenny <Fintan Halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
Since node announcements are stored and retrieved from disk (due to the
PoW), the timestamp on it is never updated. This means it can be too old
to cache and not sent/received by nodes. Fix this by having a max age.
Instead of creating the DB policies handle when needed, construct the
handle when the `Worker` pool is being constructed.
This replaces the usage of the DB handle with the `Worker`'s field
instead.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
If a node had removed a seeding policy and another node fetched from
it, then it would perform the fetch.
Instead, check the policy during the authorization step and prevent
the fetch is the resulting policy is `Block`.
Note that the policy will be `Block` iff the `Block` policy was
specifically set or the `Block` is the default policy for the node.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
Nix Flakes[[0]] are a more modular way of expressing nix derivations.
The `flake.nix` file describes:
- How to build the project, or components of a projet, via `nix build`
- What "checks" can run for a project, which can be executed via
`nix flake check`
- What binaries can be run, e.g. `nix run .#rad` to execute the CLI
This approach is preferable since a lot of the Nix ecosystem is using
Flakes -- despite it still being marked as experimental.
One interesting benefit, that needs confirming, is that a check can be
run from a remote Git source. So if someone creates a patch on
Radicle, someone else with the `nix` binary can execute:
nix flake check git+https://app.radicle.xyz/nodes/seed.radicle.xyz/rad:z3gqcJUoA1n9HaHKufZs5FCSGazv5?refs=patches/<patch id>
This means that anyone with `nix` can check patches without having to
fetch the patch itself.
Another obvious benefit will be that it allows the Radicle binaries to
enter the Nix ecosystem.
The big caveat to both of the above is that, currently, the tests are
failing while running `nix flake check`. They are currently disabled
with `doCheck = false`. They are potentially failing due to some
strange sandboxing issues which are being looked into.
[0]: https://nixos.wiki/wiki/Flakes
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
Since forking creates your `rad/sigrefs`, which then needs to be backed
up forever, we avoid forking on `rad clone`. This should make little or
no difference for users, as the "fork" will be created on `git push`.
Update to git2-0.18.1, which also requires updating radicle-git-ext
and radicle-surf.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
If a delegate is missing the default branch for a project certain
computations will fail, e.g. getting the canonical mainline branch.
One prevention for corrupting data is refusing to fetch a delegate
that does not have the default branch, when the repository is a
project. It is assumed that the repository is not a project if calling
`Doc::project` returns an error, since the error variants are not
found and parsing errors.
This is only checked in the case of a delegate since non-delegates are
safe to created COBs that don't require the default branch,
e.g. creating an issue.
Instead of having a routing database, and an addresses database,
consolidate that in a single database called `node.db`.
We also simplify database access, by providing a single type for opening
the database, and use traits to access the various functions.
We create a new `seed::Store` that has seed-related methods split out of
the `address::Store`.
Note that tracking is still in its own database currently, as it isn't
ephemeral state. This makes it easy to delete the new `node.db` without
risk of losing any important information.
To avoid renaming all database methods, we create a `Stores` wrapper
that allows us to access routing, addresses, seeds etc. without type
ambiguity.
This is a breaking change and requires `addresses.db` and `routing.db`
to be deleted.
Currently, if refs are updated while the node is stopped, these refs are
not immediately announced when the node starts, due to the node being
unaware of the updates.
To remedy this, on startup, we compare the refs we have in storage with
the refs we announced to the network. Since only our own refs could have
changed while the node was stopped, this change only applies to owned
refs. If we detect that storage has changed while the node was stopped,
we create ref announcements for the updated repos and store them in our
gossip database. These announcements will be included with other
historical gossip announcements when nodes connect to us.
To keep track of what was announced, we store our local node's sync
status in the database. This is also useful when using the `rad sync status`
command, as it tells us what oid our local node is at, and whether it
is aware of our local changes yet.
In addition to this, we make a small fix: previously, inventory and ref
announcements originating from our local node were not stored in the
announcement database. We ensure that they are via the new `announce`
method.
This command shows the sync status of a repository by consulting the
database via the node.
To make it work, we call into the address store to update the
`repo-sync-status` table when receiving ref announcements that are
relevant to us.
An informational message for the protocol is useful for sending
messages to a node that is not required to be relayed.
This message is introduced by starting with an ACK message for a set
of references being synced. This fixes an issue for announcing to a
node that is already synced, since it can now ACK that it received the
message and replies that it is synced.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
It can be useful to return the `RefsAt` that was announced for this
node, for example, to compare it to node events.
Change the `Handle` method to return `RefsAt` so that when it is
called it returns the local node's `RefsAt`.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
The bounds on `collect_from` were overly restrictive. Allow any
`IntoIterator` be used as an argument, while taking ownership, rather
mutably borrowed.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
The previous system allowed the same RID to be fetched from multiple
nodes concurrently, which caused issues, especially when cloning.
Worker code assumes that there is only one fetch per repository going on
at any given time. This change ensures that.
We keep a global fetch queue, instead of having a queue per session. We
also enqueue fetches if a session is at capacity. Finally, we allow
multiple fetch requests to receive results for the same fetch.
Previously, it was possible for the `wire::serialize` function to panic
due to large ref announcements. Now that ref announcements are properly
bounded in size, this shouldn't happen anymore.
It's possible that a temporary directory can be created on a different
mount, and when moved during a clone, creates an I/O error.
To prevent this, add a method to `Storage` which creates a temporary
directory and `Repository` with the path in the form
`<rid>.lock`.
This `.lock` path name can easily be skipped when listing
repositories. Further tooling can be added for detecting leftover
`.lock` directories if, for some reason, they were not cleaned up by
the `tempfile` crate.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
Previously, we would process any valid inventory or refs announcement.
This could cause problems for keeping sync statuses, as we could get
a refs announcement from a node that isn't in our database.
In practice, there is no good reason why we should receive a refs or
inventory announcement before the announcer's node announcement, unless
nodes are not following the protocol.
Implementing this change meant that many of the tests needed updating,
as it wasn't assumed that a node announcement was needed before
receiving inventories etc. from a peer.
We also had to implement a basic form of node address banning to make
the tests pass after this change.
* Set peer's `last_active` even if they are being throttled
* Make sure not to break out of loop in wire module
* Add some more logging
* Use different variable for session filtering to make it clearer
We improve the API so that error messages can be sent back and are
always expected as potential responses to commands. This means that
`CommandResult` is always the type that is sent for commands.
This includes some breaking changes, for example the default "success"
message is simply `{}` instead of `{"status":"ok"}`.
It is desirable to keep the Git repositories in Radicle efficient by
ensuring objects are packed and unreachable objects are removed.
Add a process call to `git gc` to achieve this. Two flags are used for
garbage collection: `--auto` and `--prune`. Their details are outlined below:
--auto
With this option, git gc checks whether any housekeeping is
required; if not, it exits without performing any work.
--prune=<date>
Prune loose objects older than date (default is 2 weeks
ago, overridable by the config variable
gc.pruneExpire). --prune=now prunes loose objects
regardless of their age and increases the risk of
corruption if another process is writing to the repository
concurrently; see "NOTES" below. --prune is on by default.
For `--prune` the default used is `1.hours.ago`. This was chosen over
`now` since that may be too aggressive after just fetching new data.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
Ref announcements contain the namespace (NID) and the SHA of the new
`rad/sigrefs`. This information can be used by the fetch protocol to
fetch only those `Oid`s for those namespaces.
Add a new stage `SigrefsAnnouncement` that captures this behaviour by
only asking for the advertised `Oid`s and updating the corresponding
`rad/sigrefs` for each namespace. Note that this is a special case of
the `SpecialRefs` stage and so the protocol chooses between these two
by checking if there was announcement passed through. The following
stage, `DataRefs`, stays the same.
Note that this stage can only be activated via a `pull`, and `clone`
stays the same.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
The RefsAnnouncement previous announced its complete set of
`rad/sigrefs`. This was useful in that verification can occur on the
receiveing side immediately by verifying the refs and proceeding with
the fetch -- note that it did nothing with the ref information it
received.
This can be minimised to only advertising the namespace, i.e. the
NID, and the new SHA of the `rad/sigrefs`. This is represented by a
new message type, `RefsAt`, which contains the NID and the Oid.
For checking if an announcement is stale or fresh, the receiving
side can check the existence of the SHA in its Git repository.
The tests are appropriately modified to use NIDs and Oids.
To prepare for a follow-up change, the announcement data is also
threaded through for the fetch protocol to focus fetching only the
advertised set of Oids.
N.B.: because the refs aren't communicated, the verification happens
when fetching since the data needs to be fetched before
verifying. This is a trade-off with minimising the message payload for
verifying earlier.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
While there is a method, `unbound`, for converting a `BoundedVec` to
its inner `Vec`, it is common practice to implement the `From` trait.
Add the `From` conversion for `BoundedVec` and `Vec`.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
When performing a clone there are two potential issues.
The existing issue is that when a repository is created to clone into,
it does not contain any data initially. This leaves other node
operations thinking that a repository does exist but when they try to
perform operations on it, those will fail. These will be surfaced as
warnings but the node will continue to operate.
The other issue, while writing fetch v2, is that if a clone fails,
then cleanup needs to be performed -- to remove the empty repository
from storage.
To tackle both these issues, a temporary directory is used in the
place of the repository. If a clone fails then the temporary directory
is dropped and nothing needs to be done. If the clone succeeds then it
can be safely moved to the storage.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
The fetch V2 improvements introduced a `UserInfo` type that is used to
set the Git config's name and email when cloning a repository. This
`UserInfo` can also be used when creating a new `Repository`.
The `Storage` type now takes a `UserInfo` so that it can be passed to
`Repository::create`, for creating a new repository.
For testing, this requires adding a new `fixtures::user` so that
testing `Storage` instances can be easily constructed.
It also means that the `radicle-cli` tests all changed with new SHAs
since the name and email are used as part of the hashing input.
N.b. the order of the timelines while testing the identity cob
changed. It's not clear why that is.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
Previously, if a delegate's namespace was fetched and its rad/sigrefs
was behind the current state, then it would fail to perform the fetch.
Instead, when the ancestry path is checked the outcome can be used
during the update:
1. If the tip is equal, then the update can be marked as skipped.
2. If the tip is behind, then the update can be rejected, iff the
policy does not allow it.
3. If the tip is ahead, then the update is a fast-forward.
4. If the tip is diverged, then the update results in a failure, iff
the policy does not allow it.
The protocol also prunes any delegates's updates when the `rad/sigrefs`
is behind and returns an error if they are diverged.
For any non-delegates, it will simply prune them.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
Previously, we would always ask nodes for messages from one hour ago.
This meant that we could miss messages if our node was offline for more
than an hour. This change uses a variable backlog by checking the
timestamp of the last gossip message we received.
Instead of storing received gossip messages in an in-memory `BTreeMap`,
we persist them to the database (`addresses.db`). There are two reasons
for this:
1. Node restarts do not wipe the state, meaning that it is much less
likely that some gossip message will get lost before reaching all
nodes.
2. This will allow us in the next commit to know how far behind we are
after a restart, to be able to request the correct time range of
gossip messages that we missed while offline.
Note that we now share the addresses.db file between two connections and
stores, which we don't do anywhere else. Having multiple connections to
the same database is fine, but the way we're having to do it is a bit
awkward. In a future patch, we could pull out the database schema from the
`address` module and rename the database to make things clearer.
The git-daemon code is no longer needed since git-upload-pack is used
in its place.
Remove the git-daemon code from the runtime and worker.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
Wire up the `radicle-fetch` crate to the fetch logic in the
`radicle-node` worker.
Any required traits for storage, identity, tracking, and transport
used by `radicle-fetch`.
The worker adds a new module to run the `git upload-pack` process and
pipes the `stdout`/`stdin` to the respective channels.
The `Worker` type required adding a `FetchConfig` to easily allow the
configuration of fetches and passing the signer and tracking store
location through to the fetch `Handle`.
The channels code is adapted to have a new writer that always flushes
when a call to `write_all` is made. This is necessary since the
Gitoxide code never calls `flush`, and so it will hang since no data
is sent until `flush` is called.
Since `UserInfo` is used to set up a newly cloned repository, this
changed some of the SHA1 hashes output in the CLI test examples. It
was also necessary to update the CLI's clone code to be deterministic
in its output for tests.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
Make the validation logic customisable by extracting it out into a
separate trait that can be defined by other crates.
The Repository validation logic stays the same.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
Instead of returning the unsigned references, the validation of a
repository's remotes returns the set of validation errors. The caller
is then expected to handle these validation errors appropriately, for
example, by logging them and re-raising an error.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
The `service::tracking` module has nothing specific to the
radicle-node code. This means that it can be moved to radicle under
the `node::tracking` module.
This makes it more resuable in other contexts, for example, for
fetching logic.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
This change allows users to track/untrack peers and repos without a
running node. This fixes a bug when running `rad init` without the node
running, where the repo would be tracked with a different scope than
what was specified on the CLI.
Introduces a new COB type to store repository identity documents.
The reason for this change is to:
1. Simplify the code, as the identity document logic resembled a COB,
yet it had custom logic. This allows existing COB code to be used for
identities.
2. Make identity document update logic more flexible, since COB actions
can be added in the future.
3. Re-purpose existing tools around COBs to work on identities, eg. `rad
cob`.
4. Unify the concept of an identity change proposal, with regular identity
changes. This means we can remove the `id.proposal` COB in favor of
using the `id` COB itself.
Notes
-----
* Each repository has one Identity COB.
* The `Proposal` COB has been repurposed into the `Identity` COB.
* Identity documents are stored as *embeds* inside the Identity COB
actions.
* The action that contains new document versions is called `revision`,
just like the Patches COB.
* The namespaced `rad/id` ref is a symbolic reference to that Identity
COB.
* The canonical `rad/id` ref is a direct reference to the commit in the
Identity COB that contains the latest *accepted* revision of the
document.
* All commands for managing identities have been folded into `rad id`.
Hence `rad delegate` and `rad edit` are removed.
* The concept of "rebasing" an identity document is gone.
* The `rad id` output has been updated to match the style of other
commands.
* When a revision has enough signatures, it is automatically adopted as
the current identity, there is no longer the need for a "commit"
action.
* When an identity revision is proposed, and the current identity has a
threshold of `1`, that identity is automatically accepted due to the
above point.
* The idea of "verifying a peer's identity branch" no longer applies, as
COBs cannot be verified one history at a time.
* Since the root commit of the Identity COB does not have a "Resource"
to point to (it would have to point to itself, which is impossible),
we've made the resource id optional for COBs.
We were being too strict by requiring that no routes existed that
weren't passed into the function. This isn't really useful for testing,
since we're more interested in checking that certain routes are there,
rather than testing that *only* those routes are there.
When a fetch is user-requested, a timeout can be supplied that is passed
down to the worker.
When a fetch is service-requested, a default timeout is used.
Private repos are implemented by extending the identity document with a
`visibility` attribute, that can either be `"public"` (default) or
`"private"`.
In case of `private` visibility, only the delegates are allowed to view
the repo, as well as any DIDs added to the allow list.
To implement repo visibility, we simply block fetches from and
announcements to peers for whom the repo should remain invisible.
Private repos are also not announced in the `inventory` message, since
the full list of peers that *may* have the repo is retrievable from the
repo identity. This could cause errors if eg. a peer who is allowed to
view the repo doesn't actually have it. However this is an ok trade-off
for now to keep the complexity low. For repos to truly be private, it's
important that the RIDs don't leak either.
Finally, we modify `radicle-httpd` for now to only list public repos.
Eventually, we would want to change this depending on whether an allowed
peer is authenticated with the service or not.
---
It's also worth mentioning why this approach was taken, vs. end-to-end
encryption. The reasons are as follows:
1. Nodes that do not have access to a private repo will generally not want to
replicate encrypted data that they cannot examine or use.
2. The chosen solution is trivial, while encrypting git objects isn't.
3. Performance of the chosen solution is much better, there is no
overhead.
4. Privacy of the chosen solution is better: RIDs are never leaked, and
neither is the existence of a private repo, nor who has access to it.
There is one downside: Paying for storage of private repos is no better
in terms of privacy than what GitHub offers. Hosting providers will have
access to your private repos, if this solution is used.
Add `deny.toml` with a few exceptions, and update certain dependencies
to get rid of duplicates.
Running `cargo deny check` will yield some warnings still, but no
errors.
This change allows for files (blobs) to be embedded into COB entries.
This allows for things like image attachments in issue comments for example.
For now, we only enable this in the `issue` COB.
The way it works is that relevant COB actions carry metadata about which
files are attached to them; and we store those files as blobs inside the
COB entry's *tree* object, under an `embeds/` folder.
The `Embed<T>` type is used for the above, and either carries actual
content, or carries a content-id. Retrieving the actual content is as
simple as asking the repository for a blob with that content-id.
As a possible future extension, MIME types could be stored alongside the
files in a "metadata" file. This could help clients display the content
appropriately.
This is a breaking protocol change that prefixes all gossip messages
with their length.
This is to allow message extensions, as we'll be able to read the whole
message inclulding the extensions, without having to decode the
extensions.
It also makes it easier to check the message length before we start
decoding, so that we can reject messages that are too big.
When an option is available as part of the node configuration
(`config.json`), we remove it from the CLI options of `radicle-node`.
Instead, we allow a different config file to be specified via the
`--config` option.
This adds functionality so that on `rad sync` and `rad clone`, we
connect to known seeds if necessary, before attempting to fetch.
In passing, some related changes were made:
* The `rad sync` command's arguments were reworked:
`--replicas <count>` can be used to specify a replica count when
fetching
* The `rad patch` command no longer has a `--fetch` option
* It's now possible to tell the node not to automatically connect to
peers in the background
* `AddressBook` is being used underneath `Seeds` to allow for shuffled
iteration
Only return once the connection is established or failed.
It's still not perfect, as several failure scenarios are not handled,
but it's an improvement.
We move the session address to the top-level struct, since it's needed
in a bunch of places. This change is required for the rate-limiting
code that is coming next.
Improve the logic when choosing addresses to connect to. Previously we
would wait much too long, even for good addresses. This was especially a
problem when the reason for disconnection was simply a node restart.
As a starting point, we populate the address book (if empty) with some
bootstrap nodes which are trusted.
To prevent these nodes being connected to during tests (including e2e
tests), we create a new constructor for `Config` that is used in tests.
Note that a `cfg(test)` check is not enough given that e2e tests don't
have that set.
Instead of using a command name plus arguments, we use a new `Command`
enum that is similar to the one in `radicle_node::service`. This way,
we're able to have more complex commands and parsing is simplified.
It turns out that the CRDT formed by the union of Git DAGs
is enough to guarantee everything we need for COBs.
This changes the following things:
* COB operations no longer need to be commutative
* COB histories are traversed in the same deterministic order on all
replicas
* It's now possible to implement RSMs on top of COBs, eg. scripting
* Lamport clocks have been removed
* `radicle-crdt` is no longer a dependency of `radicle`
* COBs are no longer instances of `Semilattice`
* The `Ops` type was removed in favor of having `Op` contain multiple
actions
This commit addresses a workaround previously implemented to
include the SQLite library bundled inside the binary.
The upstream have successfully merged the patch that implements this
solution [1], allowing us to safely incorporate it
into our codebase.
[1] Link to the merged patch: https://github.com/stainless-steel/sqlite/pull/71/files
Fixes: 431a389944
Signed-off-by: Vincenzo Palazzo <vincenzopalazzodev@gmail.com>
Help users confirm all their radicle binary versions by providing the
'--version' flag.
Move the print_version function to the radicle crate for use by the node
and remote-rad binaries, and remove tests to confirm the output format
as it is deemed redundant.
Co-Developed-by: Vincenzo Palazzo <vincenzopalazzodev@gmail.com>
Instead of asking for the passphrase in `radicle-node`, we expect the
`RAD_PASSPHRASE` env var to be set.
This keeps `radicle-node` simpler and lower level, while the passphrase
prompt is still available via `rad node start`.
This new trait is used when only signing refs is needed, and not general
write access to a repository.
We also implement `ReadRepository` and `SignRepository` for
`DraftStore`.
The tracking database can be opened in read-only or read-write
mode. Knowing which is being used is significant in concurrent
scenarios, e.g. it's safer to have a read-only handle on multiple
threads.
Provide static type information by signifying what kind of database
handle is being worked with, read or read-write.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
This is a fairly substantial change, which adds a new configuration file
to the user's `$RAD_HOME` that includes node configuration, including
the alias, and also makes node aliases required.
When running `rad auth`, the user is now prompted for an alias, which
defaults to `$USER`.
When running `rad self`, the alias is now shown.
If the user runs radicle without a config file, the defaults are loaded,
and `$USER` is used as the alias.
* Add `--home` flag
* Organize the information better
Now looks like this:
DID did🔑z6MknSLrJoTcukLrE435hVNQT4JUhbvWLX4kUzqkEStBU8Vi
└╴Node ID (NID) z6MknSLrJoTcukLrE435hVNQT4JUhbvWLX4kUzqkEStBU8Vi
SSH running (823)
├╴Key (hash) SHA256:UIedaL6Cxm6OUErh9GQUzzglSk7VpQlVTI1TAFB/HWA
└╴Key (full) ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIHahWSBEpuT1ESZbynOmBNkLBSnR32Ar4woZqSV2YNH1
Home [..]
├╴Storage [..]/storage
├╴Keys [..]/keys
└╴Node [..]/node
For the purposes of debugging running nodes, it's useful to know which
threads are doing what. This allows tools like `htop` to reveal which
threads are running.
We consolidate the setup done by `rad checkout`, `rad clone` and `rad
remote add`, so that they all check for node aliases, and are all
capable of setting up tracking branches etc.
* Return the correct alias from a node announcement, stripping '\0'
bytes.
* Accept `AliasStore` types as references and as owned.
* Store a `null` in the database if the alias received is empty.
Most of the test flakiness was caused by the git-daemon not being able
to bind to a port. It seems like using an RNG was not a good enough
solution.
After this change, tests seem to be passing without issue.
Our build was faking the inclusion of SQLite and dynamically
linking it in the 'radicle' crate and other components.
This patch solves the following error and requires reinstallation
of the `radicle-cli`, `radicle-node`, and, if used, the `radicle-httpd`
```
✗ Sync failed: internal error: malformed database schema (node-policies) - near "strict": syntax error (code 11): malformed database schema (node-policies) - near "strict": syntax error (code 11)
```
Tested-by: Vincenzo Palazzo <vincenzopalazzodev@gmail.com>
Signed-off-by: Vincenzo Palazzo <vincenzopalazzodev@gmail.com>
Previously, we only sent the announcement when there were external
addresses. But it contains other information like the node alias,
therefore we always send it.
Since the proof-of-work can be expensive to compute everytime, we add
the ability to load the local node announcement from the file system.
We also change the proof-of-work parameters so that they are relaxed
when running in debug mode, instead of when running tests.
Finally, we remove the boolean validation of the PoW. Instead, we store
the work per-node, in the database, to be used in the future in case of
DoS attacks or network congestion.
Previously, only one connection to the node control socket could be
handled at a time. This usually worked fine, but if one connection hangs
for whatever reason, no other process can access the node.
It turns out it's possible to create a deadlock when using
`child.wait()` with `Stdio::piped()`, if we don't read the output while
the child process is running. This is because the pipe has an internal
kernel buffer that will eventually fill up, causing the child process to
block on writing to stdout and never exit.
To remedy this situation, we read the stdout in a thread while the child
process is running.
This change updates to the latest radicle-git-ext, which required a
few changes within the radicle family of crates.
One set of changes is that the radicle-git-ext crate subsumes
git-commit, git-trailers, and git-ref-format -- so all those imports
go through radicle-git-ext.
The commit code requires that parent `Oid`s point to commits, which
made some tests fail. One of the reasons for these failures is that
the patch tests used fabricated `Oid`s and so the verification would
fail. This is fixed by allowing the test context code to create
`PatchRequest`s that create valid commits.
The other reason for failure was that the identity `Action` passed an
`Oid` that pointed to a blob. This is fixed by removing that code and
making a note of it in documentation.
The final major change was moving the Refspec type into git-ref-format
and removing the AsRefspecs trait. The trait was not required and
could its usage could be replaced by simpler code.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
When attempting to connect to a peer, and when succeeding, write the timestamp
to the database.
This will allow us to better select addresses and not retry peers to
soon.
On macOS, shutting down a socket returns an error if the socket is
already closed. We don't consider that a problem, as it just returns
`Ok(())` on Linux.
We fix the failing fetch tests by:
1. Not verifying our own refs, unless we're cloning, since we're
otherwise not fetching our own refs.
2. Always force-fetching `sigrefs` from remotes into the staging copy.
3. Making sure that sigref updates are fast-forward before transfering
the remote into the production copy.
Ensure that the production repository is validated after fetching and
performing deletions. This is to highlight any inconsistencies that
may show up between fetching and deleting references.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
Given the scenario of 3 peers, A, B, and C each having a project R. If
peer A has B's fork, but C does not, then when peer A attempts to
fetch from C the fetch will be rejected since Git will fail when a
refspec that is asked for does not exist on the remote side.
To avoid this, the fetch logic is changed so that the client first calls
ls-remote to the remote side, which results in the references that the
remote has which the client is interested in. The follow-up fetch then
uses these refs as the refspecs -- which should succeed unless there
was a race for a deletion on the remote side.
The rest of the verification logic stays the same, so the storage
should still be in a working state before transferring from the staged
repository to the production repository.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
Ensure that non-existent tracking relationships do not affect
fetching.
This is tested by first performing a clone, by `bob` from `alice`,
after `bob` tracks a remote that does not exist. `alice` creates an
issue which is then fetched by `bob` -- to ensure the fetching
scenario is also tested.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
We were allowing remotes to be constructed via a function that didn't
check its inputs.
This is now fixed by moving the public key into the `SignedRefs` struct.
It's currently possible for a channel to be blocked forever while
waiting for data, if the remote peer keeps the connection alive.
This can occupy a worker indefinitely.
With these timeouts, we ensure that workers exit if there is no
activity after a certain amount of time.
This command announces refs to peers and waits for them to be
in sync.
Adds a subscribe method to `Handle` so that we can get the events
from the seed to allow us to confirm that the configured seeds fetched
from us.
By having the keep-alive delta be the same as the ping timeout,
we essentially disconnected peers right after we sent them our
first ping.
This fixes that issue by setting the ping timeout to double the
keep-alive delta.
Instead of returning the namespaces that we fetched with,
we return the remotes that were actually fetched from.
This can differ if some trusted peers were not available
on the remote node.
Previously, if that was the case, it would cause errors
since the remotes were looked up and that lookup failed.
With this change, we not only verify the signed refs before
the repository "transfer", but also validate that all refs in the
repo are signed and there is no discrepancy with the signed refs
file.
If `git-fetch` returns a non-zero code on exit, we keep going in the
case of an existing repository.
The reason is that `git-fetch` will return an error code if any ref
is rejected during the fetch. But this can happen if someone
pushes conflicting `rad/sigrefs` branches for example, and the node
already has a copy of that branch.
cloudhead
Fintan Halpenny
Fintan Halpenny
Michalis Zampetakis
Alexis Sellier
Slack Coder
Vincenzo Palazzo
xphoniex