The construction of the debug object is unwieldy, and error prone
(for example, renamed struct members have to be manually renamed
in the serialization code, see "refs" vs. "refsAt").
Use derived serializers where possible to make this easier to maintain.
`radicle-protocol` depends on `cyphernet`, but only uses the module
`cyphernet::addr`, which is a re-export of `cypheraddr`, which is
unnecessary (as `cypheraddr` alone would do), and invites accidentally
pulling in more dependencies via `cyphernet`.
To prevent accidentally depending on further types from `cyphernet`,
change this to more conservativeley depend on `cypheraddr`.
Using empty `Vec`s to mean "all refs" can be confusing. Avoid the chance
of confusion by introduction of a new type that clearly describes which
refs should be fetched.
It was found that if a fetch was started for a specific `Vec<RefsAt>`
and a following general, i.e. empty, `Vec<RefsAt>` was queued, the
subscriber for the general one would be removed when the specific
fetch completed.
This behaviour is undesirable, since the general fetch could
potentially fetch more data.
To prevent this from happening, the `Vec<RefsAt>` is added to the
`FetchKey`, so that `subscribers` are matched with the set of
references they want to hear results for.
A test was added for the above scenario.
Wire up the new `FetcherService`. This reduces the `Service` fetch
methods down to:
- `fetch`
- `fetched`
- `dequeue_fetches`
- `fetch_refs_at`
This simplifies the code by off-loading the logic to the `fetcher`
family of types, rather than handling the intricacies in the `Service`
itself.
The `Service` now just performs the necessary I/O based on the
returned events from the `fetcher`.
This also removes the fetching state from the node sessions. This
follows the single responsibility, and sessions now only care about
their connectivity.
Breaking changes:
- The `Connected` state of a peer no longer contains fetching
information, which was being returned as part of the JSON payload
result.
- The `rad debug` information for ongoing fetches contained the number
listeners awaiting for results, this was removed.
The `FetcherService` wraps the `FetcherState` alongside keeping track
of subscribers for a given fetch.
Typically, this is a channel for which the fetch result should be sent
to.
These subscribers are coalesced by maintaing a map from `(rid, node)`
pairs to the subscriber.
This continues the sans-IO approach where the type of subscribers are
not specified, but rather are kept as a type parameter – to be
specified by the caller.
This patch introduces the `FetcherState` which encapsulates the logic
and state for keeping track of fetches.
It uses a sans-IO approach, where the state of fetches transition
based on the current state and provided input.
Callers can then decide to perform I/O based on the returned event.
Be more defensive by preventing a node, that is not configured for
Tor, to not connect to `.onion` addresses via the `Service::connect`
method.
This means that an `Outbox::connect` should not be produced, and the
address table should not insert an entry.
When maintaining persistent connections, the config was consulted for
a peer address. Multiple addresses for a single `NodeId` can be
listed. This can result in a different address being used for
reconnecting if the `HashSet` returns in a different order.
The original address that was used for connection is in the `Session`,
so this should be used instead.
Evaluate the `error!` logging in `radicle-protocol`.
The majority of cases, the `error!` is downgraded to `warn!`. These
cases are generally useful to know if there's an issue, but the
operator cannon necessarily do anything about it.
In a few cases, `debug!` was chosen. These are generally when an error
means that a result is not part of an accumulation, or the system will
eventually correct itself during another event or restart.
If the previous message started with "Error", then the wording is
changed to say "Failed to".
Evaluate the `warn!` logging in `radicle-protocol`.
If the event is a result of something being suspicious in the running
of the protocol, then it remains a `warn!`.
However, if the system ends up ignoring the event and can continue
without issues, the log level is reduced to `debug!`.
Introduce the `radicle-core` crate for housing data types that are at
the core of the Radicle protocol. The initial data type being added is
`RepoId`. To make the crate as lightweight as possible many of
dependencies are optional and gated by feature flags.
The crate is marked as `no_std` even though it unconditionally depends
on `radicle-crypto`, which is *not* `no_std`. This is to invite
refactoring of `radicle-crypto` to `no_std` at a later time.
The `localtime` crate was defined by cloudhead, and is a minimal
repository with a single `lib.rs`.
Instead of using it as an external dependency, copy the code directly
into a new workspace crate `radicle-localtime`.
The default `serde` implementation goes through the `LocalTime`'s
seconds values rather than milliseconds, since this is the more common
format. This allows the removal of the `serde_ext` functions.
The one place milliseconds was used was for the
`radicle::cob::common::Timestamp` type, where the `Serialize` and
`Deserialize` implementations are manually written.
It also adds a `schemars` feature to remove `schemars_ext` functions
in `radicle` as well.
Implement `IntoIterator` for the `BoundedVec`.
This allows the use of `into_iter` for returning the owned data,
rather than references via the `Deref` implementation.
There is an unfortunate lack of single-responsibility with handling the
addresses to connect to.
This is a band-aid fix until the point in time where this can better improved.
Previously, the `RepoId` was backed by the `git2::Oid` type, which has a `Hash`
implementation that uses `std:#️⃣:Hasher::hash`. This uses a length prefixed
form for hashing.
When switching to `radicle_oid::Oid` and implementing `Hash` without the length
prefix, this subtly changed the hashes going forward.
These hashes are used in `BloomFilter`, which is the type underlying `Filter`.
This resulted in cases where a previous `Filter` that certainly contained
`rad:z3gqcJUoA1n9HaHKufZs5FCSGazv5` were suddenly reporting that they did not
contain this repository.
Changing the implementation of `Hash` for `radicle_oid::Oid` fixes this, and is
confirmed by adding the regression test in `filter.rs`.
Make `git2` an *optional* dependency of `radicle-cob`, and refactor
`radicle` to depend on crates that in turn do not depend on `git2`
*non-optionally*
The main offending dependency of `radicle-cob` is `radicle-git-ext`
from the `radicle-git` workspace in repository
(rad:z6cFWeWpnZNHh9rUW8phgA3b5yGt) which *non-optionally* depends on
`git2`.
So, to achieve removal of this dependency:
1. The crate is refactored to depend on the new crates
`radicle-git-ref-format` `radicle-git-metadata`, and
`radicle-oid` introduced in the previous commits, instead of
`radicle-git-ext`.
2. Some code from the `radicle-git-ext` crate in the `radicle-git`
workspace in repository (rad:z6cFWeWpnZNHh9rUW8phgA3b5yGt) is
copied. See `crates/radicle-cob/src/backend/git/commit.rs`.
This cascades to `radicle` and its dependents.
Firstly, the there is an
`impl Deref<Target=git2::Oid> for radicle_git_ext::Oid`. This made
it very convenient to just deref to obtain the wrapped `git2::Oid`,
so there are many expressions of the shape `*oid` in `radicle` and
its dependents. However `radicle-oid` does not provide
`impl Deref<Target=git2::Oid> for radicle_oid::Oid`, as notably,
`Target` is an associated type, and not a type parameter, so an
implementation of `Deref` would tie the new `Oid` too tightly to a
particular implementation.
Instead, work with `impl From<radicle_oid::Oid> for git2::Oid`
(which can be enabled using the feature flag `radicle-oid/git2`).
This explains the changes from `*oid` to `oid.into()` at every
transition to "`git2` land".
Secondly, `radicle` and its dependents are refactored to also depend
on `radicle-git-ref-format` and (much less prominently)
`radicle-git-metadata` instead of `radicle-git-ext`
This is to avoid pulling in `git2` via these dependencies.
Thirdly, as the re-exports in `crates/radicle/src/git.rs` change,
they are at the same time also cleaned up. Notably, the types from
`radicle-git-ref-format` are re-exported under `fmt` only, not "twice".
While overall this obviously is very much a breaking change, these
changes should mostly amount to changing from `Deref` to `Into`, i.e.,
`*oid` → `oid.into()` and rewriting imports for `radicle::fmt`.
This is indicated by the mostly mechanical nature of the changes to
`crates/radicle-{cli,node,remote-helper}`.
The "seeds" command does only take a repository ID, and
carries the implicit assumption that the namespace with the same public
key as the Node ID of the receiving `radicle-node` process should be
used as reference for determining whether other nodes are in sync,
but no other namespace.
In the spirit of separating user and node identity, relax this, so that
the command also carries the public keys relative to which the sync
status should be computed. This allows to inspect sync status for
arbitrary namespaces via command.
For now, this feature is not exposed to the CLI, but `rad sync status`
always passes the public key of the active profile.
The "announce references" command does only take a repository ID, and
carries the implicit assumption that the namespace with the same public
key as the Node ID of the receiving `radicle-node` process should be
announced, but no other namespace.
In the spirit of separating user and node identity, relax this, so that
the command also carries the public keys for which the announcement
should be made. This allows to announce arbitrary namespaces out of
storage via command.
For now, this feature is not exposed to the CLI, but rather:
- `rad sync` always announce for the public key of the active profile.
- `git-remote-rad` ditto, as it calls the same codepath.
In many tests we assert that encoding and decoding gives the same
object. Let's call that a "roundrip", and also provide a proc macro to
generate such tests.
Whenever the node fetches new updates, it checks if canonical references can be
updated. The node has learned how to return these results and emit them as node
events. This is a breaking change since it adds a new variant the `Event` type,
which is not forwards-compatible.
I was greeted by `rad patch redact` with
called `Result::unwrap()` on an `Err` value:
Error { code: None, message: Some("failed to convert") }
stack backtrace:
[…]
3: core::result::Result<T,E>::unwrap
at …/rust-1.88.0/lib/rustlib/src/rust/library/core/src/result.rs:1137:23
4: sqlite::cursor::Row::read
at …/index.crates.io-1949cf8c6b5b557f/sqlite-0.32.0/src/cursor.rs:136:9
5: radicle::cob::patch::cache::query::find_by_revision
at ./crates/radicle/src/cob/patch/cache.rs:624:65
6: <… as radicle::cob::patch::cache::Patches>::find_by_revision
at ./crates/radicle/src/cob/patch/cache.rs:553:9
7: radicle_cli::commands::rad_patch::redact::run
at ./crates/radicle-cli/src/commands/patch/redact.rs:23:9
8: radicle_cli::commands::rad_patch::run
at ./crates/radicle-cli/src/commands/patch.rs:1026:13
[…]
It turns out that `sqlite::cursor::Row::read` is the panicky version of
`sqlite::cursor::Row::try_read` (which returns a `Result`).
While it is somewhat rare that SQLite reads fail, it is not unheard of.
In `radicle-cli` it might not be critical, but also `radicle-protocol`
and `radicle-fetch` are affected, and they could potentially panic a
`radicle-node` process.
Use `try_read` instead, and propagate down the error handling.
Remove `fn deserialize` and instead provide a trait `fn decode_exact`
for testing.
Refactor errors, so that we can now cleanly distinguish between
`Error::UnexpectedEnd` which means that decoding might work given
more data, and `Error::Invalid`, which means that no amount of
additional data will decode.
While at it, cleanup the variants of `Error::Invalid`, and provide `ControlType`.
What made the issue fixed in the parent commit harder to spot
for me was the multitude of concepts and terms involved in
encoding: What's *serializing* vs. *encoding*? Why is there
this odd `Frame::to_bytes`?
So I decided to clean up a bit. I removed `fn serialize` so that the
logic stays close to the data, and instead of it provide
`fn encode_vec` in `trait Encode` so that we don't add up with
one-offs like `Frame::to_bytes` again.
Also we make use of `encode_vec` in the tests.
In 3c5668edd2 I mistakenly moved the check
the length of an encoded `wire::Message` from its `impl wire::Encode` to
the `fn serialize`, so that it would not apply only to messages (as
originally intended), but to *any* argument of `serialize`.
This is not a problem for gossip messages, but catastrophic for Git
streams, which tend to send a lot of data.
The fix is simple: Move the check back into `impl Encode for Message`.
The upgrade of the `radicle-crypto` crate was missed when upgrading
semver versions.
Unfortunately, since it is at the base of the dependency tree, all
crates that depend on it require an additional upgrade.
The logic for computing canonical references conflated the semantics of
annotated and lighweight tags, yielding confusing/wrong results. The
main culprit was the call to `peel_to_commit` in
`impl ReadRepository for Repository`, peeling tag objects away.
To resolve this, we separate out quroum computation per object type: one
implementation for commits, and one for tags.
Also move move canonical error types and methods into their own
module to have a cleaner file structure for the main logic.
Capture the `BTreeMap<Oid, u8>` type into a `Votes` struct, with its own API.
We were getting the objects from the repository twice – once in `Canonical::new`
and once again in `ensure_commit_or_tag` – so the latter was removed.
Use an `enum CanonicalObject` to specify which object types the canonical
process supports.
During `converges`, separate commits and tags, and ensure that we are only
looking at objects of one type. Note, that Fintan think this is actually the
*wrong* place to do it because we already filter the objects – so we should
keep track of that information higher up the callstack.
Co-authored-by: Fintan Halpenny <fintan.halpenny@gmail.com>
Pay more attention to when a Node ID is displayed in compact form. As
this was sometimes used to save space, e.g. in `rad node status` in
combination with long node addresses (like .onion names), also implement
a compact format for node addresses and use that instead in some cases.
Progress notifications in spinners keep the short form.
The `[T]` and `[I]` are interpretted as links when running `cargo doc`. This
table is surrounded in a code fence to prevent this, but also provide a better
view of the table when rendered to HTML.
Mostly changes of `use`s and moving components to `radicle-protocol`,
with the goal of just getting `radicle-node` to work on top of the new
`radicle-protocol` crate.
Lorenz Leutgeb
Fintan Halpenny
Defelo
Yaroslav Halchenko
Lorenz Leutgeb
Sebastian Martinez
