After feature levels are introduced to Signed References, and downgrade
attack protections are implemented, new errors are encountered when
reading Signed References that are in a downgraded state.
Such downgrade errors are recoverable by advancing history in a way that
restores the feature level that the history requires.
However, the implementation in `radicle-fetch` must be adjusted to not
abort fetching on downgraded histories, but rather inspect the error in
this case and decide whether continuing the fetching process would
actually help recover from the downgraded state.
The two cases are treated slightly differently for non-delegates and
delegates. Since delegates are crucial for evaluating repository
identity, also the fact that they cannot recover from a downgraded state
is logged at the "info" level.
Add a configuration knob `node.fetch.signedReferences.featureLevel.minimum` to
allow node operators to enforce more strict verification when fetching
Signed References.
In order to allow automatic migration to the feature level 'parent',
introduce a way to force writing signed references, even if the
user-controlled refs are unchanged.
To detect the special case of a Signed References history with a single
and root commit (which would otherwise be detected as feature level
'parent', even though the `refs/rad/sigrefs-parent` ref cannot be
written), keep information about the parent commit in `SignedRefs`.
The output of `rad inspect --sigrefs` is changed to match the more
strict interpretation of the feature level for histories with a single
and root commit.
To prevent from inability to write sigrefs in case of a verification
error, swallow verification errors about the head commit, log, and
carry on writing.
In order to respond to the caller with the correct feature level in
case sigrefs are to be saved but unchanged, the write side needs to
care about feature levels. This is done by sharing more code with the
read side.
To be able to reason about backwards compatibility, downgrade attacks,
and future configurable enforcement of minimum required (security)
features, introduce the concept of a "feature level" in Signed
References.
In this implementation feature levels are assumed to be monotonic, i.e.,
that higher feature levels behave like lower ones, and only add
semantics on top.
Implementing this as an `enum` that is `Ord` flows nicely.
Treatment of the write-side is left for the next commit. Currently,
the write side "promises" `FeatureLevel::Parent`, but is not able
to "keep the promise", since it does not actually inspect the
feature level of the head in case it is unchanged.
CHANGELOG
The scenario A₁ ← B₁ ← A₂ ← B₂ would previously have resulted in A₁
being loaded. This is undesriable, since the latest non-replayed commit
is B₁.
The information regarding the order of duplicates, which is preserved by
`seen`, can be used to recover the first commit.
Due to a bug introduced in `df8e4e6c88a8bfb6c1ec6b07dcda64093b477cbe`, IPv6
addresses in the configuration file might have ended up in the database
tagged with `type = "dns"`, which is incorrect. The bug was fixed in
`a2e72b48e79d090d33f6c13c485239947de0522e`.
However, Radicle 1.7.0 was released in between those two versions, so a
this migration to repair the entries is added.
Tests are added in `test::migration_8` module. This requires some test
setup, mainly a `Database::memory_up_to_migration` constructor that is only
available to test code.
More sigrefs on the network than previously thought actually are missing
the reference `refs/rad/root`. Revert the change to make this an error,
which was part of commit `d3bc868e84c334f113806df1737f52cc57c5453d`.
IPv6 addresses are already persisted in `config.json` files.
The parsing of these old addresses, e.g. `FE80::0202:B3FF:FE1E:8329:5976` should remain valid.
Emit a warning when they are found.
The output of the address will still enclose the host in `[]`.
Previously, trying to load `SignedRefs` for any given remote would
result in a fetch failure.
Teach the fetch to be more resilient by pruning remotes that result in
an error when loading `SignedRefs`, and add the error to the
validation failures.
Split up signed references into its read and write components.
On the write side:
- Preserve the old behavior of writing references to the blob `/refs`
and sign over the blob.
- Ensure `refs/rad/root` is contained in the `/refs` blob.
- Ensure `refs/rad/sigrefs` is *not* contained in the `/refs` blob.
- Introduce a new (internal) reference `refs/rad/sigrefs-parent`
so that no two `/refs` blob are equal, even if they contain
the same set of (non-internal) refs.
On the read side:
- Preserve the verification of the signature in `/signature` and
the reference `refs/rad/root` (if present).
- Fail verification of `refs/rad/root` is not present.
- Protect against replay attacks by walking the history of the
head of `refs/rad/sigrefs`, skipping interpretation of `/refs`
blobs in case they are identical to a previous `/refs` blob.
This is achieved by searching for repeated contents of the
`/signature` blob.
The reference `refs/rad/sigrefs-parent` is never read from or written to
the Git repository in storage.
The pre-existing implementation of signed references did not include
a nonce, thus duplicate but legitimate sets of references could not
be distinguished from maliciously replayed sets of references.
The new implementation uses `radicle-git-metadata` which is moved from
`dev-dependencies` to `dependencies`.
The format for IPv6 addresses was changed in
df8e4e6c88 to require '[' and ']'.
IPv6 addresses that were stored in the database in the past must be
migrated.
Most consumers of `Device` are interested in the public key of the
device, and `Keypair` is the trait from `signature` which captures
this, so implement it.
Further, for boxing a signer, introduce a new trait `BoxableSigner`
(to remain dyn-compatible) which additionally requires `Keypair`.
Ensure that the public interface of signed references does not leak its
implementation details.
This allows the evolution of the interface in a safer manner, and does
not leak implementation details to the rest of the crate or any
dependents.
Avoid leaking the types of the signed reference types by moving their
`Arbitrary` implementations to a `refs::arbitrary` sub-module.
Since `SignedRefsAt` require a correctly signed payload, a helper
constructor is added: `signed_refs_at`.
In turn, a function, `arbitrary::with_gen` is introduced so that this
constructor can be easily called with a `Gen` value.
radicle: refactor `Home::load`
- Refactor out getting the subdirectories to ensure they are the same
across `new` and `load`
- Return all missing directories in the error message
- Document `load` and make it `pub`
Configuration of database connections is not performed on `open`, which
leaves room for error (e.g. to miss specifying configuration).
Methods on `Profile` automatically supply the configuration of the
profile.
In testing code, just using the default configuration suffices.
Change the default value for the `synchronous` pragma from `FULL` to
`NORMAL`.
With this change, SQLite will not aggressively `fsync()` after every
transaction, so there is considerably less disk pressure as disk I/O
can be batched.
See <https://sqlite.org/pragma.html>:
> `WAL` mode is safe from corruption with `synchronous=NORMAL`, and
> probably `DELETE` mode is safe too on modern filesystems. `WAL` mode is
> always consistent with `synchronous=NORMAL`, but `WAL` mode does lose
> durability. A transaction committed in `WAL` mode with
> `synchronous=NORMAL` might roll back following a power loss or system
> crash.
> Transactions are durable across application crashes regardless of the
> synchronous setting or journal mode.
Also:
> You lose durability across power lose with synchronous `NORMAL` in `WAL`
> mode, but that is not important for most applications. Transactions
> are still atomic, consistent, and isolated, which are the most
> important characteristics in most use cases.
So, there is no risk of database corruption, and in the extreme
cases of sudden power loss or system crash, some transaction may
roll back.
See also <https://sqlite.org/wal.html>
Co-authored-by: Yorgos Saslis <yorgos.work@proton.me>
Instead of introducing our own names and aliases, directly model SQLite
pragmas with the defaults that they also take in SQLite, to avoid
confusion.
Co-authored-by: Lorenz Leutgeb <lorenz.leutgeb@radicle.xyz>
Introduces the ability to explicitly block a peer via the node control
socket. Previously, the node only exposed follow and unfollow commands.
While the underlying policy database schema supported a Block variant,
there was no mechanism to trigger this state via the client handle.
The new block command:
1. Updates the node's follow policy to Block.
2. Immediately disconnects the peer if a session is active.
The `SeedingPolicy` type now requires a `Scope`, however existing
configurations allow it to not be specified.
Introduce a `radicle::node::config::Scope` type that allows for an
optional `policy::Scope`. The default value resolves to
`policy::Scope::All`. This preserves backwards compatibility for
existing configurations.
Recent versions of Windows support Unix Domain Sockets natively, see
<https://devblogs.microsoft.com/commandline/af_unix-comes-to-windows/>.
By using that feature instead of Windows named pipes, the difference
for handling communication via the control socket comparing Unix-like
systems and Windows becomes smaller:
1. No special paths like `\.\pipe\…` have to be handled.
2. Not two I/O mechanisms are abstracted (named pipe and UDS) but
just one.
3. `winpipe` relies on background threads while `uds_windows` does
not.
Once the feature `windows_unix_domain_sockets` (which is tracked at
https://github.com/rust-lang/rust/issues/150487) stabilizes, it will
likely be possible to just drop the dependency `uds_windows` and use
the implementation in `std::os::windows::net` directly.
Git for Windows horribly mangles the configured binaries through a POSIX
`sh`-like environment even on Windows. We might be able to figure out
which `sh.exe` we would have to execute on Windows, but this is too
brittle. Windows users will have to use other mechanisms (like `$EDITOR`
and `$PAGER`) to configure their programs.
Wire up the new `FetcherService`. This reduces the `Service` fetch
methods down to:
- `fetch`
- `fetched`
- `dequeue_fetches`
- `fetch_refs_at`
This simplifies the code by off-loading the logic to the `fetcher`
family of types, rather than handling the intricacies in the `Service`
itself.
The `Service` now just performs the necessary I/O based on the
returned events from the `fetcher`.
This also removes the fetching state from the node sessions. This
follows the single responsibility, and sessions now only care about
their connectivity.
Breaking changes:
- The `Connected` state of a peer no longer contains fetching
information, which was being returned as part of the JSON payload
result.
- The `rad debug` information for ongoing fetches contained the number
listeners awaiting for results, this was removed.
The previous error `UnexpectedState` was opaque, and ended up in a
confusing message when a user would try to edit the identity document
while not being a deleagte themselves.
Instead, provide a more specific error that mentions the `Did`
performing the action, and the action itself.
Note that the `UnexpectedState` variant was matched on during
evaluation, meaning that operations would still appear in the timeline
of the COB. With the change to the other variant, the timeline does
not record the operation anymore.
Previously, the `references_of` implementation restricted the set of
references to certain categories: heads, tags, notes, rad, and cobs.
This is too restrictive to allow peers to share any references they
want. The only exceptions is references `refs/tmp/heads` that are
created for creating patches.
A call to `Database::prune` with no limit could result in a
domain-defined overflow error.
If the limit is not provided, then it uses `i64::MAX`. It is
reasonable to expect that if the `usize` provided is more than
`i64::MAX` then the `i64::MAX` value is acceptable to use for pruning.
Introduce the `radicle-core` crate for housing data types that are at
the core of the Radicle protocol. The initial data type being added is
`RepoId`. To make the crate as lightweight as possible many of
dependencies are optional and gated by feature flags.
The crate is marked as `no_std` even though it unconditionally depends
on `radicle-crypto`, which is *not* `no_std`. This is to invite
refactoring of `radicle-crypto` to `no_std` at a later time.
The `localtime` crate was defined by cloudhead, and is a minimal
repository with a single `lib.rs`.
Instead of using it as an external dependency, copy the code directly
into a new workspace crate `radicle-localtime`.
The default `serde` implementation goes through the `LocalTime`'s
seconds values rather than milliseconds, since this is the more common
format. This allows the removal of the `serde_ext` functions.
The one place milliseconds was used was for the
`radicle::cob::common::Timestamp` type, where the `Serialize` and
`Deserialize` implementations are manually written.
It also adds a `schemars` feature to remove `schemars_ext` functions
in `radicle` as well.
For some COBs, parent commits can be specified for actions, for
example, patches will bundle their base and head commits as parents.
When the COB stream performs the revwalk, these commits will be
included, and will not have the manifest file – resulting in an error.
Instead, this error can be matched against, and the commit is skipped
instead. The other errors should still be resurfaced, since a commit
with a manifest should be expected to load the operation correctly,
and the commit should exist in the repository when attempting to load
it.
One might argue that a valid operation with a missing manifest could
occur, but that would mean that `radicle-cob` has performed an invalid
write. This is undetectable by the stream API, and is ambiguous with
the case of non-Op commits.
All IPv6 addresses would be considered globally routeable, even though
the Rust standard library offers convenience functions to check for
loopback, link-local addresses etc.
Improve checks for IPv6 routeability to catch the most obvious cases of
local or unspecified addresses.
Refactor the check for IPv4 routeability to be more readable and refer
to RFCs, IANA lists, and Rust stabilization tracking issues as
appropriate.
`Address:is_local` would return `false` for all DNS names. This is
incorrect, with one counterexample being the name "localhost", which
generally resolves to a local (usually loopback) address.
The function is changed to catch "localhost", but also the top level
domain ".localhost", which is reserved in RFC 2606, Section 2.
`Address::is_routable` would return `true` for all DNS names. While it
is much harder to decide global routeability based on a domain name, as
these usually have to be resolved to an address before being able to
judge routability, there is one particular class of names, namely local
ones (see above), which are not globally routable.
Due to connection retries in `radicle-node` not choosing new addresses,
mistakenly using an IPv6 address when IPv6 is not supported would result
in failure to bootstrap.
Work around by removing IP addresses again. This means that DNS or Tor
will be required to bootrstrap.
There is an unfortunate lack of single-responsibility with handling the
addresses to connect to.
This is a band-aid fix until the point in time where this can better improved.
Instead of failing with an `InvalidId` error in
`radicle::storage::ReadStorage::repositories` for repos that don't have
a valid repository ID, skip them and log a warning.
Co-authored-by: Lorenz Leutgeb <lorenz.leutgeb@radicle.xyz>
Change `repositories_by_id` to return `impl Iterator<Item = Result<RepositoryInfo,
RepositoryError>>`
instead of `Result<Vec<RepositoryInfo>, RepositoryError>`.
This allows callers to handle failures on a per-repository basis rather
than having the
entire operation fail if a single repository lookup fails.
Previously, the method would stop processing and return an error as soon
as any repository failed to load. Now it processes all repositories and
returns individual results, making the API more resilient and giving
callers
more control over error handling.
Due to the idiosyncracies of this command there are a few things to note about
this migration.
The `rad sync` command acts as a default subcommand in itself, in that it has
options that do not apply to its subcommand, `rad sync status`. This means that
the options will print with `rad sync --help`, but they will not print with `rad
sync status`.
The `--inventory` flag is not compatible with any of the other flags and
options, thus they are all marked with `conflicts_with`. This cannot be done for
a positional argument, so `clap` will "helpfully" print out the usage, for
example, as:
error: the argument '--inventory' cannot be used with '--fetch'
Usage: rad sync --inventory [RID]
For more information, try '--help'.
Which is incorrect, since `RID` is ignored. This is explained in the `--help`
documentation for `--inventory`.
Co-authored-by: Lorenz Leutgeb <lorenz.leutgeb@radicle.xyz>
Make `git2` an *optional* dependency of `radicle-cob`, and refactor
`radicle` to depend on crates that in turn do not depend on `git2`
*non-optionally*
The main offending dependency of `radicle-cob` is `radicle-git-ext`
from the `radicle-git` workspace in repository
(rad:z6cFWeWpnZNHh9rUW8phgA3b5yGt) which *non-optionally* depends on
`git2`.
So, to achieve removal of this dependency:
1. The crate is refactored to depend on the new crates
`radicle-git-ref-format` `radicle-git-metadata`, and
`radicle-oid` introduced in the previous commits, instead of
`radicle-git-ext`.
2. Some code from the `radicle-git-ext` crate in the `radicle-git`
workspace in repository (rad:z6cFWeWpnZNHh9rUW8phgA3b5yGt) is
copied. See `crates/radicle-cob/src/backend/git/commit.rs`.
This cascades to `radicle` and its dependents.
Firstly, the there is an
`impl Deref<Target=git2::Oid> for radicle_git_ext::Oid`. This made
it very convenient to just deref to obtain the wrapped `git2::Oid`,
so there are many expressions of the shape `*oid` in `radicle` and
its dependents. However `radicle-oid` does not provide
`impl Deref<Target=git2::Oid> for radicle_oid::Oid`, as notably,
`Target` is an associated type, and not a type parameter, so an
implementation of `Deref` would tie the new `Oid` too tightly to a
particular implementation.
Instead, work with `impl From<radicle_oid::Oid> for git2::Oid`
(which can be enabled using the feature flag `radicle-oid/git2`).
This explains the changes from `*oid` to `oid.into()` at every
transition to "`git2` land".
Secondly, `radicle` and its dependents are refactored to also depend
on `radicle-git-ref-format` and (much less prominently)
`radicle-git-metadata` instead of `radicle-git-ext`
This is to avoid pulling in `git2` via these dependencies.
Thirdly, as the re-exports in `crates/radicle/src/git.rs` change,
they are at the same time also cleaned up. Notably, the types from
`radicle-git-ref-format` are re-exported under `fmt` only, not "twice".
While overall this obviously is very much a breaking change, these
changes should mostly amount to changing from `Deref` to `Into`, i.e.,
`*oid` → `oid.into()` and rewriting imports for `radicle::fmt`.
This is indicated by the mostly mechanical nature of the changes to
`crates/radicle-{cli,node,remote-helper}`.
For further details on the crate see its documentation in
`crates/radicle-oid/src/lib.rs`.
Changes to `crates/radicle{,-crypto}/Cargo.toml` are to preserve the
default features of their dependencies `serde` and `schemars`,
which are disabled to support `radicle-oid` being `no_std`.
In commit 2149770a4b the dependency
`tempfile` was made optional, and added to be enabled by the feature
flag `test`. However, the *tests of this crate itself* need the
dependency *unconditionally* as they want to create temporary
directories.
This was not noticed earlier, since the error does not surface when
the whole workspace is tested (`cargo test --workspace`), but only
when the crate itself is tested (`cargo test --package=radicle`),
because when testing the workspace, dependents of `radicle` will
enable the `test` feature flag.
`impl Display for RepoId` will produce a `rid:` prefix, which
contains a colon, and this trips up various filesystems. Use
`RepoId::canonicial` instead.
The following tests generated an unstable object ID, thus linearizing
history in a different order, causing failures. Fix this by using the
`stable-commits` feature of `radicle-cob`.
thread 'cob::identity::test::test_identity_redact_revision'
panicked at crates\radicle\src\cob\identity.rs:1338:9:
assertion `left == right` failed
left: [Oid(b4307ded046befba374bf8cd9fd787592ceb615c),
Oid(a04165e3d3717c5a1413ec78e852bd8e1bb049d4),
Oid(820d3faf5b507888173b26ccd1a2a81666bd2573),
Oid(30ca6f078a401bf542049594fbb1c8d2371c9819),
Oid(9e9c46971014b123a31cd1195078f95c2e319419)]
right: [Oid(b4307ded046befba374bf8cd9fd787592ceb615c),
Oid(a04165e3d3717c5a1413ec78e852bd8e1bb049d4),
Oid(820d3faf5b507888173b26ccd1a2a81666bd2573),
Oid(9e9c46971014b123a31cd1195078f95c2e319419),
Oid(30ca6f078a401bf542049594fbb1c8d2371c9819)]
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
thread 'cob::identity::test::test_identity_reject_concurrent'
panicked at crates\radicle\src\cob\identity.rs:1494:9:
assertion `left == right` failed
left: [Oid(b4307ded046befba374bf8cd9fd787592ceb615c),
Oid(bce4233f6a09d6e9c021ba436e3c6923a1a78f31),
Oid(7cacb23be7079c16c8ad1b2458ae5bac95a1295b),
Oid(12b00e2e871b699c84c20427a64e029934c15ec4),
Oid(07431c8d80c11492ef248e07126370e874de3435),
Oid(98b42450159a359c2f482652c66d79b05d0af624)]
right: [Oid(b4307ded046befba374bf8cd9fd787592ceb615c),
Oid(bce4233f6a09d6e9c021ba436e3c6923a1a78f31),
Oid(7cacb23be7079c16c8ad1b2458ae5bac95a1295b),
Oid(98b42450159a359c2f482652c66d79b05d0af624),
Oid(12b00e2e871b699c84c20427a64e029934c15ec4),
Oid(07431c8d80c11492ef248e07126370e874de3435)]
Fixes test `profile::test::canonicalize_home`.
Since `Home::new` canonicalizes its argument (which is what the test
wants to ensure), also canonicalize the expected value, which is
especially important on Windows as the canonicalization is more
complex than on Linux.
For example:
thread 'profile::test::canonicalize_home' panicked at crates\radicle\src\profile.rs:802:9:
assertion `left == right` failed
left: "C:\\Users\\runneradmin\\AppData\\Local\\Temp\\.tmpMnNkr6\\Home\\Radicle"
right: "C:\\Users\\RUNNER~1\\AppData\\Local\\Temp\\.tmpMnNkr6\\Home\\Radicle"
A refactoring internal to the `radicle` crate, with the goal of
making dependency on `git2` clearer and more controlled.
`radicle::raw` is changed from a complete re-export of `git2`
to a module that selectively re-exports (many) members of `git2`.
required to build the workspace (potentially breaking dependents
outside the workspace, but given just how many types are
re-exported, this seems unlikely).
In the future, no more `use git2::…` statements should be added
outside of `crates/radicle/src/git/raw.rs`. This is in an effort to
decrease dependence on `git2` in the future.
The dependencies `netservices`, `io-reactor` and `popol` served us well,
however they do not support Windows and are not actively maintained.
This change removes the aforementioned dependencies (and `libc` along
with them). It reuses the integration with `cyphernet` from
`netservices` for Noise and SOCKS.
The new module `reactor` is a rewrite of `io-reactor` on top of Mio.
Note that no tests were changed.
The `git-ref-format-core` crate is enough to implement everything
required by `radicle-crypto`, and has a much smaller dependency
footprint than `radicle-git-ext`. Notably, it does not depend on
`git2` and also does not depend on procedural macros.
The "seeds" command does only take a repository ID, and
carries the implicit assumption that the namespace with the same public
key as the Node ID of the receiving `radicle-node` process should be
used as reference for determining whether other nodes are in sync,
but no other namespace.
In the spirit of separating user and node identity, relax this, so that
the command also carries the public keys relative to which the sync
status should be computed. This allows to inspect sync status for
arbitrary namespaces via command.
For now, this feature is not exposed to the CLI, but `rad sync status`
always passes the public key of the active profile.
The "announce references" command does only take a repository ID, and
carries the implicit assumption that the namespace with the same public
key as the Node ID of the receiving `radicle-node` process should be
announced, but no other namespace.
In the spirit of separating user and node identity, relax this, so that
the command also carries the public keys for which the announcement
should be made. This allows to announce arbitrary namespaces out of
storage via command.
For now, this feature is not exposed to the CLI, but rather:
- `rad sync` always announce for the public key of the active profile.
- `git-remote-rad` ditto, as it calls the same codepath.
Some tests actually do not use qualified patterns. The removed lines
*should* not compile because the `qualified_pattern!` proc macro
*should* validate the string literals at compile time, but it contains
a bug and actually does not assert well-formedness of qualified
refspec patterns. Rectify.
Lorenz Leutgeb
Fintan Halpenny
Defelo
Sebastian Martinez
Yorgos Saslis
Adrian Duke
Yaroslav Halchenko
Johannes Kühlewindt
Erik Kundt
Lorenz Leutgeb
