This prepares the crate versions for a new release. The version bumps were
determined by the `cargo-semver-checks` tool.
However, there were some issues with the `radicle-fetch` crate – where when
attempting to check it, it was unable to compile due to the previous version
expecting the kind of `Doc` with a generic parameter. I believe this is due to
the version for `radicle` being specified as simply `0`. This prompted changing
the version specifications to use `major.minor` for all the internal crates.
This, for example, would mean that `radicle-fetch` that relies on `0.15` can use
`>=0.15.0,<0.16.0` – allowing any patch versions but must require a minor
version bump.
Leverage `schemars` to generate a JSON Schema from our structs for
configurations and those occurring within them.
The output of `rad config schema` can be used by editors to provide a
smoother editing experience for the configuration file. Discovery
attributes (both keys and values) is greatly improved with the schema
helping in the background.
Refer to:
- https://json-schema.org
- https://code.visualstudio.com/docs/languages/json#_json-schemas-and-settings
- https://schemastore.org
The motivation of this change is to move away from tying signing to be
specifically for ed25519. The reason being that the protocol will want move
towards two different kinds of signing – the node signing artifacts, and an
author signing artifacts.
This change captures the former, node signing, by introducing a `Device` type
that is the `NodeId` – a PublicKey underneath the hood – and a signing
mechanism. This allows the replacement of the `Signer` trait being used – which
always assumed a `PublicKey`. Instead, the `Device` is constructed with
`NodeId`.
In `radicle-cob`, a signer is expected to implement
`signature::Signer<ExtendedSignature>`, and everywhere in `radicle`,
`radicle-node`, `radicle-cli`, and `radicle-remote-helper` is expected the
signer is expected to implement `signature::Signer<Signature>`.
A `Device` implements both of these but only requires
`signature::Signer<Signature>` to do so – since an `ExtendedSignature` is
essentially `(PublicKey, Signature)`, and the `NodeId` of the `Device` can be
used.
This introduces rate limits for the `ChannelReader` to limit DDoS attacks and
attempts to upload repositories that are larger than a node is will to permit.
The limiter sets the total number of bytes it is will to accept in a single
exchange, defaulting to 500MB. This means that initial fetches will prevent
large repositories, but is plenty for new packfile data to be sent in subsequent
fetch exchanges.
The limit can be configured within the node's config file, under the limits.
This resolves issues:
- 111cf90 (`handler` is not async-signal-safe)
- c8e0300 (`libc::signal` should not be used)
- b91c6c3 (Mutex can cause signals to be missed)
See also: patch 8e05702.
The API of `radicle-signals` is the same as before, and so uses of it
don't need to be and aren't changed. The behavior is slightly different
in that:
- If the channel is full then signals will not be lost, which is an
improvement. This is achieved without blocking in the signal handler.
This is possible because of the counters approach along with the
internal receipts-processing thread of the `signals_receipts` crate.
- `install()` and `uninstall()` might block very briefly if necessary to
acquire the mutex, which is now internal to and managed by the
`signals_receipts` crate, only if there are concurrent calls to them
(which is unlikely), but such blocking is guaranteed to be bounded to
be very brief. This is done so they no longer can fail to do their
purpose, which is an improvement. They still return errors if the
handling is already installed or uninstalled, respectively, which
preserves the previous use cases.
- The new `finish()` function is introduced. This is provided in case
it's ever needed to completely clean-up the facility, by terminating
the internal receipts-processing thread, to be like it hadn't been
installed before.
- The user must ensure that the notifications channel is disconnected,
by dropping the receiver(s), when doing `uninstall()` or `finish()`.
Such dropping usually occurs naturally, and already occurs for all the
preexisting uses of `radicle_signals`. (The `signals_receipts` crate
is capable of a more robust approach, but this commit doesn't use
that, to avoid changing the preexisting uses of `radicle_signals`.)
- The `TryFrom` impl for `Signal` is of `SignalNumber` which is `c_int`,
instead of `i32`, because `c_int` (the type of signal numbers) might
not be `i32` on all platforms (POSIX only requires `c_int` to be at
least 32-bit).
The version of `radicle_signals` is incremented, to reflect those
changes and the substantially different internal implementation.
The new dependency on the `base64` crate is needed by the
`signals_receipts` crate for it to work on macOS, because its dependency
on the `sem_safe` crate uses `base64` as part of creating anonymous
semaphores on macOS (which lacks support for unnamed semaphores (in
violation of POSIX)).
Signed-off-by: Derick Eddington <kcired@pm.me>
Signed-off-by: Lorenz Leutgeb <lorenz@leutgeb.xyz>
Version 0.5.13 was yanked, resulting in the following warning when
building with --locked:
warning: package `crossbeam-channel v0.5.13` in Cargo.lock is yanked
in registry `crates-io`, consider running without --locked
The nix flake's checks were failing for various reasons. The following fixes are
applied to get it up to date again.
- Update to release-24.11, since `crane` requires it – also to keep up-to-date
- Add `.json` to the source filters to allow for testing patch migration
- Ensure that the `rad` and `git-remote-rad` binaries are built for the nextest
check. N.b. the binaries need to be in the target dir of `radicle-cli` since
that is where the nextest check runs from.
- Add `cargo-audit` and `cargo-deny`, to the devShell, to allow for running those
tools locally
- Bump `gix-path` to 0.10.11 to avoid a vulnerability[^0]
[^0]: https://rustsec.org/advisories/RUSTSEC-2024-0371
Extend the ranges of emojis that are supported, listed
[below](#new-emojis-supported).
To ensure that single `char` emojis are supported, the `emojis` crate
is added as a dev dependency, using it to check that a `Reaction` can
be constructed for the two groups `SmileysAndEmotion` and
`PeopleAndBody`.
### New Emojis Supported
'🩵' 0x1FA75
'🩶' 0x1FA76
'🩷' 0x1FA77
'🫀' 0x1FAC0
'🫁' 0x1FAC1
'🫂' 0x1FAC2
'🫃' 0x1FAC3
'🫄' 0x1FAC4
'🫅' 0x1FAC5
'' 0x1FAC6
'🫠' 0x1FAE0
'🫡' 0x1FAE1
'🫢' 0x1FAE2
'🫣' 0x1FAE3
'🫤' 0x1FAE4
'🫥' 0x1FAE5
'🫦' 0x1FAE6
'🫨' 0x1FAE8
'' 0x1FAE9
'🫰' 0x1FAF0
'🫱' 0x1FAF1
'🫲' 0x1FAF2
'🫳' 0x1FAF3
'🫴' 0x1FAF4
'🫵' 0x1FAF5
'🫶' 0x1FAF6
'🫷' 0x1FAF7
'🫸' 0x1FAF8
Ensures dead connections are not kept around longer than necessary.
Note that the ping/pong mechanism is more expensive and checks that the
session is active and healthy, while TCP_KEEPALIVE is lower level,
cheaper, and checks that the connection is still alive.
Upgrade the family of `gix` crates and resolve any breaking API changes.
Fixes RUSTEC-2024-0335[[0]].
[0]: https://rustsec.org/advisories/RUSTSEC-2024-0335.html
Co-authored-by: Fintan Halpenny <fintan.halpenny@gmail.com>
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
Emit the `UploadPack` events from the `upload_pack` method.
A wrapper type, `Reporter`, is introduced for implementing a `Write` instance
that will parse the given bytes as `gix_protocol::RemoteProgress`, if possible,
and emit it via the `Emitter`, and then write the bytes to the underlying
writer. This allows any subscribers to see the progress of an upload-pack, i.e.
`Compressing objects`, `Counting object`, `Enumerating objects`.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
This is a pager (like `less`) that takes an `Element`, and dynamically
re-renders the element when the terminal size changes. This ensures that
the content is always properly wrapped and rendered.
Many (most?) system clocks are not monotonic, and even though we're
getting a `LocalTime`, which *is* monotonic when using `now()`, it is
constructed from a `SystemTime` which isn't.
There was an issue with the old version due to the use of `RawFd` as
peer IDs, since they are not unique for the lifetime of a process.
With this change, we use the new `ResourceId` type, though these are not
immediately available, as they are generated when the resource is
registered.
To be able to use log setup across the different components of
heartwood, expose a `logger` module from `radicle`. This also includes
a `test` logger.
These modules are guarded behind a `logger` feature flag, so that
other crates are not bringing any uneeded dependencies.
This allows logging to be setup in `radicle-cli-test` so that when a
test fails, the log output is present for debugging purposes.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
Instead of `rad-web` just creating a session for an already running HTTP
API the command now starts the HTTP API, creates a session and opens the
authentication link.
Signed-off-by: Thomas Scholtes <geigerzaehler@axiom.fm>
There are two vulnerabilities that were due to `ssh-key`'s
dependencies[[0]][[1]].
Updating to ssh-key-0.6.3 removes the vulnerability[[0]], however,
there is no existing patch for [[1]] but it is being worked on.
[0]: https://rustsec.org/advisories/RUSTSEC-2022-0093
[1]: https://rustsec.org/advisories/RUSTSEC-2023-0071
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
Update to git2-0.18.1, which also requires updating radicle-git-ext
and radicle-surf.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
Wire up the `radicle-fetch` crate to the fetch logic in the
`radicle-node` worker.
Any required traits for storage, identity, tracking, and transport
used by `radicle-fetch`.
The worker adds a new module to run the `git upload-pack` process and
pipes the `stdout`/`stdin` to the respective channels.
The `Worker` type required adding a `FetchConfig` to easily allow the
configuration of fetches and passing the signer and tracking store
location through to the fetch `Handle`.
The channels code is adapted to have a new writer that always flushes
when a call to `write_all` is made. This is necessary since the
Gitoxide code never calls `flush`, and so it will hang since no data
is sent until `flush` is called.
Since `UserInfo` is used to set up a newly cloned repository, this
changed some of the SHA1 hashes output in the CLI test examples. It
was also necessary to update the CLI's clone code to be deterministic
in its output for tests.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
The `radicle-fetch` crate introduces a method for fetching from a
server using Git protocol V2, which provides the implementation
control over performing a sequence of fetches and validation over
domain data.
The protocol is built on top of the suite of Gitoxide crates[[0]]. It
builds wrappers on top of Gitoxide's refdb, odb, and transport layers
-- as well as taking the ls-refs and fetch code straight from the code
base to simplify for the needs of this crate.
[0]: https://github.com/Byron/gitoxide
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
Introduces a new COB type to store repository identity documents.
The reason for this change is to:
1. Simplify the code, as the identity document logic resembled a COB,
yet it had custom logic. This allows existing COB code to be used for
identities.
2. Make identity document update logic more flexible, since COB actions
can be added in the future.
3. Re-purpose existing tools around COBs to work on identities, eg. `rad
cob`.
4. Unify the concept of an identity change proposal, with regular identity
changes. This means we can remove the `id.proposal` COB in favor of
using the `id` COB itself.
Notes
-----
* Each repository has one Identity COB.
* The `Proposal` COB has been repurposed into the `Identity` COB.
* Identity documents are stored as *embeds* inside the Identity COB
actions.
* The action that contains new document versions is called `revision`,
just like the Patches COB.
* The namespaced `rad/id` ref is a symbolic reference to that Identity
COB.
* The canonical `rad/id` ref is a direct reference to the commit in the
Identity COB that contains the latest *accepted* revision of the
document.
* All commands for managing identities have been folded into `rad id`.
Hence `rad delegate` and `rad edit` are removed.
* The concept of "rebasing" an identity document is gone.
* The `rad id` output has been updated to match the style of other
commands.
* When a revision has enough signatures, it is automatically adopted as
the current identity, there is no longer the need for a "commit"
action.
* When an identity revision is proposed, and the current identity has a
threshold of `1`, that identity is automatically accepted due to the
above point.
* The idea of "verifying a peer's identity branch" no longer applies, as
COBs cannot be verified one history at a time.
* Since the root commit of the Identity COB does not have a "Resource"
to point to (it would have to point to itself, which is impossible),
we've made the resource id optional for COBs.
This commit introduces the `FileStats` struct, that adds the insertions
and deletions for each file to `DiffContent`.
Signed-off-by: Sebastian Martinez <me@sebastinez.dev>
When `EDITOR` or `VISUAL` are not set, we fallback to the most common
editors. It turns out many users don't have these variables set.
We also check if an editor has been configured with `git`.
Add `deny.toml` with a few exceptions, and update certain dependencies
to get rid of duplicates.
Running `cargo deny check` will yield some warnings still, but no
errors.
This change allows for files (blobs) to be embedded into COB entries.
This allows for things like image attachments in issue comments for example.
For now, we only enable this in the `issue` COB.
The way it works is that relevant COB actions carry metadata about which
files are attached to them; and we store those files as blobs inside the
COB entry's *tree* object, under an `embeds/` folder.
The `Embed<T>` type is used for the above, and either carries actual
content, or carries a content-id. Retrieving the actual content is as
simple as asking the repository for a blob with that content-id.
As a possible future extension, MIME types could be stored alongside the
files in a "metadata" file. This could help clients display the content
appropriately.
This improves response time of:
- GET /api/v1/projects/<RID>/tree/<OID>/ from 205.71ms down to 376.4µs
- GET /api/v1/projects/<RID>/tree/<OID>/.nix from 111.06ms down to 351.70µs
Signed-off-by: xphoniex <dj.2dixx@gmail.com>
This commit addresses a workaround previously implemented to
include the SQLite library bundled inside the binary.
The upstream have successfully merged the patch that implements this
solution [1], allowing us to safely incorporate it
into our codebase.
[1] Link to the merged patch: https://github.com/stainless-steel/sqlite/pull/71/files
Fixes: 431a389944
Signed-off-by: Vincenzo Palazzo <vincenzopalazzodev@gmail.com>
Instead of asking for the passphrase in `radicle-node`, we expect the
`RAD_PASSPHRASE` env var to be set.
This keeps `radicle-node` simpler and lower level, while the passphrase
prompt is still available via `rad node start`.
The `cyphernet` package has two feature flags that delegate to
underlying crates. These are:
* ed25519 = cyphergraphy/ed25519
* p2p-ed25519 = cypheraddr/p2p-ed25519
When attempting to install binaries, the compiler will result in an
error:
```
error[E0432]: unresolved import `cyphernet::addr::PeerAddr`
--> radicle/src/node.rs:26:9
|
26 | pub use cyphernet::addr::PeerAddr;
| ^^^^^^^^^^^^^^^^^^^^^^^^^ no `PeerAddr` in the root
```
The `PeerAddr` type is only exported when `p2p-ed25519` or
`cypheraddr/p2p-secp256k1` is enabled. The `radicle` crate was using
the `ed25519` flag. Instead, the `p2p-ed25519` flag should be used.
No explanation as to why the previous flag was working could be found,
but is being discussed on Zulip[[0]].
To update the dependencies, a `cargo update` was also excuted to
update the dependencies.
[0]: https://radicle.zulipchat.com/#narrow/stream/369277-heartwood/topic/cyphernet.20dubious.20feature.20flag
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
Our build was faking the inclusion of SQLite and dynamically
linking it in the 'radicle' crate and other components.
This patch solves the following error and requires reinstallation
of the `radicle-cli`, `radicle-node`, and, if used, the `radicle-httpd`
```
✗ Sync failed: internal error: malformed database schema (node-policies) - near "strict": syntax error (code 11): malformed database schema (node-policies) - near "strict": syntax error (code 11)
```
Tested-by: Vincenzo Palazzo <vincenzopalazzodev@gmail.com>
Signed-off-by: Vincenzo Palazzo <vincenzopalazzodev@gmail.com>
This adds the ability to the git-remote-helper to open and update
patches via a `git push` to the magic ref `refs/patches` and
`refs/heads/patches/<patch-id>`.
The main change is that we had to move away from offering the `connect`
capability in the remote helper, to offering `push` and `fetch`
individually.
To get a feel for how this works, see the `rad-patch-via-push.md`
example.
This change updates to the latest radicle-git-ext, which required a
few changes within the radicle family of crates.
One set of changes is that the radicle-git-ext crate subsumes
git-commit, git-trailers, and git-ref-format -- so all those imports
go through radicle-git-ext.
The commit code requires that parent `Oid`s point to commits, which
made some tests fail. One of the reasons for these failures is that
the patch tests used fabricated `Oid`s and so the verification would
fail. This is fixed by allowing the test context code to create
`PatchRequest`s that create valid commits.
The other reason for failure was that the identity `Action` passed an
`Oid` that pointed to a blob. This is fixed by removing that code and
making a note of it in documentation.
The final major change was moving the Refspec type into git-ref-format
and removing the AsRefspecs trait. The trait was not required and
could its usage could be replaced by simpler code.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
By introducing a small limitation: only allowing entries in the change
graph to be addressable, instead of individual operations; we
drastically simplify the CRDT implementation.
There are four advantages:
1. Op ids are just regular SHA-1s
2. There's no need for relative IDs, ops never refer to other ops within the
same commit
3. There's no need for a nonce, since commits can't collide, and neither can op IDs
4. `OpId` can just be an alias of `EntryId`
The disadvantage of course, is that we have to be mindful of how we
create op transactions, since each transaction creates an addressable
unit. For example, we must not include multiple patch revisions in the
same transaction.
It's fairly easy for a user to (by mistake or intentionally) create two
operations with the same OpId. This patch makes it much less likely,
and ensures that if the OpId is equal, it's because the operations
are identical.
Improvements to the CLI output as well as converging towards
`termion` as a backend.
The crates we were using for prompting and spinners were based on
a terminal backend called `console`, which is not really used
anywhere else. There were also limitations in terms of the output
of these crates.
Therefore, we switched to the `inquire` crate for prompting, using
the `termion` backend, which is very standard.
Additionally, we simplify the terminal output to use symbols:
! for Warning
✓ for Success
✗ for Error
... Instead of a mix of words and symbols.
We also include a modified version of the `yansi` crate which plays
well with our table formatting functions by automatically truncating
and padding output in a unicode-aware way.
The olpc-json crate's CanonicalFormatter does not escape certain
control characters, including new lines. This means that when a COB
gets serialized that includes new lines, those new lines are
serialized verbatim. Upon deserialization `serde_json` -- and any
other JSON deserializer -- will fail due to the encounter of these
control characters.
This behaviour was prevented in radicle-link with the definition of
its own CanoncialFormatter that encodes the control characters
correctly.
Inline this definition of the formatter, under MIT license, to ensure
the correct and remove oplc-cjson.
The behaviour is checked using a multiline test in the issue COB.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
Instead of showing just public keys we need to show the fully qualified
spec for a NID which is prefixed by `did🔑` While we want to maintain
`Author` and `ActorId` for internal use.
Signed-off-by: Sebastian Martinez <me@sebastinez.dev>
Introduces the `rad id` subcommand for interacting with the identity
proposal cob.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
There are few unrelated changes in this commit that I wasn't able to
split out, mostly due to laziness, so I'll list them here:
* The node startup logic is moved out of the main function, and into a
dedicated functionin `client.rs`.
* There is a tentative worker implementation that is untested, and
handles git replication both ways. It's likely lacking in various ways
but I wanted to get it in there anyway.
* `Address` was moved from `radicle-node` to `radicle`, because it's
needed to send `Connect` commands from the node handle.
* The `netservices` dependency was updated.
* Finally, there is a first end-to-end node test, which currently
doesn't use NoiseXK, but uses the new reactor in `netservices`.
Signed-off-by: Alexis Sellier <self@cloudhead.io>
We're using petgraph to encode an operation based CRDT (each graph node
is an operation) as a DAG, and one of the important things to ensure
is that the final state is not influenced by the order of concurrent
operations. In this case a concurrent operation would mean two potential
graph traversal orders, as there's no direct edge between the two
concurrent ops. This is essential a "partial order" of operations.
To be able to test this, we'd want some kind of control over how
neighbors are iterated over I guess, when there's more than one possible
sort order. Say we have a graph like this:
┌────────b◄─────┐
▼ │
a d
▲ │
└────────c◄─────┘
There are two possible topological traverse orders: [a, b, c, d] and [a,
c, b, d]. Having a way to go through these different orders would
be super handy.
One option would be to allow random order traversal. This would allow us
to test all orders by running the test enough times to likely test all
permutations.
Since petgraph doesn't support this, we implement our own simple DAG in
`radicle-dag`, which implements random-order topological orders.
For now, we don't make explicit use of these improvements, we simply
replace the underlying graph with our own.
Fintan Halpenny
n4ch7
Sebastian Thiel
Derick Eddington
Ivan Stanković
Lorenz Leutgeb
cloudhead
Sebastian Martinez
Alexis Sellier
Dr Maxim Orlovsky
Thomas Scholtes
xphoniex
Vincenzo Palazzo
Erik Kundt
Alexis Sellier
Dr Maxim Orlovsky
xla
Slack Coder
Han Xu
Dr Maxim Orlovsky