Add an enum to represent the special references, `rad/id` and
`rad/sigrefs`. These can be used for cases where it's more manageable
to work with the enum type over checking for reference string equality.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
Make the validation logic customisable by extracting it out into a
separate trait that can be defined by other crates.
The Repository validation logic stays the same.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
Instead of returning the unsigned references, the validation of a
repository's remotes returns the set of validation errors. The caller
is then expected to handle these validation errors appropriately, for
example, by logging them and re-raising an error.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
Add a type to capture user information that can be used for Git
configuration and signatures.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
The `service::tracking` module has nothing specific to the
radicle-node code. This means that it can be moved to radicle under
the `node::tracking` module.
This makes it more resuable in other contexts, for example, for
fetching logic.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
Similar to the `id` function, add a `sigrefs` function to construct
the reference name for a given remote's `rad/sigrefs`.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
Introduce a SignedRefsAt type that combines a SignedRefs along with the Oid
it was loaded at.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
This change allows users to track/untrack peers and repos without a
running node. This fixes a bug when running `rad init` without the node
running, where the repo would be tracked with a different scope than
what was specified on the CLI.
The `stable-commit-ids` feature flag was being included in non-dev,
non-test builds because of how cargo works (or doesn't work).
For now, we ignore the tests that this affects, and they can be run with
the `RAD_COMMIT_TIME=1514817556` environment.
If a `rad` command that requires an RID is executed within a
subdirectory of a Radicle project, then it will fail saying:
✗ Error: Current directory is not a radicle project
Improve this flow by searching parent directories and/or using the
GIT_DIR environment variable.
This is safe as long as the operations are read-only when calling
`cwd`. Currently they are:
- `Repository::open_ext` -- will not create a repository, only open
one.
- `Repository::find_remote` -- this will only read the remote from the
config and get the URL to parse the RID.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
`cargo clippy --all --all-targets` is warning that there are multiple
redundant clones.
Apply the suggestions by removing the calls to `clone`.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
When testing the CLI, "since"-calculations will result in different
values over time:
now
1 month ago
2 months ago
10 months ago
1 year ago
10 years ago
Initially, the solution was to use the wildcard match from `snapbox`,
`[..]`, to match on these values. However, since these values change
in length it means that they will affect the width of the tables
produced by the CLI.
To ensure the width is stable across runs over time it's possible to
use a time that is the same as the commit time used at testing.
To do this the `Timestamp` type is now a newtype which has a
constructor `now` which will use the `RAD_COMMIT_TIME` env variable
for construction, if it's available and in debug mode. This ensure
that all "since"-calculations will result in `now`.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
Introduces a new COB type to store repository identity documents.
The reason for this change is to:
1. Simplify the code, as the identity document logic resembled a COB,
yet it had custom logic. This allows existing COB code to be used for
identities.
2. Make identity document update logic more flexible, since COB actions
can be added in the future.
3. Re-purpose existing tools around COBs to work on identities, eg. `rad
cob`.
4. Unify the concept of an identity change proposal, with regular identity
changes. This means we can remove the `id.proposal` COB in favor of
using the `id` COB itself.
Notes
-----
* Each repository has one Identity COB.
* The `Proposal` COB has been repurposed into the `Identity` COB.
* Identity documents are stored as *embeds* inside the Identity COB
actions.
* The action that contains new document versions is called `revision`,
just like the Patches COB.
* The namespaced `rad/id` ref is a symbolic reference to that Identity
COB.
* The canonical `rad/id` ref is a direct reference to the commit in the
Identity COB that contains the latest *accepted* revision of the
document.
* All commands for managing identities have been folded into `rad id`.
Hence `rad delegate` and `rad edit` are removed.
* The concept of "rebasing" an identity document is gone.
* The `rad id` output has been updated to match the style of other
commands.
* When a revision has enough signatures, it is automatically adopted as
the current identity, there is no longer the need for a "commit"
action.
* When an identity revision is proposed, and the current identity has a
threshold of `1`, that identity is automatically accepted due to the
above point.
* The idea of "verifying a peer's identity branch" no longer applies, as
COBs cannot be verified one history at a time.
* Since the root commit of the Identity COB does not have a "Resource"
to point to (it would have to point to itself, which is impossible),
we've made the resource id optional for COBs.
The mock transport was not thread-safe when multiple threads used the
same node ids. This is fixed by including the thread id as part of the
storage key.
We employ various tricks to ensure that tests are reproducible.
In this case, having stable commit ids allows us to test specific
traversal orders of a COB history in cases of concurrent updates.
On push, we check whether the resulting state would cause a
divergence/fork in the canonical head, and if so, prevent the push from
happening.
This is to avoid situations where delegates have to then rollback their
heads.
Note that this doesn't prevent forks from happening altogether, as they
could happen asychronously, but it mitigates the problem.
We employ a better algorithm for determining which commit should be
canonical when there are multiple possible options.
The algorithm still doesn't work in certain cases when there are
multiple merge commits as candidates, with shared parents.
We leave some (ignored) failing tests to be resolved in the future.
The `Patch` API and semantics are changed a little bit:
* It's now valid again to propose a revision without being the patch
author.
* To get revisions by the patch author, `Patch::updates` is used.
* `Patch::latest` now only looks at revisions by the patch author.
* Added `Patch::revisions_by`, `Patch::authors` and `Patch::latest_by`.
* The merge behavior on git-push is improved to favor revisions that
merge more recent commits. We also allow for merging the non-latest
revision.
We remove the YAML front-matter, since that was annoying to use.
Issues are now edited like patches and commit messages.
Labeling and assigning are done via dedicated commands.
Instead of returning histories from `radicle-cob`, we return an
evaluated object which we build during initial graph traversal.
We do this so that branches can be correctly pruned when operations are
invalid at the application level. This way, new operations are not
building on top of invalid ones contained in the history.
To achieve this, we introduce a new `Evaluate` trait that is implemented
by all COBs, and we make change graph evaluation fallible.
Doing this means that we get `apply` errors returned for free when an
invalid transaction is applied, and no longer need to check for action
validity in multiple places.
We also implement a new `Dag::prune` method to avoid having to copy
graph nodes during traversal.
Clippy warns that EntryId::from and RevisionId::from are not
required.
Remove the unecessary calls in radicle-cob and radicle.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
Sets the current branch's upstream to point to the given patch.
This allows for pushes to update the patch.
When opening a patch, if the current branch already has an upstream,
it isn't updated automatically. This is why this command exists.
It happened that a small bug in the remote helper caused a patch not to
merge correctly on push. Hence, I implemented a `rad-merge` tool for
these cases where we need to fix broken state.
When a fetch is user-requested, a timeout can be supplied that is passed
down to the worker.
When a fetch is service-requested, a default timeout is used.
Private repos are implemented by extending the identity document with a
`visibility` attribute, that can either be `"public"` (default) or
`"private"`.
In case of `private` visibility, only the delegates are allowed to view
the repo, as well as any DIDs added to the allow list.
To implement repo visibility, we simply block fetches from and
announcements to peers for whom the repo should remain invisible.
Private repos are also not announced in the `inventory` message, since
the full list of peers that *may* have the repo is retrievable from the
repo identity. This could cause errors if eg. a peer who is allowed to
view the repo doesn't actually have it. However this is an ok trade-off
for now to keep the complexity low. For repos to truly be private, it's
important that the RIDs don't leak either.
Finally, we modify `radicle-httpd` for now to only list public repos.
Eventually, we would want to change this depending on whether an allowed
peer is authenticated with the service or not.
---
It's also worth mentioning why this approach was taken, vs. end-to-end
encryption. The reasons are as follows:
1. Nodes that do not have access to a private repo will generally not want to
replicate encrypted data that they cannot examine or use.
2. The chosen solution is trivial, while encrypting git objects isn't.
3. Performance of the chosen solution is much better, there is no
overhead.
4. Privacy of the chosen solution is better: RIDs are never leaked, and
neither is the existence of a private repo, nor who has access to it.
There is one downside: Paying for storage of private repos is no better
in terms of privacy than what GitHub offers. Hosting providers will have
access to your private repos, if this solution is used.
Instead of having two types, we simply have one, called `Entry`, which
we rename from `Change`.
There was no real benefit to having two almost identical types, but lots
of added complexity. This patch simplifies the crate by removing one of
the types.
Add `deny.toml` with a few exceptions, and update certain dependencies
to get rid of duplicates.
Running `cargo deny check` will yield some warnings still, but no
errors.
This change allows for files (blobs) to be embedded into COB entries.
This allows for things like image attachments in issue comments for example.
For now, we only enable this in the `issue` COB.
The way it works is that relevant COB actions carry metadata about which
files are attached to them; and we store those files as blobs inside the
COB entry's *tree* object, under an `embeds/` folder.
The `Embed<T>` type is used for the above, and either carries actual
content, or carries a content-id. Retrieving the actual content is as
simple as asking the repository for a blob with that content-id.
As a possible future extension, MIME types could be stored alongside the
files in a "metadata" file. This could help clients display the content
appropriately.
This adds functionality so that on `rad sync` and `rad clone`, we
connect to known seeds if necessary, before attempting to fetch.
In passing, some related changes were made:
* The `rad sync` command's arguments were reworked:
`--replicas <count>` can be used to specify a replica count when
fetching
* The `rad patch` command no longer has a `--fetch` option
* It's now possible to tell the node not to automatically connect to
peers in the background
* `AddressBook` is being used underneath `Seeds` to allow for shuffled
iteration
Only return once the connection is established or failed.
It's still not perfect, as several failure scenarios are not handled,
but it's an improvement.
We move the session address to the top-level struct, since it's needed
in a bunch of places. This change is required for the rate-limiting
code that is coming next.
As a starting point, we populate the address book (if empty) with some
bootstrap nodes which are trusted.
To prevent these nodes being connected to during tests (including e2e
tests), we create a new constructor for `Config` that is used in tests.
Note that a `cfg(test)` check is not enough given that e2e tests don't
have that set.
> I'm going to be making a set of breaking changes to COBs in order to
stabilize the data formats. This is hopefully a one-time change that
bundles various breaking changes.
All COBs have been reworked: issue, patch, id.
The changes included are:
* Revise the assign and tag actions to take a single list of
assignees/tags to set, instead of an "add" and a "remove" list. This
makes API usage simpler when editing issues, and simplifies the apply
function
* Rename "tags" to "labels", and the tag action to label. This is
because tag is confusing in the context of git, as it could mean a git
tag. Using label removes that confusion.
* Use DIDs instead of PublicKeys for assignees -- this is more
future-proof
* Modify the manifest file format in the COB tree. Mainly, remove the
`history_type` key which is redundant, and use camelCase for keys
* Flatten the `Thread` actions into the parent action type
* Ensure that operations on redacted objects do not fail, since
redactions could have happened concurrently
* Use a consistent naming scheme for actions, using `.` as separator
* Consolidate comment types and remove `CodeComment`, by adding an
optional `location` field to `Comment`
* Add many placeholder actions that are not yet implemented
To preserve backwards compatibility, a `legacy` module is created with
the old `apply` function. When loading the manifest, we check whether it
is a legacy COB or a "stable" COB, and in the legacy case, use the
legacy code to materialize the state and then convert the object into
the stable type. Eventually, we'll delete the legacy code.
Some of the old patches had missing objects and that causes problems
during the ref advertisement, as the object is not found.
In this patch, we check that the object exists and also that the patch
is still open. Otherwise a remote tracking branch is created for every
single patch ref.
It's easier if we use camel-case everywhere, because some types may be
used on the web as well as on the control socket. This way, all
json-encoded data looks the same.
Instead of using strings, use numbers.
*This is a breaking change and affects `config.json` parsing.*
To upgrade, simply unquote any number in your `config.json`.
We were doing a workaround stripping prefixes, but I think it makes more
sense to skip the test all together.
Signed-off-by: Sebastian Martinez <me@sebastinez.dev>
Instead of using a command name plus arguments, we use a new `Command`
enum that is similar to the one in `radicle_node::service`. This way,
we're able to have more complex commands and parsing is simplified.
It turns out that the CRDT formed by the union of Git DAGs
is enough to guarantee everything we need for COBs.
This changes the following things:
* COB operations no longer need to be commutative
* COB histories are traversed in the same deterministic order on all
replicas
* It's now possible to implement RSMs on top of COBs, eg. scripting
* Lamport clocks have been removed
* `radicle-crdt` is no longer a dependency of `radicle`
* COBs are no longer instances of `Semilattice`
* The `Ops` type was removed in favor of having `Op` contain multiple
actions
Help users confirm all their radicle binary versions by providing the
'--version' flag.
Move the print_version function to the radicle crate for use by the node
and remote-rad binaries, and remove tests to confirm the output format
as it is deemed redundant.
Co-Developed-by: Vincenzo Palazzo <vincenzopalazzodev@gmail.com>
This new trait is used when only signing refs is needed, and not general
write access to a repository.
We also implement `ReadRepository` and `SignRepository` for
`DraftStore`.
This storage backend for COBs stores changes in a `draft/cobs/*` namespace,
which allows for some of the features needed for code review. For
example, users can draft comments and later decide to publish
them.
The additional flexibility in choosing a backend for the COB store will
allow things like "draft" COBs that are stored in a different location,
without changing much of the logic.
The tracking database can be opened in read-only or read-write
mode. Knowing which is being used is significant in concurrent
scenarios, e.g. it's safer to have a read-only handle on multiple
threads.
Provide static type information by signifying what kind of database
handle is being worked with, read or read-write.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
This is a fairly substantial change, which adds a new configuration file
to the user's `$RAD_HOME` that includes node configuration, including
the alias, and also makes node aliases required.
When running `rad auth`, the user is now prompted for an alias, which
defaults to `$USER`.
When running `rad self`, the alias is now shown.
If the user runs radicle without a config file, the defaults are loaded,
and `$USER` is used as the alias.
This patch incorporates several important changes to code review:
* Code review verdict is no longer editable
* Made reviews redactable instead
* Renamed review top-level comment to "summary"
* Renamed inline comments to "comments"
* Made inline comments addressable by `EntryId`
* This is to allow for editing
* Removed inline comments from review creation
* Added a `CodeComment` action instead
* Removed `Semilattice` instance from `Review`
* Added a `EditReview` action instead to have more control
* Made it valid for revisions to be redacted concurrently with a review/merge
Most of these changes are needed to support proper inline code comments.
There were a few subtle issues with the apply logic of certain COBs
that should be fixed now.
Note that the underlying store guarantees exactly-once delivery, and so
it does not make sense to test idempotence at the patch level.
* `CodeLocation` had a `blob` and `commit` field. These are redundant,
since the review lives under a revision.
We also change the line range to be diff-compatible, with an old and
new range. The ranges are optional to account for different scenarios
like file deletion, move etc.
* `CodeComment` had redundant getters which were removed in favor of
public fields. We also remove the `timestamp` field, as it will be
the same as the review timestamp.
* Return the correct alias from a node announcement, stripping '\0'
bytes.
* Accept `AliasStore` types as references and as owned.
* Store a `null` in the database if the alias received is empty.
Remove this command in favor of the `git push` flow.
We also remove the `rad/patches` remote configuration which wasn't
working properly with `git fetch --all`.
Previously, we only sent the announcement when there were external
addresses. But it contains other information like the node alias,
therefore we always send it.
Since the proof-of-work can be expensive to compute everytime, we add
the ability to load the local node announcement from the file system.
We also change the proof-of-work parameters so that they are relaxed
when running in debug mode, instead of when running tests.
Finally, we remove the boolean validation of the PoW. Instead, we store
the work per-node, in the database, to be used in the future in case of
DoS attacks or network congestion.
We test to make sure that in the event of concurrent operations,
traversal order is deterministic.
This required some changes to the testing infrastructure. We're also
able to simplify the `Actor` type that now has redundant functionality.
Simplify graph traversal and evaluation by building in some of the
functionality into `radicle-dag`, namely the pruning fold.
We avoid building vectors of graph nodes this way, and simply iterate
over the graph in one go.
Though these are perhaps not the final error types, we name them the same
across COBs and have them work the same way.
All COB modules now have a `Error` enum.
In a future change, we may want the `FromHistory` trait to support
multiple error types, eg. one for `apply` and one for `validate`.
This adds a simple `validate` function to COBs that is run on load. It
allows COBs to check whether they are valid, once all operations are
loaded, and return an error if not.
We change `Patch::latest` to always return something, because a patch is
malformed if it doesn't have at least one revision, and thus would not
load in the first place.
This also implies that there is a latest revision in the face of
concurrent updates.
Note that currently, traversal order for concurrent updates is still random.
This will be fixed in another patch.
Before this change, if a merge op referred to a commit that was not on
the default branch, the merge op and all descendant ops would be pruned.
This is a bad idea because the state of the default branch can change
*after* the merge op is created, which is out of the control of the
merge op author.
Therefore, instead of exiting, we keep processing ops (but ignore the
merge), in case the merge is invalid for this specific reason.
Previously, only one connection to the node control socket could be
handled at a time. This usually worked fine, but if one connection hangs
for whatever reason, no other process can access the node.
When applying a `Merge` action, we verify that the commit is in the
history of the default branch of that user.
Since this is not the case when issuing a `rad merge`, and since that
command is no longer needed with the push-based merge, we remove the
`rad merge` command completely.
Move `merges` from `Revision` to `Patch`, and key them by actor.
This ensures that only one merge is possible per actor. We also make
sure that only one revision can be merged at a time, and signal a
conflict if more than one revision is considered merged.
Finally, we make sure that merges have the same target-branch commit
to be considered equal.
Allows patches to have their state updated to `Merged` with a `git push`
to the default branch.
Currently, due to the limitations of the COB store, this requires a
linear search through all patches. This will eventually become a problem
when projects have thousands of patches, so we should look into
long-term solutions.
When updating sigrefs with `storage::refs::save` we weren't checking
the `RAD_COMMIT_TIME` variable, so the sigrefs commit was using current
time, and we weren't able to reproduce a deterministic sigrefs commit.
Make `refs::save` reproducible in debug mode using `debug_assertions`
and checking the `RAD_COMMIT_TIME` environment variable.
Signed-off-by: Sebastian Martinez <me@sebastinez.dev>
The `refs/tmp` namespace is used by Git for the purpose of storing
temporary refs. We use it here instead of `refs/patch` to indicate
that the refs can be safely deleted.
This adds the ability to the git-remote-helper to open and update
patches via a `git push` to the magic ref `refs/patches` and
`refs/heads/patches/<patch-id>`.
The main change is that we had to move away from offering the `connect`
capability in the remote helper, to offering `push` and `fetch`
individually.
To get a feel for how this works, see the `rad-patch-via-push.md`
example.
This commit addresses the issue of recursion in the TryFrom
call of Reference by resolving the parent in a lazy way.
Due to constraints posed by the need to access `git2::Repository`
or `git2::Commit` to resolve the parent, this commit decides to
pull out the parent resolution logic from the `cob::object::Commit`
module and move it inside the `cob::store::Store` implementation by adding
a new method `parents_of`.
Suggested-by: Fintan Halpenny <fintan.halpenny@gmail.com>
Tested-by: Vincenzo Palazzo <vincenzopalazzodev@gmail.com>
Signed-off-by: Vincenzo Palazzo <vincenzopalazzodev@gmail.com>
This change updates to the latest radicle-git-ext, which required a
few changes within the radicle family of crates.
One set of changes is that the radicle-git-ext crate subsumes
git-commit, git-trailers, and git-ref-format -- so all those imports
go through radicle-git-ext.
The commit code requires that parent `Oid`s point to commits, which
made some tests fail. One of the reasons for these failures is that
the patch tests used fabricated `Oid`s and so the verification would
fail. This is fixed by allowing the test context code to create
`PatchRequest`s that create valid commits.
The other reason for failure was that the identity `Action` passed an
`Oid` that pointed to a blob. This is fixed by removing that code and
making a note of it in documentation.
The final major change was moving the Refspec type into git-ref-format
and removing the AsRefspecs trait. The trait was not required and
could its usage could be replaced by simpler code.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
If the 'rad' remote is still in the git config, then some of the CLI
tooling will think the project exists -- even after running `rad rm`.
For example, in the current state, if `rad rm` is followed by `rad
init` then the operation will fail.
During `rad rm`, attempt to delete the 'rad' remote.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
When attempting to connect to a peer, and when succeeding, write the timestamp
to the database.
This will allow us to better select addresses and not retry peers to
soon.
This change ensures that we include as a parent to the initial commit
of a patch COB, the head of the branch we are proposing as a patch.
This was already happening for patch updates, but we forgot to also
include it for the initial patch revision.
This ensures that the code is always fetched alongside the patch.
If the reference did not exist it is not necessary to create a new
`sigrefs` entry. Check that the reference exists before removing from
the cob store and only sign if it did.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
It wasn't being set correctly, and it's not the place for it,
so we remove it and set it in `httpd`.
Note that `Remote` is now simply a wrapper around `SignedRefs`.
If we want to remove the type or turn it into an alias, it will
require touching lots of different parts of the codebase. I opted
not to do that here as it's lower priority.
We were allowing remotes to be constructed via a function that didn't
check its inputs.
This is now fixed by moving the public key into the `SignedRefs` struct.
This commit fixes the inconsistency with the `rad patch` command,
which previously displayed an empty table when there were no issues. Now,
the CLI will display a message indicating that there are no issues to display.
Signed-off-by: Vincenzo Palazzo <vincenzopalazzodev@gmail.com>
This command announces refs to peers and waits for them to be
in sync.
Adds a subscribe method to `Handle` so that we can get the events
from the seed to allow us to confirm that the configured seeds fetched
from us.
This commit adds the cob state of either patches or issues to the
corresponding endpoint.
So we can either query a cob listing with the required state or expect
the default state.
And once filtered by cob state we paginate.
Signed-off-by: Sebastian Martinez <me@sebastinez.dev>
Fintan Halpenny
cloudhead
Michalis Zampetakis
Sebastian Martinez
Alexis Sellier
Arastoo Bozorgi
xphoniex
Slack Coder
Vincenzo Palazzo