It is desirable to keep the Git repositories in Radicle efficient by
ensuring objects are packed and unreachable objects are removed.
Add a process call to `git gc` to achieve this. Two flags are used for
garbage collection: `--auto` and `--prune`. Their details are outlined below:
--auto
With this option, git gc checks whether any housekeeping is
required; if not, it exits without performing any work.
--prune=<date>
Prune loose objects older than date (default is 2 weeks
ago, overridable by the config variable
gc.pruneExpire). --prune=now prunes loose objects
regardless of their age and increases the risk of
corruption if another process is writing to the repository
concurrently; see "NOTES" below. --prune is on by default.
For `--prune` the default used is `1.hours.ago`. This was chosen over
`now` since that may be too aggressive after just fetching new data.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
Ref announcements contain the namespace (NID) and the SHA of the new
`rad/sigrefs`. This information can be used by the fetch protocol to
fetch only those `Oid`s for those namespaces.
Add a new stage `SigrefsAnnouncement` that captures this behaviour by
only asking for the advertised `Oid`s and updating the corresponding
`rad/sigrefs` for each namespace. Note that this is a special case of
the `SpecialRefs` stage and so the protocol chooses between these two
by checking if there was announcement passed through. The following
stage, `DataRefs`, stays the same.
Note that this stage can only be activated via a `pull`, and `clone`
stays the same.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
The RefsAnnouncement previous announced its complete set of
`rad/sigrefs`. This was useful in that verification can occur on the
receiveing side immediately by verifying the refs and proceeding with
the fetch -- note that it did nothing with the ref information it
received.
This can be minimised to only advertising the namespace, i.e. the
NID, and the new SHA of the `rad/sigrefs`. This is represented by a
new message type, `RefsAt`, which contains the NID and the Oid.
For checking if an announcement is stale or fresh, the receiving
side can check the existence of the SHA in its Git repository.
The tests are appropriately modified to use NIDs and Oids.
To prepare for a follow-up change, the announcement data is also
threaded through for the fetch protocol to focus fetching only the
advertised set of Oids.
N.B.: because the refs aren't communicated, the verification happens
when fetching since the data needs to be fetched before
verifying. This is a trade-off with minimising the message payload for
verifying earlier.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
The minimal error for the `ReadRepository::reference_oid` method is
`git::raw::Error`.
Change the error type from `git::ext::Error` to `git::raw::Error`.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
While there is a method, `unbound`, for converting a `BoundedVec` to
its inner `Vec`, it is common practice to implement the `From` trait.
Add the `From` conversion for `BoundedVec` and `Vec`.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
Add debugging code for inspecting the in-memory Refdb of the Cache.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
When performing a clone there are two potential issues.
The existing issue is that when a repository is created to clone into,
it does not contain any data initially. This leaves other node
operations thinking that a repository does exist but when they try to
perform operations on it, those will fail. These will be surfaced as
warnings but the node will continue to operate.
The other issue, while writing fetch v2, is that if a clone fails,
then cleanup needs to be performed -- to remove the empty repository
from storage.
To tackle both these issues, a temporary directory is used in the
place of the repository. If a clone fails then the temporary directory
is dropped and nothing needs to be done. If the clone succeeds then it
can be safely moved to the storage.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
It's difficult to debug errors that mention a resource not being
found, but not which resource was not found.
Return the error message for not found variants.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
The fetch V2 improvements introduced a `UserInfo` type that is used to
set the Git config's name and email when cloning a repository. This
`UserInfo` can also be used when creating a new `Repository`.
The `Storage` type now takes a `UserInfo` so that it can be passed to
`Repository::create`, for creating a new repository.
For testing, this requires adding a new `fixtures::user` so that
testing `Storage` instances can be easily constructed.
It also means that the `radicle-cli` tests all changed with new SHAs
since the name and email are used as part of the hashing input.
N.b. the order of the timelines while testing the identity cob
changed. It's not clear why that is.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
Instead of calling `std::fs::remove_dir_all` in the `rad rm` command,
use the `WriteStorage::remove` method.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
Previously, if a delegate's namespace was fetched and its rad/sigrefs
was behind the current state, then it would fail to perform the fetch.
Instead, when the ancestry path is checked the outcome can be used
during the update:
1. If the tip is equal, then the update can be marked as skipped.
2. If the tip is behind, then the update can be rejected, iff the
policy does not allow it.
3. If the tip is ahead, then the update is a fast-forward.
4. If the tip is diverged, then the update results in a failure, iff
the policy does not allow it.
The protocol also prunes any delegates's updates when the `rad/sigrefs`
is behind and returns an error if they are diverged.
For any non-delegates, it will simply prune them.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
Previously, we would always ask nodes for messages from one hour ago.
This meant that we could miss messages if our node was offline for more
than an hour. This change uses a variable backlog by checking the
timestamp of the last gossip message we received.
Instead of storing received gossip messages in an in-memory `BTreeMap`,
we persist them to the database (`addresses.db`). There are two reasons
for this:
1. Node restarts do not wipe the state, meaning that it is much less
likely that some gossip message will get lost before reaching all
nodes.
2. This will allow us in the next commit to know how far behind we are
after a restart, to be able to request the correct time range of
gossip messages that we missed while offline.
Note that we now share the addresses.db file between two connections and
stores, which we don't do anywhere else. Having multiple connections to
the same database is fine, but the way we're having to do it is a bit
awkward. In a future patch, we could pull out the database schema from the
`address` module and rename the database to make things clearer.
This makes the divergence check on push slightly more lenient, by
allowing rolling back to a previous commit in the canonical branch
history.
Without this, it can become impossible to roll-back commits in certain
situations.
With the latest changes to fetching, `--scope all` will now clone all
remotes listed by the serving node.
Simplify by the `rad-clone-all` example by removing the extra fetch
and removing the corresponding TODO.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
The git-daemon code is no longer needed since git-upload-pack is used
in its place.
Remove the git-daemon code from the runtime and worker.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
Wire up the `radicle-fetch` crate to the fetch logic in the
`radicle-node` worker.
Any required traits for storage, identity, tracking, and transport
used by `radicle-fetch`.
The worker adds a new module to run the `git upload-pack` process and
pipes the `stdout`/`stdin` to the respective channels.
The `Worker` type required adding a `FetchConfig` to easily allow the
configuration of fetches and passing the signer and tracking store
location through to the fetch `Handle`.
The channels code is adapted to have a new writer that always flushes
when a call to `write_all` is made. This is necessary since the
Gitoxide code never calls `flush`, and so it will hang since no data
is sent until `flush` is called.
Since `UserInfo` is used to set up a newly cloned repository, this
changed some of the SHA1 hashes output in the CLI test examples. It
was also necessary to update the CLI's clone code to be deterministic
in its output for tests.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
The `radicle-fetch` crate introduces a method for fetching from a
server using Git protocol V2, which provides the implementation
control over performing a sequence of fetches and validation over
domain data.
The protocol is built on top of the suite of Gitoxide crates[[0]]. It
builds wrappers on top of Gitoxide's refdb, odb, and transport layers
-- as well as taking the ls-refs and fetch code straight from the code
base to simplify for the needs of this crate.
[0]: https://github.com/Byron/gitoxide
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
Add the ability to remove a repository, given its RID.
This is useful in the case of attempting to clone a repository, and if
that process fails -- for example, if the repository is private --
then it's necessary to clean up the initially created repository.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
The `identity_doc_at` and `identity_doc` methods on `ReadRepository`
returned the unverified identity document. Every usage of these
methods would always verify the document after.
Simplify the code by calling verified in the methods and directly return the
verified identity document.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
Add an enum to represent the special references, `rad/id` and
`rad/sigrefs`. These can be used for cases where it's more manageable
to work with the enum type over checking for reference string equality.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
Make the validation logic customisable by extracting it out into a
separate trait that can be defined by other crates.
The Repository validation logic stays the same.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
Instead of returning the unsigned references, the validation of a
repository's remotes returns the set of validation errors. The caller
is then expected to handle these validation errors appropriately, for
example, by logging them and re-raising an error.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
Add a type to capture user information that can be used for Git
configuration and signatures.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
The `service::tracking` module has nothing specific to the
radicle-node code. This means that it can be moved to radicle under
the `node::tracking` module.
This makes it more resuable in other contexts, for example, for
fetching logic.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
Similar to the `id` function, add a `sigrefs` function to construct
the reference name for a given remote's `rad/sigrefs`.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
Introduce a SignedRefsAt type that combines a SignedRefs along with the Oid
it was loaded at.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
This change allows users to track/untrack peers and repos without a
running node. This fixes a bug when running `rad init` without the node
running, where the repo would be tracked with a different scope than
what was specified on the CLI.
We were using a breadth-first search which doesn't work as-is for
topological ordering in all cases. Instead of using Kahn's algorithm
which is a little complex, we switch to a depth-first search.
The `stable-commit-ids` feature flag was being included in non-dev,
non-test builds because of how cargo works (or doesn't work).
For now, we ignore the tests that this affects, and they can be run with
the `RAD_COMMIT_TIME=1514817556` environment.
This is very much not up to the standards of an official Debian
package. To make a proper package of Radicle will require packaging
all the dependencies first, and the Radicle crates need to be
published to crates.io, among other things. This is just a lazy first
cut to see if there's any interest in a .deb.
Currently we only handle data Uris, but when a user sends a comment that
has a existing git Oid, due to editing a comment we also need to handle
this case.
Signed-off-by: Sebastian Martinez <me@sebastinez.dev>
The `AliasStore` mechanism was added but not used in the `rad node`
command. Since the launched node itself can display aliases based on
the address book, it would make sense if the `rad node` command could
do this as well.
Use the `Profile`'s `AliasStore` implementation to fallback to an
alias if the tracking store does not contain an alias -- essentially
using the address book instead.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
If a `rad` command that requires an RID is executed within a
subdirectory of a Radicle project, then it will fail saying:
✗ Error: Current directory is not a radicle project
Improve this flow by searching parent directories and/or using the
GIT_DIR environment variable.
This is safe as long as the operations are read-only when calling
`cwd`. Currently they are:
- `Repository::open_ext` -- will not create a repository, only open
one.
- `Repository::find_remote` -- this will only read the remote from the
config and get the URL to parse the RID.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
The `rad patch checkout` command would blindly create a branch when
being executed. If there was a branch already existing and user had
committed to that branch, switched without updating the patch, and
then switched back, then they would lose the commit on that branch --
the original commit of the patch would be taken instead.
Fix this by first checking if the branch existed beforehand, and
setting the workspace to use that branch's commit.
Add a test to ensure that this works as expected.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
When specifying a `patch.base` to point to for stacking patches, it's
useful to accept any kind of revspec, e.g. `HEAD^`.
Change the argument parsing to use the `Rev` type instead, and change
the `rad-patch-ahead-behind` to use `HEAD^` to test the example usage.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
`cargo clippy --all --all-targets` is warning that there are multiple
redundant clones.
Apply the suggestions by removing the calls to `clone`.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
When testing the CLI, "since"-calculations will result in different
values over time:
now
1 month ago
2 months ago
10 months ago
1 year ago
10 years ago
Initially, the solution was to use the wildcard match from `snapbox`,
`[..]`, to match on these values. However, since these values change
in length it means that they will affect the width of the tables
produced by the CLI.
To ensure the width is stable across runs over time it's possible to
use a time that is the same as the commit time used at testing.
To do this the `Timestamp` type is now a newtype which has a
constructor `now` which will use the `RAD_COMMIT_TIME` env variable
for construction, if it's available and in debug mode. This ensure
that all "since"-calculations will result in `now`.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
Introduces a new COB type to store repository identity documents.
The reason for this change is to:
1. Simplify the code, as the identity document logic resembled a COB,
yet it had custom logic. This allows existing COB code to be used for
identities.
2. Make identity document update logic more flexible, since COB actions
can be added in the future.
3. Re-purpose existing tools around COBs to work on identities, eg. `rad
cob`.
4. Unify the concept of an identity change proposal, with regular identity
changes. This means we can remove the `id.proposal` COB in favor of
using the `id` COB itself.
Notes
-----
* Each repository has one Identity COB.
* The `Proposal` COB has been repurposed into the `Identity` COB.
* Identity documents are stored as *embeds* inside the Identity COB
actions.
* The action that contains new document versions is called `revision`,
just like the Patches COB.
* The namespaced `rad/id` ref is a symbolic reference to that Identity
COB.
* The canonical `rad/id` ref is a direct reference to the commit in the
Identity COB that contains the latest *accepted* revision of the
document.
* All commands for managing identities have been folded into `rad id`.
Hence `rad delegate` and `rad edit` are removed.
* The concept of "rebasing" an identity document is gone.
* The `rad id` output has been updated to match the style of other
commands.
* When a revision has enough signatures, it is automatically adopted as
the current identity, there is no longer the need for a "commit"
action.
* When an identity revision is proposed, and the current identity has a
threshold of `1`, that identity is automatically accepted due to the
above point.
* The idea of "verifying a peer's identity branch" no longer applies, as
COBs cannot be verified one history at a time.
* Since the root commit of the Identity COB does not have a "Resource"
to point to (it would have to point to itself, which is impossible),
we've made the resource id optional for COBs.
The mock transport was not thread-safe when multiple threads used the
same node ids. This is fixed by including the thread id as part of the
storage key.
Fintan Halpenny
cloudhead
Sebastian Martinez
Rūdolfs Ošiņš
Michalis Zampetakis
Lars Wirzenius