In issue #[2836] of `libgit2`, the speed of `libgit2`'s `file://` protocol for
git operations is found to be very slow. This is further corrobarated by `rad
init` taking 13 hours to initialise the hardenedbsd [ports] repository.
There are two areas where the `heartwood` project uses `libgit2` to push using
the file protocol. The first is when via the `trasnport::local` smart transport
registration and the second is the final push to storage in the `git-remote-rad`
binary.
When both these `push` operations are changed to use the `git` binary instead,
the [ports] repository can be initialised in less than 10 minutes (nearly 100x
speed up).
This change is clearly required if `heartwood` wishes to support larger
repositories.
[2836]: https://github.com/libgit2/libgit2/issues/2836
[ports]: https://git.hardenedbsd.org/hardenedbsd/ports
Adds the new `migrate` sub-command to migrate the COB database.
Checks for version when opening the COBs database and suggests the
command on out of date version.
Also adds the 2nd migration to the COBs cache, which updates the patch
JSON schema.
The `no quorum found` error can be opaque and lead to a lot of confusion. It is
better to give more information for this error so that it leads to easier
debugging, since it can be quite a common warning/error.
Instead of a single error, there are two cases provided: `NoCandidates` and
`Diverging`, to capture the two different variants of errors that can occur.
`Display` implementations are provided for both to provide more information.
The aim of this change is to make the `Doc` type more safe to use by approaching
the design via [Parse don't validate][[0]] approach.
The problem with the previous approach was that all field were `pub` and thus a
`Doc<Verified>` could easily be mutated and serialized. Granted, the code that
used the serialization would tend to verify the `Doc` first, however, this
approach *ensures* that only a verified `Doc` can be serialized. It also meant
that trying to add new data that would follow the parse approach would require
more generic parameters on top of the existing `PhantomData` parameter, i.e. we
need to do something like: `Doc<RawField, V> -> Doc<ValidField, V>`.
The new approach splits the type into two separate types: `RawDoc` and `Doc`.
The former is allowed to be mutated at will, and uses types that are less
strict. The latter is the valid type that can only be constructed by validating
a `RawDoc` (or the `initial` constructor). The `Doc` type's fields can then only
be accessed by read-only methods.
Solves the problems above by only allowing mutations to `RawDoc`, as well as,
new fields being added to `RawDoc` which are then validated via
`RawDoc::verified`.
[0]: https://lexi-lambda.github.io/blog/2019/11/05/parse-don-t-validate/
When creating a new upstream for the patch workflow, the remote name is always
assumed to be `rad`.
It's possible for users to use another remote name but still use a `rad://` URL
for pushing. To allow for this, the function that creates the upstream entry now
takes a remote name. In the remote helper, `rad patch checkout`, and `rad patch
set` the remote can be optionally specified and fall back to the `rad` remote
name as a default.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
When pushing to the default branch and updating the canonical head,
check to see if any merged patches are reverted due to this change.
If so, make sure the COB cache reflects this.
The rules for checking the quorum of the remote helper, when pushing to the
canonical branch, can prevent the operator from even attempting to try converge
to a state of agreeance.
Instead, the checking of the quorum rules are relaxed. It is still the case that
if the peer is pushing a diverging commit while in a working state they are
warned and the push does not go through. However, there are two cases where it
is fine to force push a change:
1. When pushing a new tip, as a delegate, there is still a canonical tip and the
new tip still converges with the history, i.e. it is equal to, behind, or
ahead of another delegate tip.
2. If there is no quorum in the current state, for example there were multiple
concurrent pushes, then the peer is free to continue to push to their default
branch in attempt to converge with the other peers.
The example tests ensure that these cases are covered, while also preserving the
old behaviour.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
Remove the old `quorum` function in favour of using the newly introduced
`Canonical` code. Port over the tests to use the `Canonical` type. The change is
minimised by introducing a helper function called `quorum` in the test module
that acts like the old function.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
* Try to use constants instead of strings
* Move crypto seed override out of radicle-crypto
* Use GIT_COMMITTER_DATE for commit overrides
* Use RAD_LOCAL_TIME as general time override
* Allow variables to be used in release mode
There are two areas where we can be more tolerant of delegate namespaces
being missing or not validating:
1. Calculating the canonical HEAD
2. Fetching from a remote
In 1. the protocol is tolerant in that if the local node does not have
the default branch for a delegate, it will still attempt to use any of
the delegates it does have to reach the threshold.
This is made safe by ensuring that if the threshold is being updated
then the node performing the update must have a threshold of delegates
locally in their storage. It also made safe by 2.
In 2. the protocol is tolerant by allowing delegates to be missing
from the serving side, as long as they can still meet a threshold of
delegates. This is further tolerant, when validating the received
data, a threshold of delegates are valid to consider the fetch
successful -- otherwise it will fail.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
Implement caching for the `Issue` and `Patch` COB. This is achieved by
having the `Transaction::initial` and `Transaction::commit` methods
take a cache which can update the newly created/updated object. The
`Store::remove` method also takes the cache for removing the object
from the cache, as well as the repository.
This meant that the `*Mut` types for the respective COBs are required
to carry a cache alongside the repository store. Identities will not
be cached, and so this is always set to `NoCache` -- which performs no
caching, and always succeeds.
To perform cache reading for issues and patches, the `cache::Issues`
and `cache::Patches` traits are introduced. They capture the minimal
amount of methods that are needed for the `rad issue` and `rad patch`
CLI commands.
The implementor for each of these traits is there respective `Cache`
types, which combines their backing repository store and the SQLite
database. They both provide convenience methods:
- `create` (and `draft` for patches)
- `get_mut`
- `remove`
which can be used instead of their repository store counterparts.
All uses of the repository stores (where needed) are replaced with
`Cache` types.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
Shows you updates on local repositories.
Included here is a basic implementation of a notification system. A new
database is created and updated after every fetch, with refs that have
changed.
The `rad inbox` command can then list items in the database and delete
them.
To allow `rad sync` to break early from the call to `announce`,
control flow is introduced for the callback.
In `rad sync`, the number of synced nodes is checked against the
`replicas` -- if specified -- and sends a `Break`.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
Shell out to `git`, avoiding using `git2`. This is to avoid an
issue where somewhere within the fetch there is an attempt to lookup a
`rad/sigrefs` object, which says that the object is missing. It is
suspected that this is due to the object being localised in the same
packfile as other objects we are fetching. Since the `rad/sigrefs`
object is never needed nor used in the working copy, this will always
result in the object missing. This seems to only be an issue with
`libgit2`/`git2` and not `git` itself.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
The hint for updating a patch incorrectly printed:
git push rad -f HEAD:rad/patches/<patch id>
The remote, `rad`, is already specified so the the right hand side of
the refspec should only be `patches/<patch id>`.
Fix this by stripping the remote from the name of the branch in the
remote helper.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
It's actually not such a great idea to sync on git-fetch and
git-ls-remote, as syncing is supposed to happen in the background
already, and if you want to force a sync, you can just call `rad sync`.
Previously, it would be annoying that when you called `rad sync` and
then `git fetch`, it would still attempt another sync, for example.
Moving forwards, it might make sense to call `git-fetch` on `rad sync`
(so the other way around), because `git-fetch` is free (it doesn't talk
to peers), and you almost always want your working copy up to date with
your storage.
It's not currently possible to force-push rad/master if it's not a
rollback, even if you are the only delegate. Since there is no risk
in doing that, we allow it in this patch.
Update to git2-0.18.1, which also requires updating radicle-git-ext
and radicle-surf.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
Lorenz Leutgeb
Fintan Halpenny
cloudhead
Sebastian Martinez
Alexis Sellier