There are two areas where we can be more tolerant of delegate namespaces
being missing or not validating:
1. Calculating the canonical HEAD
2. Fetching from a remote
In 1. the protocol is tolerant in that if the local node does not have
the default branch for a delegate, it will still attempt to use any of
the delegates it does have to reach the threshold.
This is made safe by ensuring that if the threshold is being updated
then the node performing the update must have a threshold of delegates
locally in their storage. It also made safe by 2.
In 2. the protocol is tolerant by allowing delegates to be missing
from the serving side, as long as they can still meet a threshold of
delegates. This is further tolerant, when validating the received
data, a threshold of delegates are valid to consider the fetch
successful -- otherwise it will fail.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
When deciding whether or not to fetch a repo based on a refs
announcement, we need to know whether our signed refs are stale or not,
compared to the ones in the announcement. This can be quite expensive as
it requires multiple reads from Git's ODB, which often results in
packfile loading.
To remedy this, we introduce a refs cache in our node database that is
consulted when an announcement is received.
We also remove the logic that generates `info` messages, since they are
rarely needed and introduced more complexity.
Additionally, we revert commit 3ad2b4431f,
which introduced logic to minimze wants and haves with additional logic,
since it potentially also increases the load on the ODB.
Instead of dropping the existing connection(s), we drop the same
connection on both sides, by using a simple rule that yields the same
result whether the node sees the connection as inbound or outbound.
The trick is to use something that both nodes agree on, for instance
whos public key is greater than the other, and use that information
to close the same connection on both sides.
Promote message logging to `debug` level, and change default level to
`info`.
This is because `trace` level often covers details at a different level
of detail than for eg. messages sent and received. This means we don't
have to use `trace` level just to see the gossip.
Previously, we only kept tracked of unsynced nodes we were connected to.
Now we track all unsynced nodes, since announcements are relayed.
We also cap the replicas to exit earlier in case of very low replica
count.
Remove the borrow of `rid` to fix the clippy warning.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
Instead of re-loading the full inventory on `rad init`, we simply add
the new repository to the in-memory cache.
We also keep track of whether a fetch was a full clone, to know when
to update our routing table.
For large seeds with many repos, it's expensive to traverse the
inventory everytime the node connects to a new peer, for example.
To improve the situation, we cache the inventory per `Storage` instance,
and add a `refresh` method to refresh the inventory.
This method is called when the node is told to synchronize its
inventory.
This change limits the amount of `wants` and `haves` data that is sent
to the serving side of a fetch.
When a `RefsAt` announcement is sent, the fetching peer can calculate
the difference between the `rad/sigrefs` they're aware of -- if it
exists -- and the newly advertised `rad/sigrefs`. This means they can
efficiently ask for the `wants` and `haves` of references that have
changed -- saving some data sent over to the serving side.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
Announcement are logged at the debug level and are very frequent. This
causes a lot of noisy logging in a node.
Downgrade the log levels for these messages from debug to trace.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
For larger fetches, a timeout for the channels and socket control are
required. Bump the timeout from 9s to 30s.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
The timeout was never being used, so it is being removed for the time
being.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
Two concurrent fetches can occur from the same peer, while also being
for a different set of references. This was seen in the
`test_outdated_sigrefs` test. Alice would have an ongoing fetch from
Eve, and concurrently, Eve will create a new issue -- thus creating a
new `RefsAt` message -- and Alice would attempt to fetch this.
The previous logic would only check that Alice is fetching from Eve
and ignore the new updates -- meaning the test could fail. However, if
the `RefsAt` are checked then the other fetch can be recognised as
new, and queue it.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
If a peer deletes a COB, it does not mean that the COB should be
deleted. This was incorrectly assumed in the `radicle-node` fetch
code.
To fix this, when a `RefUpdate::Deleted` is seen after a fetch, the
cache is updated with the new state of the COB.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
Implement caching for the `Issue` and `Patch` COB. This is achieved by
having the `Transaction::initial` and `Transaction::commit` methods
take a cache which can update the newly created/updated object. The
`Store::remove` method also takes the cache for removing the object
from the cache, as well as the repository.
This meant that the `*Mut` types for the respective COBs are required
to carry a cache alongside the repository store. Identities will not
be cached, and so this is always set to `NoCache` -- which performs no
caching, and always succeeds.
To perform cache reading for issues and patches, the `cache::Issues`
and `cache::Patches` traits are introduced. They capture the minimal
amount of methods that are needed for the `rad issue` and `rad patch`
CLI commands.
The implementor for each of these traits is there respective `Cache`
types, which combines their backing repository store and the SQLite
database. They both provide convenience methods:
- `create` (and `draft` for patches)
- `get_mut`
- `remove`
which can be used instead of their repository store counterparts.
All uses of the repository stores (where needed) are replaced with
`Cache` types.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
1. Simplify latency calculation to be more predictable, otherwise,
setting a latency is not super useful.
2. Turn a hard error into a log. It's not worth fixing it now though, as
it isn't a problem for the tests.
cloudhead
Fintan Halpenny
Alexis Sellier