This fixes an issue that came up in tests where we would be reading
and writing to the database at the same time, and it would fail with
an error due to the lock being contended on.
The One variant caused many paint points for fetching logic, where it
was not necessary. It was only constructed in one place, which could
be replaced by using the variant that holds a set of keys.
Remove the Namespaces::One variant and rename Namespaces::Many to
Namespaces::Trusted.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
When a repository does not yet exist during a fetch, i.e. a clone,
then only delegates are being fetched.
Augment Namespaces::Many to hold the trusted and optional delegate peers
separately. Doing so allows the construction of the variant without
the repository existing, but tracking relationships existing.
This variant can then be used to build refspecs for both the trusted
peers and delegates when doing a cloning fetch.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
Previously, if a peer created a project which was replicated and then
moved to another setup, ultimately deleting that project, then they
would not be able to fetch it again since their remote namespace would
be excluded from the fetch.
This test confirms that a peer can remove their project and clone it
again from a peer that has replicated the project. This is simulated
by directly removing the project directory in the test and fetching
the project from the other peer.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
With the latest fetch updates, the test for fetching trusted remotes
can be enabled.
The caveat is that its expected behaviour is to fetch the delegate
upon a clone and fetch all trusted peers on a subsequent
fetch. Fetching all trusted peers is left as a follow-up task.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
The previous fetching logic had some pitfalls regarding its logic:
* Fetched 'rad' references could end up not being verifiable and leaving
the storage in a corrupt state
* Forced pushes would be prohibited, since they would be rejected by
the client side due to the refspecs being used.
The new approach is to stage a fetch in 2 steps. The first step is to
fetch the relevant `rad/id` and `rad/sigrefs` for verification. For
all remotes that are verified, fetch all references listed in their
respective `rad/sigrefs`, using the `+` marker in the refspec,
allowing to update for any force pushes -- note that this becomes a
safe operation since the `sigrefs` are signed by the remote that
created them.
To prevent unverifiable 'rad' references from polluting storage, all
fetch negotiations are done in a staging repository set up in a
temporary directory. All successful fetches are then transferred,
using the 'file://' protocol, from the temporary directory into the
radicle storage.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
The tracked nodes include nodes marked tracked in the db, and
the delegates and `Namespaces::All` if it is a new repo.
Signed-off-by: Han Xu <keepsimple@gmail.com>
First, we make sure to always attempt reconnection to persistent peers,
no matter the reason.
Second, we don't re-attempt connections when an inbound peer
disconnects. This minimizes the chance of connections crossing.
Third, we clean up `session::Error`, and don't use it to signal a
missing session, since it doesn't really belong there.
1. We were announcing inventory every 30 seconds. This is now changed
to 1 hour.
2. We were using a variable to track whether our inventory has been
announced to the network, but this var was set based on what we
tracked, not the inventory we had. This var has been removed.
Better handle case where the connection to the daemon fails, in the
responder scenario.
Instead of trying to read the `done` packet once, we loop until we've
read it. This handles a scenario where the uploader fails before the
git protocol is initiated, and the fetcher is still sending git packets.
This change ensures that on `Subscribe`, we only relay the latest
announcement of each node, per announcement category.
Previously, we would relay all announcement, while only the last one
was relevant.
If there is an unexpected error, eg. `BrokenPipe` in the upload process
process of the worker, we may have more than four bytes to read from the
remote socket. This change ensures that we don't attempt to read more
into a buffer of size `4`.
For whatever reason, we weren't using the deserializer for message
decoding. This meant that if a message came in two pieces, it would
cause decoding problems, since the first piece would be discarded.
It's possible that the daemon or stream closes the git connection
without reading the final `done` packet. If this is the case, we
try to read it after either of the connections is closed.
This may potentially mitigate the issue where the node then receives
the `done` packet while in gossip mode, and disconnects the remote
for misbehavior.
Due to never removing peers from the peer map in the wire
protocol module, it was possible for there to be multiple
connected peers with the same Node ID. Some might be in a
"connected" state, while others might be "disconnected".
This caused a bug where a peer, although connected would
be considered disconnected by the wire module, because
when searching for a peer with that Node ID, the disconnected
one would return first.
To fix this, we make sure to remove peers from the map once
the connection is dropped.
It's not a good idea to have fallible operations run after `FetchOk`
is received, since there's no easy way to communicate the error
to the remote at that point. What would then happen is that the fetch
just "hangs", since nothing is there to cancel it.
Hence, we compute namespaces earlier in the process, and store the
namespaces in the session. If there is a failure, the fetch is
never initiated.
It was possible to trigger multiple connection attempts to the same
peer, since we didn't track connection attempts that hadn't yet
returned to the service via `Service::attempted`.
With this change, we add an extra `Initial` state to sessions, so
that redundant connections are not attempted when one is already under
way.
The radicle-node was forced into using Scope::Trusted for its default
scope.
Allow the Scope to be specified at the command line using a
`--tracking-scope` option.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
Periodically, make sure we try to fetch inventory that is tracked
but missing from local storage.
We decide not to fetch immediately on `TrackRepo`, as this limits
the command's flexibility. For example, when `rad clone` is used,
we want to be able to fetch manually after tracking.
This avoids race conditions, especially in tests, where we try to
connect to the node right after spawning it, but since the
`Runtime::run` function is often called in a different thread, the
socket is not created in time.
It's possible for a peer to disconnect before responding with `FetchOk`.
In that case, we make sure to return to any pending fetch request with
an error.
To ensure the behaviour of ignoring references that are owned by the
local peer a test case is added.
The case has two peers: alice and bob, where alice creates a new
repository. bob fetches to be up-to-date with alice. alice, in turn,
creates an issue and fetches from bob -- which should be successful.
If the refspec:
^refs/namespaces/<alice>/*
was not in place, her fetch from bob would fail since the signed
references would not be a fast-forward.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
Adds a test for fetching from trusted delegates.
Currently marked as failing. When the fetch is initiated from a client
that does not have the repository then it will fetch all namespaces,
instead of the trusted set.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
To determine the correct Namespaces to use, the repo Policy and Scope
are consulted.
If the policy is Block then this is a failure state and is reported to
the other end.
Otherwise, if the policy is Track, then the scope is checked. For the
Trusted Scope then the Namespaces should consist of all tracked nodes
and delegates for the given repository. However, the repository might
not exist and so it is necessary for the whole repository to be
fetched.
For the All Scope, all Namespaces are fetched.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
Alexis Sellier
Fintan Halpenny
Alexis Sellier
Han Xu
Adam Szkoda
xphoniex