Instead of returning the namespaces that we fetched with,
we return the remotes that were actually fetched from.
This can differ if some trusted peers were not available
on the remote node.
Previously, if that was the case, it would cause errors
since the remotes were looked up and that lookup failed.
With this change, we not only verify the signed refs before
the repository "transfer", but also validate that all refs in the
repo are signed and there is no discrepancy with the signed refs
file.
If `git-fetch` returns a non-zero code on exit, we keep going in the
case of an existing repository.
The reason is that `git-fetch` will return an error code if any ref
is rejected during the fetch. But this can happen if someone
pushes conflicting `rad/sigrefs` branches for example, and the node
already has a copy of that branch.
This allows change commits to have more than one parent that isn't
part of the DAG.
We can use this to create dependencies towards git objects such as
commits, in the case of patch actions.
Instead of parsing the worker requests and responses, which was causing issues
around packfile transmission, we use threads and pass the bytes
transparently.
Previously we were announcing inventories all the time, due to
relying on the timestamp changing on insertion.
We change this to only announce our inventory when it actually changes.
Previously, during the "transfer" stage of a fetch, we used `force:
false`, which didn't allow refs to be force-updated.
This change makes it possible to fetch force-updated refs, but ensures
that the sigrefs branch is never force-updated.
To improve the reliability and flexibility of the protocol, we introduce
multiplexing over peer connections. This involves a new `Frame` type
that carries a stream-id and payload.
Three stream types are made available:
1. Control
2. Gossip
3. Git
This change brings the following improvements:
* Removed need to queue fetch requests
* Removed need to queue gossip messages
* Removed need for `Fetch` and `FetchOk` messages
* Service doesn't need to know about inbound fetches
* Removed one round-trip for fetch negotiation
* Removed special `done` git packet
* Removed session logic and state around Git/Fetch protocols
* Removed code around upgrading/downgrading transport
* Worker in responder mode is able to process any number of fetches
* Connections support any number of concurrent fetches
We had to introduce a few extra things however to make it all work:
* A `VarInt` type for variable-length integers, since we want the frames
to be lightweight.
* A custom "tunnel" implementation, since we couldn't use the existing
one anymore.
Overall the change removes more complexity than it adds, while improving
the protocol along the way.
Erik Kundt
Alexis Sellier