Instead of having a routing database, and an addresses database,
consolidate that in a single database called `node.db`.
We also simplify database access, by providing a single type for opening
the database, and use traits to access the various functions.
We create a new `seed::Store` that has seed-related methods split out of
the `address::Store`.
Note that tracking is still in its own database currently, as it isn't
ephemeral state. This makes it easy to delete the new `node.db` without
risk of losing any important information.
To avoid renaming all database methods, we create a `Stores` wrapper
that allows us to access routing, addresses, seeds etc. without type
ambiguity.
This is a breaking change and requires `addresses.db` and `routing.db`
to be deleted.
* Set peer's `last_active` even if they are being throttled
* Make sure not to break out of loop in wire module
* Add some more logging
* Use different variable for session filtering to make it clearer
Ref announcements contain the namespace (NID) and the SHA of the new
`rad/sigrefs`. This information can be used by the fetch protocol to
fetch only those `Oid`s for those namespaces.
Add a new stage `SigrefsAnnouncement` that captures this behaviour by
only asking for the advertised `Oid`s and updating the corresponding
`rad/sigrefs` for each namespace. Note that this is a special case of
the `SpecialRefs` stage and so the protocol chooses between these two
by checking if there was announcement passed through. The following
stage, `DataRefs`, stays the same.
Note that this stage can only be activated via a `pull`, and `clone`
stays the same.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
Wire up the `radicle-fetch` crate to the fetch logic in the
`radicle-node` worker.
Any required traits for storage, identity, tracking, and transport
used by `radicle-fetch`.
The worker adds a new module to run the `git upload-pack` process and
pipes the `stdout`/`stdin` to the respective channels.
The `Worker` type required adding a `FetchConfig` to easily allow the
configuration of fetches and passing the signer and tracking store
location through to the fetch `Handle`.
The channels code is adapted to have a new writer that always flushes
when a call to `write_all` is made. This is necessary since the
Gitoxide code never calls `flush`, and so it will hang since no data
is sent until `flush` is called.
Since `UserInfo` is used to set up a newly cloned repository, this
changed some of the SHA1 hashes output in the CLI test examples. It
was also necessary to update the CLI's clone code to be deterministic
in its output for tests.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
When a fetch is user-requested, a timeout can be supplied that is passed
down to the worker.
When a fetch is service-requested, a default timeout is used.
This is a breaking protocol change that prefixes all gossip messages
with their length.
This is to allow message extensions, as we'll be able to read the whole
message inclulding the extensions, without having to decode the
extensions.
It also makes it easier to check the message length before we start
decoding, so that we can reject messages that are too big.
It turns out it's possible to create a deadlock when using
`child.wait()` with `Stdio::piped()`, if we don't read the output while
the child process is running. This is because the pipe has an internal
kernel buffer that will eventually fill up, causing the child process to
block on writing to stdout and never exit.
To remedy this situation, we read the stdout in a thread while the child
process is running.
When attempting to connect to a peer, and when succeeding, write the timestamp
to the database.
This will allow us to better select addresses and not retry peers to
soon.
It's currently possible for a channel to be blocked forever while
waiting for data, if the remote peer keeps the connection alive.
This can occupy a worker indefinitely.
With these timeouts, we ensure that workers exit if there is no
activity after a certain amount of time.
Instead of returning the namespaces that we fetched with,
we return the remotes that were actually fetched from.
This can differ if some trusted peers were not available
on the remote node.
Previously, if that was the case, it would cause errors
since the remotes were looked up and that lookup failed.
Instead of parsing the worker requests and responses, which was causing issues
around packfile transmission, we use threads and pass the bytes
transparently.
To improve the reliability and flexibility of the protocol, we introduce
multiplexing over peer connections. This involves a new `Frame` type
that carries a stream-id and payload.
Three stream types are made available:
1. Control
2. Gossip
3. Git
This change brings the following improvements:
* Removed need to queue fetch requests
* Removed need to queue gossip messages
* Removed need for `Fetch` and `FetchOk` messages
* Service doesn't need to know about inbound fetches
* Removed one round-trip for fetch negotiation
* Removed special `done` git packet
* Removed session logic and state around Git/Fetch protocols
* Removed code around upgrading/downgrading transport
* Worker in responder mode is able to process any number of fetches
* Connections support any number of concurrent fetches
We had to introduce a few extra things however to make it all work:
* A `VarInt` type for variable-length integers, since we want the frames
to be lightweight.
* A custom "tunnel" implementation, since we couldn't use the existing
one anymore.
Overall the change removes more complexity than it adds, while improving
the protocol along the way.
For whatever reason, we weren't using the deserializer for message
decoding. This meant that if a message came in two pieces, it would
cause decoding problems, since the first piece would be discarded.
Due to never removing peers from the peer map in the wire
protocol module, it was possible for there to be multiple
connected peers with the same Node ID. Some might be in a
"connected" state, while others might be "disconnected".
This caused a bug where a peer, although connected would
be considered disconnected by the wire module, because
when searching for a peer with that Node ID, the disconnected
one would return first.
To fix this, we make sure to remove peers from the map once
the connection is dropped.
A fetch is a client-server interaction and so there are two sides to
this interaction. These sides are generally referred to as receive --
the client side -- and upload -- the server side.
The `Fetch` type indicated whether it was doing a receive or upload by
using a field `initiated: bool`. However, this can be confusing when
you pair it with the `namespaces: Namespaces` field, since only the
receive side should be sending what refspecs it wants and the upload
side responds with those matching refspecs.
A better way to represent this is using an enum for which direction
the fetch is being considered, where the receive side contains the
Namespaces. To borrow from Noise, the variant names are 'Initiator'
for the client and 'Responder' for the server.
The resulting code then makes decisions based on what variant of the
enum was passed.
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
* Read git request line before starting daemon, so that we don't leave
git data in the socket read buffer in case of an error connecting to
the daemon.
* Send a response on the fetch channel even if we disconnect the remote.
* Don't send `done` packet if there's an error; it won't be expected.
Rust 1.67 was announced[0]. Update the necessary files for running
1.67 and fix the clippy suggestions for string interpolation.
[0]: https://blog.rust-lang.org/2023/01/26/Rust-1.67.0.html
Signed-off-by: Fintan Halpenny <fintan.halpenny@gmail.com>
X-Clacks-Overhead: GNU Terry Pratchett
cloudhead
Fintan Halpenny
Alexis Sellier
xphoniex
Alexis Sellier
Dr Maxim Orlovsky