radicle-heartwood-lfs/simulation

Simulation Environment

A suite of tools to create simulated Radicle networks to run tests in.

This environment provisions a Kubernetes cluster, deploys a configurable topology of radicle-node instances, and provides a foundation for running cross-version, cross-platform, and adverse network tests.

Prerequisites

To run the simulation environment, you need the following tools installed on your system:

• just: A command runner (replaces make).
• talosctl: CLI for creating and managing Talos Linux clusters.
• kubectl: CLI for interacting with Kubernetes.
• timoni: A package manager for Kubernetes, powered by CUE.
• cue: (Optional) Useful for debugging and formatting CUE files.
• QEMU or Docker: Required by Talos to provision the local cluster nodes. (Defaults to qemu).

NixOS

On NixOS it is necessary to enable the following in your configuration.nix:

# Kernel modules for QEMU/talosctl networking
boot.kernelModules = ["kvm-amd" "tun" "bridge" "veth"]; # or kvm-intel
# /dev/kvm access
virtualisation.libvirtd.enable = true; # or just ensure kvm group access
# Allow talos as a trusted firewall interface
networking.firewall.trustedInterfaces = ["talos+"];

Getting Started

The environment is managed entirely via just. From the simulation directory, you can run:

# Start the complete simulation (creates cluster, configures K8s, and deploys the network)
$ just start

# Note: To use Docker instead of QEMU, override the provisioner:
$ PROVISIONER=docker just start

# Inspect the cluster and see running pods
$ just show-cluster

# Tear down the network workloads (deletes pods and storage, keeps the cluster running)
$ just delete

# Destroy the entire Talos cluster and clean up your kubeconfig
$ just destroy

Run just by itself to see a list of all available commands.

Architecture Overview

Here is how the different tools interact to build the simulation:

1. Just (justfile): Acts as the orchestrator. It runs the bash scripts required to bootstrap the environment, verify tools are installed, and execute the correct CLI commands in sequence.
2. Talos (talosctl): A lightweight, immutable Linux operating system built specifically to run Kubernetes. just uses talosctl to spin up a local K8s cluster inside QEMU or Docker.
3. Kubernetes (kubectl): The orchestrator that runs the Radicle nodes in isolated pods. It manages their networking (DNS resolution between nodes) and persistent storage.
4. Timoni & CUE: The configuration engine. Instead of writing verbose YAML, we use CUE files to define network topologies. Timoni reads these CUE files, transpiles them into Kubernetes object definitions (StatefulSets, Services, ConfigMaps), and applies them to the cluster.

Defining a Topology

Network topologies are defined in instances/network.cue. This file dictates how many nodes exist, what roles they play (e.g., bootstrap, peer), and how they connect to each other.

Here is an example of how the topology is structured:

package main

// …

// Declare instances to deploy
values: {
	topology: {
		// A bootstrap node
		"bootstrap-v1-8-0": {
			role:          "bootstrap"
			version:       "1.8.0"
			replicas:      1
			nodeIdSeed:    "bootstrap-0" // Deterministically generates the NID above
			radicleConfig: #BaseBootstrapSeedConfig
		}
		
		// A peer node that connects to the bootstrap node
		"seed-v1-8-0": {
			role:          "seed"
			version:       "1.8.0"
			replicas:      1
			radicleConfig: #BasePeerConfig & {
				preferredSeeds: [
					// Uses a helper to format the K8s internal DNS address
					(#SeedAddress & {nid: #BootstrapNIDs["bootstrap-0"], name: "bootstrap-v1-8-0"}).out,
				]
			}
		}
	}
}

When you run just start-network, Timoni reads this file, merges it with the module definitions in modules/radicle-node, and deploys the resulting pods to Kubernetes.

Helpful Commands

Execute a command inside a node:

$ kubectl exec peer-v1-8-0-0 -c node -- rad self

$ kubectl exec peer-v1-8-0-0 -it -c node -- sh

Follow Radicle node events (from the sidecar):

$ kubectl logs peer-v1-8-0-0 -c events -f

View standard node logs:

$ kubectl logs -f bootstrap-v1-8-0-0 -c node

Describe a pod (Useful for debugging CrashLoopBackOff errors):

$ kubectl describe pod bootstrap-v1-8-0-0

Watch all cluster events in real-time:

$ kubectl get events --watch

Deterministic Keys

To ensure nodes can reliably connect to each other across restarts, the bootstrap nodes are configured with deterministic Node IDs (NIDs).

bootstrap-0: did:key:z6MkhJ3cwzpAoNjFnJXWETSPHcDyw2HuBVEhgkyTfbjQHY1B bootstrap-1: did:key:z6MkjcaeSHhQVJU1UeXpnHHZ6mp67zDfQYNMDotHGxbrk7Nj bootstrap-2: did:key:z6MkjNGhuJvdp2noidRMLqco4jFnNNSWzCxSZH5nJV1pGrwQ bootstrap-3: did:key:z6MkpEsXUMSnmyfwdEVkAKijTxGy9WKmNoHWpoxxLM6bbz9M

Why?

heartwood already has the following types of tests (as of 2026-04):

• Unit
• End-to-End
• CLI

However we can only run them on the currently checked out version of heartwood, this leaves gaps in our testing coverage, particularly for cross-version and cross-platform testing.

The simulation environment is intended to remedy these gaps and more. See the [Goals] section for more info.

Constraints

Non-Goals:

• Replace existing unit, CLI and end-to-end tests.
• Deterministic execution; Not creating an alternative madsim or shadow.
• Writing custom orchestration code.
• Hard to reason about.

Goals:

• Isolation between simulations and main network.
• Different node versions within a simulation.
• Cross platform (Windows, Linux & macOS).
• Realistic load generation.
• Invariant assertion across simulation network.
• Teardown and Artifact collection.
• Continuous set of running simulations.
• Realtime Observability.
• CI/CD Integration.
• Cross simulation comparative insights e.g. CPU pressure change from version A to version B.
• Flexibility to define network topologies.
• Easy to construct and run new simulations.
• Reproducible starting state.
• Adverse network emulation e.g. dropped packets, network delays…

Plan

• Migrate existing simulation environment repo into heartwood.
  1. radicle-node timoni module.
  2. radicle-node custom container builder.
  3. instances topology definition files.
  4. sim-tests rust crate.
  5. justfile orchestration.
  6. observability definition files.