Subnet splitting

High level overview of subnet splitting steps

The Internet Computer Protocol (ICP) now supports a minimal viable product (MVP) version of subnet splitting. Subnet splitting is a process, where a subset of the canisters from the parent subnet A are split off onto a newly created child subnet B for load balancing purposes and the remaining canisters stay on the trimmed down subnet A’. The MVP process is orchestrated by a series of NNS proposals and, very roughly speaking, consists of the steps visualized on the right.

Background

To understand all the details of subnet splitting we need to take a closer look at some parts of the ICP.

Checkpoints and Manifests

Up to date nodes persist their state to disk every couple hundreds of rounds (usually 500). A state that is persisted to disk by the protocol is called a checkpoint. Each checkpoint is a directory with the following structure (note that not all of the files shown below are necessarily present in every checkpoint, e.g., writing empty files may be skipped).

├── canister_states
│   ├── <hex(canister_id)>
│   │  	├── canister.pbuf
│   │	├── queues.pbuf
│   │   ├── software.wasm
│   │   ├── stable_memory.bin
│   │	└── vmemory_0.bin
│   ...
├── ingress_history.pbuf
├── split_from.pbuf
├── subnet_queues.pbuf
└── system_metadata.pbuf

For example, the state of each canister is stored in a separate directory. In contrast, there’s a single file containing the current ingress history (request statuses) of all canisters.

For each checkpoint the protocol will also compute a so-called manifest. Such a manifest consists of the individual hashes of every file and file chunk (the actual contents of files are chunked so that the chunks can be individually transmitted) and a root hash computed from them. The root hash has the property that it is intractable to come up with two different states that hash to the same root hash and can hence be used to verify the integrity of a certain state relative to a root hash. A similar property holds for the file and chunk hashes in the manifest. However, they are redundant for guaranteeing the integrity of the entire state, as they are also covered by the root hash. The reason why they are included in the manifest nevertheless is to allow for efficiently computing state diffs by comparing file/chunk hashes.

CUPs

Every checkpoint will also be signed on behalf of the subnet by the protocol. This is done by creating and threshold signing a so-called catch up package (CUP). A CUP includes the root hash of the corresponding manifest as well as all the information that is required by a newly joining and/or behind node to resume execution from the respective checkpoint. The signature on the CUP together with the properties of the hash described above guarantees the authenticity of the state.

Remarks

Note that when we point to source code within this article we use links to a specific version of the code to make sure that these links don’t break if the code changes in the future. It is advisable to locate the code that is pointed to in the most recent version and/or the version that is used to verify the artifacts and look at it in this version to avoid missing changes that happened after the publication of this article.

Verification of Artifacts in Subnet Splitting

As mentioned before, the process is orchestrated via NNS proposals and it will be end-to-end verifiable, in the sense that the community will be able to verify whether the states recovered onto the new subnets are indeed the two halves of the initial state before voting on the respective proposals. A description on how to perform this verification is provided on the Verification of Artifacts in Subnet Splitting page.