Verification of Artifacts in Subnet Splitting
This page describes how to verify the artifacts during the subnet splitting process. Note that when we point to source code within this article we use links to a specific version of the code to make sure that these links don't break if the code changes in the future. It is advisable to locate the code that is pointed to in the most recent version and/or the version that is used to verify the artifacts and look at it in this version to avoid missing changes that happened after the publication of this article.
The first step is to obtain a vanilla clone of the IC repository and check out the commit corresponding to the version currently rolled out on mainnet. Once this is done one can enter the dev container via ./gitlab-ci/container/container-run.sh. From within the container one should be able to run the commands as provided below.
Splitting the State
The subnet state is split in such a way that the content of most of the individual checkpoint files does not change and so the file and chunk hashes in the manifest can be directly compared to the corresponding ones in the manifest on the initial parent subnet. For example, a canister is retained without modification on exactly one of the resulting subnets. There are a handful of subnet (as opposed to canister) state files where this is not possible, e.g., subnet queues. For these files it is the case, however, that they will remain on the source subnet in their entirety and the split off subnet will get new files, initialized based on the parameters passed to the splitting logic. So the expected hashes of these new files are also fully determined.
The ingress history is a special case: because ingress messages are essentially tied to their respective target canisters both child subnets need parts of the ingress history. To avoid breaking end to end verifiability here, the full ingress history of the parent subnet will be preserved on both child subnets, keeping the hash of the ingress history file the same on the parent and both child subnets. The unnecessary entries are then pruned in the beginning of the first execution round of each of the child subnets.
Subnet Splitting Artifacts
The artifacts needed to verify a subnet split are as listed below. Artifacts 2-5 will be published (presumably on the forum) during a subnet split so that the community can factor them into their decision on whether to accept the related proposals.
- The NNS public key. This key serves as the root of trust of the validation and will not be distributed as part of the artifacts. We recommend that everyone who wants to verify the artifacts obtains this key out of band. For example the agents and the rosetta node have the mainnet NNS public key baked in and one can obtain them from the respective repositories on GitHub.
- A certificate from the NNS that includes the public key of the subnet to be split. This will allow to verify the authenticity of the subnet public key relative to the NNS public key.
- The subnet ID of the subnet to be split. If the certificate from the previous step is deemed valid relative to the NNS public key, this ID can be used to extract the subnet’s public key from the certificate.
- The CUP of the subnet to be split at the height where it was halted. The CUP signature can be verified using the subnet public key from the previous step and, if verification succeeds, the authentic root hash of the state can be extracted.
- A textual representation of the manifest of the subnet to be split at the height where it was halted. One can recompute the root hash from the file and chunk hashes in the textual representation of the manifest using the state tool. If the recomputed root hash matches the one extracted from the CUP in the previous step, one knows that the file and chunk hashes in the manifest must be authentic and can hence be used as a basis of comparison, without knowing the actual file contents, i.e., just based on the hashes.
The verification procedure of the artifacts can be triggered via the validate subcommand of the subnet splitting tool. We have prepared demo artifacts we obtained from a split that was carried out on a testnet. They can be downloaded here. In the commands below we assume that you have the IC repository at git commit 02b79bcf3cefbb86b5252a2576ccb4dd7e0a9090 checked out and unzipped the artifacts archive into the /tmp/splitting-artifacts directory.
bazel run //rs/recovery/subnet_splitting:subnet-splitting-tool \
--config local \
-- validate \
--nns-public-key-path /tmp/splitting-artifacts/demo_nns_public_key.pem \
--state-tree-path /tmp/splitting-artifacts/demo_state_tree.cbor \
--source-subnet-id gmmtp-v5kpc-wdkrx-pni4q-6qwld-ntwrv-kqxva-c3rnm-o6lx3-2eg74-oae \
--cup-path /tmp/splitting-artifacts/demo_CUP.pbuf \
--state-manifest-path /tmp/splitting-artifacts/demo_source_manifest.txt
This is the expected output upon a successful validation.
Aug 31 08:15:05.954 INFO Validating State Tree signed by the NNS
Aug 31 08:15:05.954 INFO Reading the NNS public key from /tmp/splitting-artifacts/demo_nns_public_key.pem
Aug 31 08:15:05.957 INFO Validation succeeded: extracted authentic subnet key from the NNS state tree.
Aug 31 08:15:05.957 INFO
Aug 31 08:15:05.957 INFO Validating Source Subnet's original CUP
Aug 31 08:15:05.960 INFO Dealer subnet from the CUP: gmmtp-v5kpc-wdkrx-pni4q-6qwld-ntwrv-kqxva-c3rnm-o6lx3-2eg74-oae
Aug 31 08:15:05.960 INFO CUP height: 82600
Aug 31 08:15:05.960 INFO Block time from the CUP: 2023-08-22 09:23:29.206708237 UTC (nanos since unix epoch: 1692696209206708237)
Aug 31 08:15:05.960 INFO State hash from the CUP: 46fbb99db26d751c5b2b9f35778d5800bae74566f968388fbb0b36226a488416
Aug 31 08:15:05.967 INFO Validation succeeded: source subnet CUP signature is valid.
Aug 31 08:15:05.967 INFO
Aug 31 08:15:05.967 INFO Validating Source Subnet's original state manifest
Aug 31 08:15:05.967 INFO state hash from the CUP: 46fbb99db26d751c5b2b9f35778d5800bae74566f968388fbb0b36226a488416
Aug 31 08:15:05.967 INFO state hash from the State Manifest: 46fbb99db26d751c5b2b9f35778d5800bae74566f968388fbb0b36226a488416
Aug 31 08:15:05.967 INFO Validation succeeded: recomputed manifest root hash matches the one in the CUP.
Aug 31 08:15:05.967 INFO
Computing the expected manifests and root hashes
If the verification above succeeds, one can split the authentic manifest of the parent subnet to compute the expected manifests for the child subnets. Before we start with the actual procedure, let’s look at the example manifest of the source subnet in our testnet setting.
MANIFEST VERSION: V3
FILE TABLE
idx | size | hash | path
------------+------------+------------------------------------------------------------------+------------------------------------------------------
0 | 216 | 8feafcda70181ac8920dfeba5db951f931f87758ecd5dcd7c920a8468a5bacf9 | canister_states/00000000003000000101/canister.pbuf
1 | 532 | cfa0f9a9f833b072aa0c8d1f6eb8bb9334a36168379eaa0925d9013b2d646bbb | canister_states/00000000003000010101/canister.pbuf
2 | 459789 | 4840e8e6e379f2108cec9de7ef386ea5f927f687b121b90ae765ad144bc0222a | canister_states/00000000003000010101/software.wasm
3 | 0 | 981305d7c0b2ace0f53fe822f4075278fa28511e8c34e70f37fd8425af659b36 | canister_states/00000000003000010101/stable_memory.bin
4 | 1093632 | 1ce90b6ed4c455e1c51c4c2ff466d3c6d49c76a585a014d61eebae6c09b0980b | canister_states/00000000003000010101/vmemory_0.bin
5 | 4721 | 346f8b36205998ef3c3c70d5ef64f27096714c83925d98f9e7daf93bf71bd66b | canister_states/00000000003000020101/canister.pbuf
6 | 324196 | 73b2c88c9db040794b23535ecd57fa14528d31ada4d9a82fdb6e9cd21cc0c1e6 | canister_states/00000000003000020101/software.wasm
7 | 0 | 981305d7c0b2ace0f53fe822f4075278fa28511e8c34e70f37fd8425af659b36 | canister_states/00000000003000020101/stable_memory.bin
8 | 1245184 | 0779cd1b7b5afcac3a2e19070abab88bd637261965c36dd1334928980e7a3744 | canister_states/00000000003000020101/vmemory_0.bin
9 | 749 | 493d2f01070bccb63bf75764bcdbc42b1a6946e191029afd74353f339939f51e | canister_states/00000000003000030101/canister.pbuf
10 | 968242 | 912f20ef0b5769a55316b7463cc1d55f079a2e91d1d358a5524a28c0782e82b1 | canister_states/00000000003000030101/software.wasm
11 | 0 | 981305d7c0b2ace0f53fe822f4075278fa28511e8c34e70f37fd8425af659b36 | canister_states/00000000003000030101/stable_memory.bin
12 | 1458176 | 26f9c82ae827458e84befaa73c42f0cbcf39a2c9dae0c777aa26e7fd6a75db62 | canister_states/00000000003000030101/vmemory_0.bin
13 | 2418 | b18a59a56442ddd68d9fcc6fb0103f12c0c0bfd0d01c473711d53b2aff18d596 | system_metadata.pbuf
CHUNK TABLE
idx | file_idx | offset | size | hash
------------+------------+------------+------------+------------------------------------------------------------------
0 | 0 | 0 | 216 | 0c6f866e1b60969cc48b4015bb382d45c282b46a82dd24769a15e9628e4d2f62
1 | 1 | 0 | 532 | 3b53a0d9cb9e0131a4677d86c237ed06a1aadaf46c480e448a8d78afe9c43635
2 | 2 | 0 | 459789 | c470eb0871e137fccfb419b16e2826a63c2a1dba913f5c260ab22ee2958afd28
3 | 4 | 0 | 1048576 | de5005242d69024d356dd4d03d8a6879a2b8aadfff6e0f5f80900d45d363305e
4 | 4 | 1048576 | 45056 | 1ab510085a2a3090dd1d49c578fceabd56a6433bd75e7e2ecc3107c8f8f208eb
5 | 5 | 0 | 4721 | 81af05e10350340c340db100e2b4c1dddc3a7f800ea573ddcef28d562fa70a10
6 | 6 | 0 | 324196 | accdd1e66c56db5b5ff0ac24182a10f65559cfd17cc52d86ac3aa6cae5d567b4
7 | 8 | 0 | 1048576 | a5d9f9ce12da611d7b170fc3156f2dc1255ee11fb6525ddc4dbc3cb7c5f488e5
8 | 8 | 1048576 | 196608 | 9f0bd49a6523364cbc57561e45d67597ef66e29c3fb36273465efd4d210872e8
9 | 9 | 0 | 749 | c0e5fae6857b04d035f8fd752505da9f3905514c4d8dcec7650a18e43af61fa5
10 | 10 | 0 | 968242 | f431111e6b9a9a074daf350cba6404abc47a993f7ddca098de1819c62d99a515
11 | 12 | 0 | 1048576 | de5005242d69024d356dd4d03d8a6879a2b8aadfff6e0f5f80900d45d363305e
12 | 12 | 1048576 | 409600 | 93207dec95a267dab702bb1991af9dd9c5df72ba9d87188d37772282d3948632
13 | 13 | 0 | 2418 | 679b9963d906d7ecbf818b3e73d8654bd4944d7bd35d6c7c4fde7223bd8d664b
ROOT HASH: 46fbb99db26d751c5b2b9f35778d5800bae74566f968388fbb0b36226a488416
The manifest consists of a file table listing all files and their hashes; and a chunk table enumerating the hashes of the chunks making up the individual files. Concretely, our state contains 4 canisters with IDs
fs35c-jyaaa-aaaab-qaaaa-cai(00000000003000000101),fv23w-eaaaa-aaaab-qaaaq-cai(00000000003000010101),f4zqk-siaaa-aaaab-qaaba-cai(00000000003000020101) andf3yw6-7qaaa-aaaab-qaabq-cai(00000000003000030101).
As we can see in the manifest, the first canister doesn’t have a WASM installed and therefore only has a canister.pbuf file, while others do and also have all the other files that are specific to canisters with installed WASMs.
Finally there is also a root hash that covers all the files and chunks (via their hashes). For a correct set of artifacts the root hash should match the root hash in the CUP printed by the verification tool above. As we can see, we have a root hash match for our sample artifacts. More details about how the state is hashed to obtain the manifest can be found here.
Now, recall that the hashes in the parent manifest and the subnet splitting parameters fully determine the manifests of the child subnets. This means that one can use the state tool to compute the expected manifests and root hashes for the child subnets.
To continue our example, let us assume we want to split off the last two canisters onto a subnet with the following ID:
ykq2b-vnsfx-dwzwl-tvwli-ubd4d-lr6br-hpnqf-mettk-jrgtc-n5ioc-mqe.
We can compute the expected manifests with the split_manifest command of the state tool. While we obviously need to provide the ranges we want to split off, we also need to provide the subnet IDs of the subnets involved in the split, the subnet type and the batch time in nanoseconds. The latter is supposed to be set to the batch time in the CUP as printed in the verification step above.
bazel run //rs/state_tool:state-tool \
--config local \
-- split_manifest \
--path /tmp/splitting-artifacts/demo_source_manifest.txt \
--from-subnet gmmtp-v5kpc-wdkrx-pni4q-6qwld-ntwrv-kqxva-c3rnm-o6lx3-2eg74-oae \
--to-subnet ykq2b-vnsfx-dwzwl-tvwli-ubd4d-lr6br-hpnqf-mettk-jrgtc-n5ioc-mqe \
--subnet-type application \
--batch-time-nanos 1692696209206708237 \
--migrated-ranges f4zqk-siaaa-aaaab-qaaba-cai:f3yw6-7qaaa-aaaab-qaabq-cai
Which will output the two manifests. For better readability we present and discuss them individually. The manifest for the first subnet A’ looks as follows:
Subnet gmmtp-v5kpc-wdkrx-pni4q-6qwld-ntwrv-kqxva-c3rnm-o6lx3-2eg74-oae
--------
MANIFEST VERSION: V3
FILE TABLE
idx | size | hash | path
------------+------------+------------------------------------------------------------------+------------------------------------------------------
0 | 216 | 8feafcda70181ac8920dfeba5db951f931f87758ecd5dcd7c920a8468a5bacf9 | canister_states/00000000003000000101/canister.pbuf
1 | 532 | cfa0f9a9f833b072aa0c8d1f6eb8bb9334a36168379eaa0925d9013b2d646bbb | canister_states/00000000003000010101/canister.pbuf
2 | 459789 | 4840e8e6e379f2108cec9de7ef386ea5f927f687b121b90ae765ad144bc0222a | canister_states/00000000003000010101/software.wasm
3 | 0 | 981305d7c0b2ace0f53fe822f4075278fa28511e8c34e70f37fd8425af659b36 | canister_states/00000000003000010101/stable_memory.bin
4 | 1093632 | 1ce90b6ed4c455e1c51c4c2ff466d3c6d49c76a585a014d61eebae6c09b0980b | canister_states/00000000003000010101/vmemory_0.bin
5 | 35 | f590329cc74af3afc36daf577bf29136bc81279401a37175f8a9404277cce4b0 | split_from.pbuf
6 | 2418 | b18a59a56442ddd68d9fcc6fb0103f12c0c0bfd0d01c473711d53b2aff18d596 | system_metadata.pbuf
CHUNK TABLE
idx | file_idx | offset | size | hash
------------+------------+------------+------------+------------------------------------------------------------------
0 | 0 | 0 | 216 | 0c6f866e1b60969cc48b4015bb382d45c282b46a82dd24769a15e9628e4d2f62
1 | 1 | 0 | 532 | 3b53a0d9cb9e0131a4677d86c237ed06a1aadaf46c480e448a8d78afe9c43635
2 | 2 | 0 | 459789 | c470eb0871e137fccfb419b16e2826a63c2a1dba913f5c260ab22ee2958afd28
3 | 4 | 0 | 1048576 | de5005242d69024d356dd4d03d8a6879a2b8aadfff6e0f5f80900d45d363305e
4 | 4 | 1048576 | 45056 | 1ab510085a2a3090dd1d49c578fceabd56a6433bd75e7e2ecc3107c8f8f208eb
5 | 5 | 0 | 35 | 51d26533676036bda303d066119cd7499bd39acc5d2d7b420f8c6d95ff7dd79b
6 | 6 | 0 | 2418 | 679b9963d906d7ecbf818b3e73d8654bd4944d7bd35d6c7c4fde7223bd8d664b
ROOT HASH: 8cb1880663d42b8e6c32cc894e7b7bc9304fc0f72bc89db251a81fe15256c1e8
We can see that all the files and chunks that are also present in the parent manifest have matching hashes. In addition, we have a new split_from.pbuf file that carries meta information about the split. It is introduced by the splitting logic and computed from the input parameters (see here). Note that this demo state has an empty ingress history and so doesn’t contain a file corresponding to the ingress history. But if it would, the ingress history’s hash should match the one in the original manifest (the ingress history will only be pruned once the split subnets start making progress again). The manifest for the second subnet (B) looks as follows:
Subnet ykq2b-vnsfx-dwzwl-tvwli-ubd4d-lr6br-hpnqf-mettk-jrgtc-n5ioc-mqe
--------
MANIFEST VERSION: V3
FILE TABLE
idx | size | hash | path
------------+------------+------------------------------------------------------------------+------------------------------------------------------
0 | 4721 | 346f8b36205998ef3c3c70d5ef64f27096714c83925d98f9e7daf93bf71bd66b | canister_states/00000000003000020101/canister.pbuf
1 | 324196 | 73b2c88c9db040794b23535ecd57fa14528d31ada4d9a82fdb6e9cd21cc0c1e6 | canister_states/00000000003000020101/software.wasm
2 | 0 | 981305d7c0b2ace0f53fe822f4075278fa28511e8c34e70f37fd8425af659b36 | canister_states/00000000003000020101/stable_memory.bin
3 | 1245184 | 0779cd1b7b5afcac3a2e19070abab88bd637261965c36dd1334928980e7a3744 | canister_states/00000000003000020101/vmemory_0.bin
4 | 749 | 493d2f01070bccb63bf75764bcdbc42b1a6946e191029afd74353f339939f51e | canister_states/00000000003000030101/canister.pbuf
5 | 968242 | 912f20ef0b5769a55316b7463cc1d55f079a2e91d1d358a5524a28c0782e82b1 | canister_states/00000000003000030101/software.wasm
6 | 0 | 981305d7c0b2ace0f53fe822f4075278fa28511e8c34e70f37fd8425af659b36 | canister_states/00000000003000030101/stable_memory.bin
7 | 1458176 | 26f9c82ae827458e84befaa73c42f0cbcf39a2c9dae0c777aa26e7fd6a75db62 | canister_states/00000000003000030101/vmemory_0.bin
8 | 35 | f590329cc74af3afc36daf577bf29136bc81279401a37175f8a9404277cce4b0 | split_from.pbuf
9 | 77 | c29f819c86ba8fe211a272961e15dbe1c87e801619c9391dda36f83fbe4192f2 | system_metadata.pbuf
CHUNK TABLE
idx | file_idx | offset | size | hash
------------+------------+------------+------------+------------------------------------------------------------------
0 | 0 | 0 | 4721 | 81af05e10350340c340db100e2b4c1dddc3a7f800ea573ddcef28d562fa70a10
1 | 1 | 0 | 324196 | accdd1e66c56db5b5ff0ac24182a10f65559cfd17cc52d86ac3aa6cae5d567b4
2 | 3 | 0 | 1048576 | a5d9f9ce12da611d7b170fc3156f2dc1255ee11fb6525ddc4dbc3cb7c5f488e5
3 | 3 | 1048576 | 196608 | 9f0bd49a6523364cbc57561e45d67597ef66e29c3fb36273465efd4d210872e8
4 | 4 | 0 | 749 | c0e5fae6857b04d035f8fd752505da9f3905514c4d8dcec7650a18e43af61fa5
5 | 5 | 0 | 968242 | f431111e6b9a9a074daf350cba6404abc47a993f7ddca098de1819c62d99a515
6 | 7 | 0 | 1048576 | de5005242d69024d356dd4d03d8a6879a2b8aadfff6e0f5f80900d45d363305e
7 | 7 | 1048576 | 409600 | 93207dec95a267dab702bb1991af9dd9c5df72ba9d87188d37772282d3948632
8 | 8 | 0 | 35 | 51d26533676036bda303d066119cd7499bd39acc5d2d7b420f8c6d95ff7dd79b
9 | 9 | 0 | 77 | 4c7c38987b8ef245c68643cdc0ae519b8640f4bd260245e2c5b2f3a070abe789
ROOT HASH: 00bcb78b825ce4146ce42e117d5d7256d2f1e351d974753f8e08b588ed11bcd1
Again, the canister-related file and chunk hashes match the ones of the parent manifest. It again contains a split_from.pbuf file with the same hash as the one in the manifest of the sibling subnet. Finally, it also contains a newly created system metadata file, initialized based on the parameters passed to the splitting tool but otherwise empty (see here for the code).
If all the above properties hold, and the root hashes of these manifests appear as the root hashes of the states in the respective subnet recovery proposals for the child subnets, then one can conclude that the split is authentic and it is safe to vote in favor of the respective proposals from an authenticity point of view.