New Subnet Creation

Ever wondered about the meaning behind DFINITY? It’s Decentralized + Infinity. It’s named that way because the Internet Computer is designed to scale infinitely. It means that the Internet Computer can host an unlimited number of canisters (smart contracts), store an unlimited amount of memory, process an unlimited amount of transactions per second. In simple words, Internet Computer is designed to host even large scale social media platforms in a fully decentralized way.

There are two types of widely-used approaches to improve the scalability of a system. (1) Vertical Scaling, and (2) Horizontal Scaling. Vertical scaling means adding more CPU, RAM and disk to a single computer. Horizontal scaling means adding more computers to the system. There is a limit to vertical scaling. But with horizontal scaling, one can achieve unlimited scalability. Internet Computer is one of the first blockchains to successfully use horizontal scaling.

The nodes in the Internet Computer are divided into subnets, each containing a few dozen nodes. The set of nodes in a subnet together maintain one blockchain. Each subnet can host a few thousand canisters and process messages received by those canisters. Each subnet has a limited capacity in terms of the number of canisters (a few thousand), amount of storage (a few TB), and bandwidth (a few hundred transactions per second). But as more subnets are added to the Internet Computer, its overall capacity increases proportionately. There is no limit on the number of subnets that can be added, resulting in unlimited scalability.

We need to do 2 things to create a new subnet.

Add/Register new nodes.
Create a new subnet with the registered nodes that were not yet assigned to any subnet.

Adding/Registering New Nodes

We new describe a series of steps that need to be followed to add a new node to the Internet Computer.

Node provider purchases a NitroKey (a Hardware Security Module), generates a public-key/secret-key pair, and submits an NNS proposal to add his public key to the NNS registry. The community votes on the proposal. If the majority accept the proposal, then the node provider's credentials are added to the NNS registry. From now on, the NNS canisters trust the messages signed by the node provider's secret key. The entire process is specified in the node provider onboarding article.
Node provider purchases node hardware with the recommended specifications and places it in a data center rack that meets the recommended specifications.
The node doesn't yet have any operating system. The node provider needs to install the IC-OS operating system on the node. The detailed procedure can be found in the IC-OS installation runbook articles (Installation for SuperMicro, Installation for Dell Poweredge).
The node provider inserts the NitroKey usb stick into the node machine. The NitroKey contains the secret key of corresponding to the node provider's registered public key.
The node provider then switches on the node to boot the IC-OS operation system, which starts a few processes including orchestrator, crypto and http adapter processes.
The crypto process finds that it never generated any cryptographic key material before. The crypto process then generates new cryptographic keys. This includes node signing key, NIDKG key, ECDSA key, TLS key, etc.
The cryptographic key material need to be registered with the NNS registry. For this, the crypto process sends the keys to the orchestrator, which then crafts a message containing the key material, signs the message with the node provider's signing key present in the NitroKey, and sends the message to the NNS registry canister.
The NNS registry canister creates a record for the new node and stores its cryptographic key material.
The node is now registered in the Internet Computer, but not yet assigned to any subnet.

Creating a New Subnet

We now describe the process of creating a new subnet. To create a new subnet, one just needs to submit an NNS proposal. The proposal specifies a type (create subnet) and a payload (#Payload for 'Create_Subnet' Proposal). A few sample proposals to create a new subnet can be found in the dashboard (Proposal 57048, Proposal 49018, Proposal 55730). Anyone who staked their ICP can vote on the proposal within a deadline. After the deadline, if the majority of the voters accept the proposal, then the subnet is created as described below.

The NNS has a canister called "registry", which stores the configuration of the Internet Computer including the configuration of each node and subnet. When the 'Create subnet' proposal is accepted, it automatically calls a method of the NNS Registry with input the payload included in the NNS proposal. The registry canister then creates a new subnet configuration based on the information in the payload, and stores the subnet record.

Each node in the Internet Computer maintains a local copy of the NNS registry. Each node runs a "registry replicator" process (github) which is a part of the orchestrator and is responsible for regularly syncing NNS registry and maintaining the local registry copy. Once every few seconds (can be configured by setting poll_delay_duration_ms variable, which has a default value of 5 seconds), the registry replicator queries the NNS registry for the changes that happened after the recent sync, and applies those changes to the local registry copy. The local registry is stored on the disk and any process running on the node can read the contents of the local registry.

The orchestrator monitors the local registry to see what subnet the node has been assigned to. If the unassigned node has now been assigned to a new subnet according to the registry, the orchestrator first downloads the replica software using the url in the subnet's registry configuration. The orchestrator now installs the replica software (#Installing_the_Replica) and restarts the node.

Payload for 'Create Subnet' Proposal

The payload for the 'Create Subnet' NNS proposal contains the list of parameters to be used by the new subnet. Some of the fields in the payload are as follows. The structure of payloads for every type of NNS proposal can be found on github.

subnet_type - The type of the subnet to be created. Subnets of different type might exhibit different behavior, e.g. being more restrictive in what operations are allowed or privileged compared to other subnet types. There are a few types of subnets.
- Application subnet - A normal subnet where no restrictions are applied.
- System subnet - A more privileged subnet where certain restrictions are applied, like not charging for cycles or restricting who can create and install canisters on it.
- Verified application subnet - A subnet type that is like application subnets but can have some additional features.
node_ids - List of node ids of unassigned nodes to be included in the subnet.
replica_version_id - The version id of the replica software to be installed in the nodes of the subnet.
- A replica image has to be first uploaded to a publicly accessible server.
- Then one needs to create an NNS proposal of type "Elect new binary replica revision" and specify the replica image url and version id in the payload. A sample proposal can be found in the dashboard (proposal 92338).
- When the proposal is accepted by the majority of voters, the replica version is stored in the NNS registry.
- This replica version id can now be specified in any other NNS proposals including "create subnet".
Parameters for the P2P layer -
- gossip_max_chunk_size - The maximum chunk size supported on this subnet.
- gossip_retransmission_request_ms - Period for sending retransmission request.
- gossip_receive_check_cache_size - History size for receive check.
- gossip_registry_poll_period_ms - Period for polling the registry for updates.
- gossip_max_duplicity - Max duplicate requests in underutilized networks.
- gossip_max_chunk_wait_ms - Timeout for an outstanding request.
- gossip_pfn_evaluation_period_ms - Period for re evaluating the priority function.
- gossip_max_artifact_streams_per_peer - The maximum number of outstanding request per peer MIN/DEFAULT/MAX.
- advert_best_effort_percentage -
Parameters for the Consensus layer
- max_block_payload_size - The maximum combined size of the ingress and xnet messages that fit into a block.
- max_ingress_bytes_per_message - The maximum amount of bytes per message. This is a hard cap, which means ingress messages greater than the limit will be dropped.
- max_ingress_messages_per_block - The maximum number of ingress messages allowed per block.
- ingress_bytes_per_block_soft_cap
- initial_notary_delay_millis - Initial delay for notary (in milliseconds), to give time to rank-0 block propagation.
- unit_delay_millis - Unit delay for blockmaker (in milliseconds).
Parameters for the execution layer
- max_instructions_per_message - The maximum number of instructions a message can execute.
- max_instructions_per_round - The maximum number of instructions a round can execute.
- max_instructions_per_install_code - The maximum number of instructions an "install_code" message can execute.
- max_number_of_canisters - The maximum number of canisters that may be present on the subnet at any given time. A value of 0 is equivalent to setting no limit. This also provides an easy way to maintain compatibility of different versions of replica and registry.
Parameters related to cryptographic keys
- dkg_dealings_per_block
- ecdsa_config - This field is optional. It specifies the configuration settings to be used for ECDSA signature scheme.
- dkg_interval_length - This is the number of rounds in an epoch. At the end of each epoch, the nodes run a distributed key generation protocol to reshare their secret key shares.
SSH Access to Subnet
- ssh_readonly_access - The list of public keys whose owners have "readonly" SSH access to all replicas on this subnet, in case it is necessary to perform subnet recovery.
- ssh_backup_access - The list of public keys whose owners have "backup" SSH access to nodes on the NNS subnet to make sure the NNS can be backed up.
Subnet Features - The list of feature flags to be used for the subnet. This indicates the features that are to be enabled/disabled in the subnet. This includes the following features.
- Canister sandboxing - This feature flag controls whether canister execution happens in sandboxed process or not. It is disabled by default.
- Http requests - This feature flag controls whether canisters of this subnet are capable of performing http(s) requests to the web2.
- Bitcoin testnet feature - Whether or not the subnet is capable of serving requests to the bitcoin testnet canister.
- Bitcoin - Controls whether the bitcoin feature is enabled and which bitcoin network is supported.
is_halted - If "true", the subnet will be halted: it will no longer create or execute blocks.
start_as_nns - If set to yes, the subnet starts as a (new) NNS.
subnet_id_override - This field is optional. If a principal id is specified, then that principal id will be used for the new subnet.

Executing the 'Create Subnet' Proposal

When the 'Create Subnet' NNS proposal is voted and accepted by the community, the proposal is executed automatically. To execute the proposal, a method in the registry canister is called (github), which mutates the registry to add configuration information of the new subnet. The registry is a versioned key-value storage. Each mutation to the registry is stored as a new version.

The registry canister performs the following steps.

Run the Distributed Key Generation protocol.
- The first step is to generate the cryptographic information required for the new subnet to work properly. After every round of the Internet Computer protocol, the nodes of the subnet agree upon and sign their latest replicated state. For this, the subnet utilizes a threshold BLS signature scheme. In a nutshell, each node in the subnet holds a "share" of the subnet's signing key (none of the nodes know the complete subnet's signing key). After every round, each of the nodes individually sign their replicated state using their signing key "share". If at least a "threshold" number of nodes agree upon the same replicated state, their signature "shares" can be combined to obtain the signature of the replicated state using the subnet's signing key. A subnet also utilizes threshold BLS signature scheme for many other purposes including the generation of random beacon in the consensus layer and generation of random tape in the execution layer.
- In order to generate the above signing key share for each node in the new subnet, the nodes in the NNS engage in a distributed key generation (DKG) protocol. At the end of the protocol, all the nodes in the NNS agree on a few "dealings" for each node in the new subnet. The dealings of a node can be combined to obtain the signing key share of the node. The dealings for each node are encrypted with the encryption key registered by the node.
Create a genesis block and Catch-Up Package
- The registry canister creates a genesis block. This is the first block in the blockchain. Structurally, this is a Summary Block containing the DKG dealings computed above.
- The registry canister then generates a Catch-Up Package which contains the above genesis block along with other information required for the subnet to start the consensus protocol.
Create a Subnet Record
- Each subnet has a subnet record storing all its configuration information. The subnet record of the new subnet contains all the information present in the payload of the 'Create Subnet' proposal.
Perform the following 5 mutations to the registry. The registry is a versioned key-value storage.
- Append to subnet list.
  - For the key subnet_list, the registry stores the list of all subnets. Append the new subnet to this list.
- Store subnet record.
  - Add a key-value pair with key = subnet_record_{subnet_id} and value = newly created subnet record, where {subnet_id} is the subnet id of the newly created subnet.
- Store catch-up package.
  - Add a key-value pair with key = catch_up_package_contents_{subnet_id} and value = newly created catch-up package.
- Store subnet threshold signing key.
  - Add a key-value pair with key = crypto_threshold_signing_public_key_{subnet_id} and value = threshold BLS signing key generated above.
- Routing table mutation.
  - Each subnet has a range of canister ids that are utilized for canisters in that subnet. The range of canister ids for each subnet is called the routing table. The routing table helps us route canister traffic to the appropriate subnet. When a new subnet is created, the next available set of CANISTER_IDS_PER_SUBNET (currently set to 2^20) canister ids is assigned as the range for the subnet. This information is stored in the key routing_table of the registry.

Installing the Replica

The orchestrator uses a blue/green deployment to start the replica software, which works as follows. The disk is divided into 3 partitions -- 2 partitions for the replica process and 1 partition for the data. If a node is not assigned to any subnet, both the replica partitions are empty. If the node is actively participating in a subnet, one of the replica partitions is used to run the replica process and the other replica partition is empty. To install the new replica software,

The orchestrator loads the replica image into an empty replica partition.
The orchestrator then modifies the boot loader settings to use the newly loaded replica partition.
The orchestrator restarts the node.
When the node boots up again, the newly loaded replica partition is used as per the boot loader settings.