Node Provider Node Swapping Guide

Introduction

Node providers run nodes on the Internet Computer and receive rewards depending on how well their nodes perform. Only nodes that contribute to the overall performance of the node provider are the nodes that are part of a subnet. We call those nodes assgined. All other nodes that aren't a part of any subnet and are not utilized don't contribute to the overall performance of the node provider. We call those nodes unassigned.

It is in the node provider's best interest to keep the assgined nodes as well maintained and operational as possible since that is how they can maximize their rewards. To achieve that over time the node providers can utilize the node swapping feature.

Node swapping

Node swapping feature is used to swap an assigned node for an unassigned node owned by the same node operator.

Example

A node provider notices that one of his nodes is performing poorly. In order to maximize his rewards, he wants to act quickly and remove the assigned node from a subnet for a different unassigned node within the same data center. He uses the node swapping feature to do that without any proposals or governance. After performing the swap they can perform maintenance on the node and see why it failed in the first place.

Performing a node swap

To perform a node swap one has to follow the following steps:

1. Find the principal ID of an assigned node that should be removed from a subnet. To do that one can use various tools, but for the sake of this example we will show examples from the public dashboard.
   
2. Find the principal ID of an unassigned node that should be added to a subnet in place of an assigned node picked in the previous step. To do that one can click on the data center (below the DC ID "to1" for this example) of the node from the previous step which will take them to the view which contains all the node machines from the same data center. From there, pick the one that isn't assigned to any subnet (the subnet ID column should contain a '-').
   
3. Using ic-admin utility to perform a node swap:

   # Uncomment this if the node operator identity is behind a private key file
   #AUTH=(
   # "--secret-key-pem"
   # <path-to-key-pem>
   #)
   
   # Uncomment this if node operator identity is behind an hsm
   #AUTH=(
   # "--use-hsm"
   # "--slot"
   # <hsm-key-slot>
   # "--key-id"
   # <hsm-key-id>
   # "--pin"
   # <hsm-pin>
   #)
   
   ic-admin --nns-urls https://ic0.app ${AUTH[@]} swap-node-in-subnet-directly \
       --old-node-id <principal-from-step-1> \
       --new-node-id <principal-from-step-2>
   

4. After the response from the call is successful the node doesn't leave the subnet immediately. There is graceful leaving period that needs to be awaited. It takes around 15-20 minutes depending on the subnet. If the logs of the node are observed through the BMC console there are two messages that need to show in order for the node to completely leave the subnet gracefully. The first log that indicates that the node noticed that the swapping was requested looks like the following:

   Node started leaving subnet <subnet-id>.
   

   After some time the node will show the following log which indicates it left the subnet:

   Node gracefully left a subnet <subnet-id>
   

5. Only after the second log is it safe to turn off the node and proceed to do maintenance. IMPORTANT: if the node is turned off before it gracefully leaves the subnet it will appear as malicious to the consensus which can result in rewards penalty!

Limitations and constraints

To ensure fair and safe use of the swapping feature there are some limitations and constraints about the usage of the feature that need to be shared.

Ownership and locality constraints

A node provider can only perform a swap of two nodes if and only if the following locality constraints are met:

1. The old node is an assigned node (it is a part of a subnet) and the new node is an unassigned node (it isn't a part of any subnet).
2. Both old and new nodes are owned by the same node provider and are located in the same data center. Furthermore they have to be owned by the same node operator. This is needed to ensure that the decentralization doesn't change when the swaps are performed.

Rate limiting constraints

A node provider can only perform a swap of two nodes if and only if the following rate limiting constraints are met:

1. The subnet which the old node is a part of hasn't had a swap in the last 2 hours - this limit is here to protect the subnet from losing its state from too many swaps happening at the same time.
2. The node provider performing the swap hasn't made a swap on the subnet which the old node is a part of in the last 12 hours - this limit is here to protect other node providers from being unable to swap nodes because one node provider is constantly swapping nodes within the 2 hour limit of the 1st constraint.