Node Provider Onboarding

Learn how to be accepted by the NNS as a node provider and onboard your nodes to the IC.

Requirements

• Node Hardware
• Rack space with a 10Gb connectivity, RJ45 terminated on the nodes
• Public IP addresses:
  • One /64 IPv6 range
  • One IPv4 address for every 4 node machines
• Hardware wallet
• NitroKey HSM (Optional)
• 11 ICP (10 of which are to be staked for the NNS proposal deposit)
• Basic understanding of neurons, staking, and governance proposals. For instance, understanding what it means to stake a neuron for 8 years.
• The technical knowledge to understand some minor steps that are not explicitly mentioned in these instructions. For instance, when to insert an HSM.

Note: Please allocate at least 0.5 day for going through the first part, i.e., the registration of a new NP. It may even take a couple of days, depending on how quickly the community votes for the proposals. There is a also fair amount of complexity and technical knowledge that needs to be absorbed in order to complete the steps. But this only needs to be done once.

The next step, going to the DC and bringing up and onboarding the machines, is much quicker. Estimate to spend 10-15 minutes per machine. This time should go down to ~5 minutes as you gain experience. Also, multiple machines can be brought up in parallel.

I. Install the required tools

A. Install ic-admin

ic-admin is the tool used to create and submit NNS proposals.

MacOS

1. Retrieve the file

   $ curl "https://download.dfinity.systems/ic/7445081734e6d896d090295967d50710975c4f25/openssl-static-binaries/x86_64-darwin/ic-admin.gz" -o - | gunzip > ./ic-admin
   $ chmod +x ./ic-admin
   

2. Verify the binary

   $ diff <(shasum -a 256 ./ic-admin | cut -d' ' -f1) <(echo 3f75026d2f28f171068e332a42c82a2795c93fbf5ab351baef30b30eb901cdba) && echo "ic-admin checksum matches" || echo "***ERROR***: ic-admin checksum does not match"
   

   This command should output "ic-admin checksum matches"

Linux

NOTE: The instructions below have been tested with the Ubuntu 20.04 release.

1. Retrieve the file

   $ curl "https://download.dfinity.systems/ic/7445081734e6d896d090295967d50710975c4f25/openssl-static-binaries/x86_64-linux/ic-admin.gz" -o - | gunzip > ./ic-admin
   $ chmod +x ./ic-admin
   

2. Verify the binary

   $ diff <(shasum -a 256 ./ic-admin | cut -d' ' -f1) <(echo e29bb9cc462e800b8b960ad49c412e5f5fdbb5ae2ae9fde0c13058422ba32802) && echo "ic-admin checksum matches" || echo "***ERROR***: ic-admin checksum does not match"
   

B. Install dfx

1. dfx is used to generate neuron hotkeys, among other things

   $ sh -ci "$(curl -fsSL https://smartcontracts.org/install.sh)"
   

2. Verify that dfx is up to date.

   $ export PATH=$HOME/bin:$PATH
   $ dfx upgrade
   $ dfx --version
   

3. Create an identity for the Node Provider Hotkey

   $ dfx identity new node-provider-hotkey
   Creating identity: "node-provider-hotkey".
   Created identity: "node-provider-hotkey".
   $ dfx --identity node-provider-hotkey identity get-principal
   wuyst-x5tpn-g5wri-mp3ps-vjtba-de3xs-w5xgb-crvek-tucbe-o5rqi-mae
   

4. Note: The node provider hotkey is NOT the node provider principal. This is the hotkey that is used for the NNS proposal submissions only.

II. Create and Manage Neuron via NNS Frontend Dapp and Internet Identity

1. Setup your hardware wallet: https://medium.com/dfinity/integrating-ledger-nano-with-the-nns-front-end-dapp-user-manual-9c5600925e16
2. Send at least 11 ICPs to the hardware wallet address.
3. Navigate to Neurons tab and create a Neuron by staking at least 10 ICP from your hardware wallet. Staking more ICP works as well, but 10 is the minimum.
4. IMPORTANT! Confirm the transaction on your hardware wallet.

   

5. After the Neuron has been created successfully, confirm to add NNS Dapp as hotkey in the dialogue and on your hardware wallet, and close the dialog after the action completes.

   

6. Set the dissolve delay to at least 6 months, and confirm the choice in the dialogue and on your hardware wallet. After the action completes, you can close the "Follow Neurons".

   

7. You will now see a Neuron listed with its ID. Copy the Neuron ID, since you will need it in the next steps to place the necessary proposals.

   

III. Add hotkeys

1. Select the Neuron you just created to open Neuron management view and press “Add hotkey” button.

   

2. A dialog will pop up where you can enter the principal you generated in step 2 (output from command dfx --identity node-provider-hotkey identity get-principal). This will allow you to submit NNS proposals using ic-admin and will not be used for anything else.
   

   Press the confirm button and confirm the transactions on your hardware wallet.
   

   

3. Get the Ledger Hardware Wallet Principal Id: Navigate back to ICP page and select your Ledger hardware wallet account. You will need to use this Ledger Hardware Wallet principal as the Node Provider principal in order to get the rewards directly into the secure hardware wallet.

1. Copy and save this Node Provider principal by clicking on the copy icon after the principal id. You'll need it in the next steps.

$ NODE_PROVIDER_PRINCIPAL=fharn-5vyi2-4xb4a-64yyi-3jpmj-pga23-mxy25-d5uim-fqcro-eoefh-tae   # Ledger Hardware Wallet principal, from the NNS FrontEnd dapp https://nns.ic0.app/

IV. Configure your HSM

Onboarding without a NitroKey HSM

If you will not be using a NitroKey HSM, continue to the next step.

Using a NitroKey HSM

It's first necessary to install the necessary tools.

MacOS

1. Download this OpenSC binary: https://github.com/OpenSC/OpenSC/releases/download/0.22.0/OpenSC-0.22.0.dmg
2. Double click the DMG image that you downloaded and then double click the OpenSC PKG file.
3. If your system doesn't allow the installation software from an unidentified developer please follow these steps or contact your system administrator:
   • Choose the Apple menu > System Preferences > click Security and Privacy.
   • Click the lock Icon to unlock it, then enter an administrator name and password.
   • Ensure that you're on the tab named “General”.
   • You should see the OpenSC app and you should be able to enable its installation by choosing “Open anyway”.
4. Click continue and install until the installation is complete.

Linux

NOTE: The instructions below have been tested with the Ubuntu 20.04 release.

1. Install pcscd and opensc

   $ sudo add-apt-repository universe
   $ sudo apt update
   $ sudo apt install pcscd opensc
   

V. Setup the Node Operator keys

Onboarding without a NitroKey HSM

1. Ensure dfx is at least version 0.14. Node Operator keys created with older versions of dfx will fail to join the IC. Run:

   $ dfx upgrade
   

2. Create a new principal with dfx:

   $ dfx identity new --storage-mode=plaintext node_operator
   

3. Confirm node_operator identity was created successfully:

   $ dfx identity list
   

   This list should contain node_operator.
4. Copy new key to a known location:

   $ cp ~/.config/dfx/identity/node_operator/identity.pem ./node_operator_private_key.pem
   

Using a NitroKey HSM

1. Initialize the HSM.

   $ sc-hsm-tool --initialize --so-pin 3537363231383830 --pin 358138
   

2. Change the HSM so-pin.
   • WARNING: The new HSM so pin must have 16 hexadecimal digits. This is not very well known, and some HSM users have lost access to a Nitrokey HSM because they tried using regular characters and the command below accepted it.
   • Do NOT change the user pin. It must remain as the default for the onboarding scripts to work

     $ pkcs11-tool --login --login-type so --so-pin 3537363231383830 --change-pin
     

3. Create a keypair on the HSM. Enter the default pin 358138 when prompted.

   $ pkcs11-tool -k --key-type EC:prime256v1 --login -d 01
   

   • Note: Key backup may be possible with these instructions.

VI. Get the node operator principal

Onboarding without a NitroKey HSM

1. Confirm node_operator identity was created successfully in step V.:

   $ dfx identity list
   

   This list should contain node_operator.
2. Get the principal:

   $ NODE_OPERATOR_PRINCIPAL=$(dfx --identity node_operator identity get-principal)
   $ echo $NODE_OPERATOR_PRINCIPAL
   
   uqquy-76uhn-2mys5-xxxxx-xxxxx-xxxxx-xxxxx-xxxxx-xxxxx-xxxxx-xxx
   

Using a NitroKey HSM

1. Configure dfx identity (skip this step if you already configured it for an other HSM).
   • Note: Depending on your installation, the path to the --hsm-pkcs11-lib-path might be different on your platform. You can locate the correct path with the following command:

     $ find / -name opensc-pkcs11.so 2> /dev/null
     

   • MacOS

     $ dfx identity new node-operator-hsm --hsm-key-id 01 --hsm-pkcs11-lib-path /Library/OpenSC/lib/opensc-pkcs11.so
     

   • Linux

     $ dfx identity new node-operator-hsm --hsm-key-id 01 --hsm-pkcs11-lib-path /usr/lib/x86_64-linux-gnu/opensc-pkcs11.so
     

2. Get the principal.

   $ NODE_OPERATOR_PRINCIPAL=$(DFX_HSM_PIN=358138 dfx --identity node-operator-hsm identity get-principal)
   $ echo $NODE_OPERATOR_PRINCIPAL
   
   uqquy-76uhn-2mys5-xxxxx-xxxxx-xxxxx-xxxxx-xxxxx-xxxxx-xxxxx-xxx
   

VII. Register your NP principal to the network

In the next codeblock:

• Replace the NEURON_ID value with your neuron ID from the NNS Frontend Dapp
• Replace the NODE_PROVIDER_PRINCIPAL value with the Ledger Hardware Wallet principal that you got from the NNS Frontend DAPP.
• Replace the NODE_PROVIDER_NAME value with the name of the entity that will provide the nodes.
  
• IMPORTANT: Please make sure that you also update the --summary and include a link to the forum discussion, your company's web page, and/or to another place that can convince the voting community that you are making a legitimate request. This way you will avoid the community voting NO to your proposal and you losing the staked ICPs.

1. Create the Proposal

   $ NODE_PROVIDER_NAME="My Company"
   $ NODE_PROVIDER_PRINCIPAL=fharn-5vyi2-4xb4a-64yyi-3jpmj-pga23-mxy25-d5uim-fqcro-eoefh-tae   # Ledger Hardware Wallet principal, from the NNS FE dapp https://ic0.app/
   $ NEURON_ID=13419667327548602649  # Coming from the NNS FrontEnd dapp https://nns.ic0.app/
   $ ./ic-admin \
           --nns-url https://ic0.app \
           -s ~/.config/dfx/identity/node-provider-hotkey/identity.pem \
       propose-to-add-or-remove-node-provider add \
           --proposer $NEURON_ID \
           --proposal-title "Register a node provider '${NODE_PROVIDER_NAME}'" \
           --summary "Register a node provider '${NODE_PROVIDER_NAME}', in line with the announcement and discussion at https://forum.dfinity.org/t/..." \
           --node-provider-pid "$NODE_PROVIDER_PRINCIPAL"
   

2. Find the proposal on https://dashboard.internetcomputer.org/governance and wait until it is executed before proceeding to next step.

VIII. Ensure that your datacenter is registered in the network

1. Search for your data center on https://dashboard.internetcomputer.org/centers.
   • If you found the datacenter that is hosting your nodes, remember its ID, and skip the following section. Otherwise, proceed with the registration of a new DC.

Create a data center record for a new DC

In the next block of code:

• Replace the --proposer argument value with your Neuron ID from the NNS Frontend Dapp.
• Replace the JSON fields from –data-centers-to-add argument and their corresponding values in --summary with: "id"
• The ID should be combination of two letters representing a city that your datacenter is in, and an incrementing number. Search data center IDs on https://dashboard.internetcomputer.org, and find a combination of two letters and a number that’s not yet registered. Examples:
  • dl1 (Dallas, no IDs with “dl” prefix)
  • zh10 (Zurich, numbers 0-9 are already registered)

• "region" represents the local region of a datacenter and is formulated as a three-part string divided by commas. The three parts making the string are continent, country code, and region, in the given order. Examples:
  • North America,US,Florida
  • Europe,DE,Bavaria
  • Asia,SG,Singapore

• "owner" The entity that provides your datacenter facilities.
  • Search https://dashboard.internetcomputer.org for existing data center providers.
  • If there’s match, make sure you use the same exact some name for your datacenter.
  • Otherwise, name the data center owner to your best knowledge.

• "gps" GPS coordinates.
  • Find your datacenter on https://www.google.com/maps/.
  • Right click on location, and select the GPS coordinates (first item in the menu) in order to copy them.

1. Create the proposal:

   $ NEURON_ID=13419667327548602649  # Coming from the NNS FE dapp https://nns.ic0.app/
   $ ./ic-admin \
           --nns-url https://ic0.app \
           -s ~/.config/dfx/identity/node-provider-hotkey/identity.pem \
       propose-to-add-or-remove-data-centers \
           --summary "Register a Flexential datacenter as dl1 in North America,US,Texas" \
           --skip-confirmation \
           --proposer $NEURON_ID \
           --data-centers-to-add '{
               "id": "dl1",
               "region": "North America,US,Texas",
               "owner": "Flexential",
               "gps": [
                   33.00803, -96.66614
               ]
           }'
   

2. Find the proposal on https://dashboard.internetcomputer.org/governance and wait until it's executed before proceeding to next step.

IX. Create a node operator record

In the next codeblock:

• Replace the NEURON_ID variable value with your neuron ID obtained from the NNS frontend dapp.
• Replace the NODE_PROVIDER_PRINCIPAL variable value with the principal obtained in step VI.
• Replace the DC_ID variable value with id of your datacenter.
• Replace the NODE_ALLOWANCE variable value with number of nodes you are providing.

1. Create the proposal:

   $ NEURON_ID=13419667327548602649  # Coming from the NNS FE dapp https://nns.ic0.app/
   $ NODE_PROVIDER_PRINCIPAL=fharn-5vyi2-4xb4a-64yyi-3jpmj-pga23-mxy25-d5uim-fqcro-eoefh-tae   # Ledger Hardware Wallet principal, from the NNS FE dapp https://nns.ic0.app/
   $ NODE_OPERATOR_PRINCIPAL=uqquy-76uhn-2mys5-xxxxx-xxxxx-xxxxx-xxxxx-xxxxx-xxxxx-xxxxx-xxx   # Node Operator principal from step VI
   $ NODE_PROVIDER_NAME="My Company"
   $ NODE_ALLOWANCE=8
   $ DC_ID=dl1
   
   $ ./ic-admin \
           --nns-url https://ic0.app \
           -s ~/.config/dfx/identity/node-provider-hotkey/identity.pem \
       propose-to-add-node-operator \
           $NODE_PROVIDER_PRINCIPAL \
           --summary "Node provider '$NODE_PROVIDER_NAME' is adding $NODE_ALLOWANCE nodes in the $DC_ID data center" \
           --proposer $NEURON_ID \
           --node-operator-principal-id $NODE_OPERATOR_PRINCIPAL \
           --node-allowance $NODE_ALLOWANCE \
           --dc-id $DC_ID
   

1. Find the proposal on https://dashboard.internetcomputer.org/governance and wait until it's executed before proceeding to next step.

X. Onboard nodes

1. Follow the instructions to onboard new nodes:
   • Gen2 - For NP's onboarding in 2023 and later
     • IC OS Installation Runbook
   • Gen1 - For NP's participating in the IC before 2023
     • IC OS Installation Runbook - PowerEdge R6525
     • IC OS Installation Runbook - Supermicro
2. Verify that all the nodes were successfully onboarded by checking their status on the dashboard is set to either “Up” or “Unassigned”, or by checking the output from ic-admin get-topology command.
   • The internal dashboard can be searched by your provider principal.

See Also

• The Internet Computer project website (hosted on the IC): internetcomputer.org
• IC Node Provider Matrix/Element channel: https://app.element.io/#/room/#ic-node-providers:matrix.org