Difference between revisions of "Node Provider Onboarding"

Revision as of 15:02, 28 June 2022

To participate in the Internet Computer network as a Node Provider and receive the rewards for supporting the network.

Requirements

Node Hardware
Rack space with a 10GB connectivity, RJ45 terminated on the nodes
Public /29 IPv4 range and /64 IPv6 range
Hardware wallet
NitroKey HSM
11 ICP (10 of which are to be staked for the NNS proposal deposit)
Basic understanding of neurons, staking, and governance proposals. For instance, understanding what it means to stake a neuron for 8 years.
The technical knowledge to understand some minor steps that are not explicitly mentioned in these instructions. For instance, when to insert an HSM.

Note: Please allocate at least 0.5 day for going through the first part, i.e., the registration of a new NP. It may even take a couple of days, depending on how quickly the community votes for the proposals. There is a also fair amount of complexity and the technical knowledge that needs to be absorbed in order to complete the steps. But this only needs to be done once.
The next step, going to the DC and bringing up and onboarding the machines, is much quicker. Estimate to spend 10-15 minutes per machine. This time should go down to ~5 minutes as you gain experience. Also, multiple machines can be brought up in parallel.

I. Install the required tools

A. Install ic-admin

ic-admin is the tool used to create and submit NNS proposals.

MacOS

Retrieve the file

curl "https://download.dfinity.systems/blessed/ic/40877a86674b24161c2306c6534b872c51533954/nix-release/x86_64-darwin/ic-admin.gz" -o - | gunzip > ./ic-admin
chmod +x ./ic-admin

Verify the binary

diff <(shasum -a 256 ./ic-admin | cut -d' ' -f1) <(echo 16d955092b697f3abda99f54dafb9e9181a67805d1330a4b8b34b8586a7f1401) && echo "ic-admin checksum matches" || echo "***ERROR***: ic-admin checksum does not match"

Verify that the version is 1.0 or greater
```
./ic-admin --version
ic-admin 1.0
```

Linux

NOTE: The instructions below have been tested with the Ubuntu 20.04 release.

Retrieve the file

 curl "https://download.dfinity.systems/blessed/ic/40877a86674b24161c2306c6534b872c51533954/release/ic-admin.gz" -o - | gunzip > ./ic-admin
$ chmod +x ./ic-admin

Verify the binary

diff <(shasum -a 256 ./ic-admin | cut -d' ' -f1) <(echo 26942a0cd7f89bc0ffbd01287d88b63e333889c67ac9d27e435e57ddd4d211cb) && echo "ic-admin checksum matches" || echo "***ERROR***: ic-admin checksum does not match"

Verify that the version is 1.0 or greater
```
./ic-admin --version
ic-admin 1.0
```

B. Install dfx

dfx allows generating a neuron hotkey, among other things

$ DFX_VERSION=0.9.3 sh -ci "$(curl -fsSL https://sdk.dfinity.org/install.sh)"

Verify that the version is 0.9.3

$ export PATH=$HOME/bin:$PATH
$ dfx --version dfx 0.9.3

Create an identity for the Node Provider Hotkey

$ dfx identity new node-provider-hotkey
Creating identity: "node-provider-hotkey".
Created identity: "node-provider-hotkey".
$ dfx --identity node-provider-hotkey identity get-principal
wuyst-x5tpn-g5wri-mp3ps-vjtba-de3xs-w5xgb-crvek-tucbe-o5rqi-mae

Note: The node provider hotkey is NOT the node provider principal. This is the hotkey that is used for the NNS proposal submissions only.

II. Create and Manage Neuron via NNS Frontend Dapp and Internet Identity

Setup your hardware wallet: https://medium.com/dfinity/integrating-ledger-nano-with-the-nns-front-end-dapp-user-manual-9c5600925e16
Send at least 11 ICPs to the hardware wallet address.
Navigate to Neurons tab and create a Neuron by staking at least 10 ICP from your hardware wallet. Staking more ICP works as well, but 10 is the minimum.
IMPORTANT! Confirming the transaction on your hardware wallet.
After the Neuron has been created successfully, confirm to add NNS Dapp as hotkey in the dialogue and on your hardware wallet, and close the dialog after the action completes.
Set the dissolve delay to at least 6 months, and confirm the choice in the dialogue and on your hardware wallet. After the action completes, you can close the "Follow Neurons".
You will now see a Neuron listed with its ID. Copy the Neuron ID, since you will need it in the next steps to place the necessary proposals.

III. Add hotkeys

Select the Neuron you just created to open Neuron management view and press “Add hotkey” button.
A dialog will pop up where you can enter the principal you generated in step 2 (output from command dfx --identity node-provider-hotkey identity get-principal). This will allow you to submit NNS proposals using ic-admin and will not be used for anything else.

Press the confirm button and confirm the transactions on your hardware wallet.
Get the Ledger Hardware Wallet Principal Id: Navigate back to ICP page and select your Ledger hardware wallet account. You will need to use this Ledger Hardware Wallet principal as the Node Provider principal in order to get the rewards directly into the secure hardware wallet.

Copy and save this Node Provider principal by clicking on the copy icon after the principal id. You'll need it in the next steps.

NODE_PROVIDER_PRINCIPAL=fharn-5vyi2-4xb4a-64yyi-3jpmj-pga23-mxy25-d5uim-fqcro-eoefh-tae   # Ledger Hardware Wallet principal, from the NNS FE dapp https://nns.ic0.app/

IV. Configure your HSM

It's first necessary to install the necessary tools.

MacOS

Download this OpenSC binary: https://github.com/OpenSC/OpenSC/releases/download/0.22.0/OpenSC-0.22.0.dmg
Double click the DMG image that you downloaded and then double click the OpenSC PKG file.
If your system doesn't allow the installation software from an unidentified developer please follow these steps or contact your system administrator:
- Choose the Apple menu > System Preferences > click Security and Privacy.
- Click the lock Icon to unlock it, then enter an administrator name and password.
- Ensure that you're on the tab named “General”.
- You should see the OpenSC app and you should be able to enable its installation by choosing “Open anyway”.
Click continue and install until the installation is complete.

Linux

NOTE: The instructions below have been tested with the Ubuntu 20.04 release.

Install pcscd and opensc

sudo add-apt-repository universe
sudo apt update
sudo apt install pcscd opensc

V. Setup the HSM

Initialize the HSM.

sc-hsm-tool --initialize --so-pin 3537363231383830 --pin 358138

Change the HSM so-pin.
- WARNING: The new HSM so pin must have 16 hexadecimal digits. This is not very well known, and some HSM users have lost access to a Nitrokey HSM because they tried using regular characters and the command below accepted it.
- Do NOT change the user pin. It must remain as the default for the onboarding scripts to work
```
pkcs11-tool --login --login-type so --so-pin 3537363231383830 --change-pin
```
Create a keypair on the HSM. Enter the default pin 358138 when prompted.
- Note: Before initializing the HSM key please refer to the Nitrokey HSM documentation if you wish to create a backup. Creating a backup of the HSM device is NOT possible after the key has already been created.
```
pkcs11-tool -k --key-type EC:prime256v1 --login -d 01
```

VI. Get the node operator principal from the HSM

Configure dfx identity (skip this step if you already configured it for an other HSM).

Note: Depending on your installation, the path to the --hsm-pkcs11-lib-path might be different on your platform. You can locate the correct path with the following command:
```
find / -name opensc-pkcs11.so 2> /dev/null
```

MacOS

dfx identity new node-operator-hsm --hsm-key-id 01 --hsm-pkcs11-lib-path /Library/OpenSC/lib/opensc-pkcs11.so

Linux

dfx identity new node-operator-hsm --hsm-key-id 01 --hsm-pkcs11-lib-path /usr/lib/x86_64-linux-gnu/opensc-pkcs11.so

Get the principal.

$ NODE_OPERATOR_PRINCIPAL=$(DFX_HSM_PIN=358138 dfx --identity node-operator-hsm identity get-principal)
$ echo $NODE_OPERATOR_PRINCIPAL

uqquy-76uhn-2mys5-xxxxx-xxxxx-xxxxx-xxxxx-xxxxx-xxxxx-xxxxx-xxx

VII. Register your NP principal to the network

In the next codeblock:

Replace the NEURON_ID value with your neuron ID from the NNS Frontend Dapp
Replace the NODE_PROVIDER_PRINCIPAL value with the Ledger Hardware Wallet principal that you got from the NNS Frontend DAPP.
Replace the NODE_PROVIDER_NAME value with the name of the entity that will provide the nodes.
IMPORTANT: Please make sure that you also update the --summary and include a link to the forum discussion, your company's web page, and/or to another place that can convince the voting community that you are making a legitimate request. This way you will avoid the community voting NO to your proposal and you losing the staked ICPs.

Create the Proposal

NODE_PROVIDER_NAME="My Company"
NODE_PROVIDER_PRINCIPAL=fharn-5vyi2-4xb4a-64yyi-3jpmj-pga23-mxy25-d5uim-fqcro-eoefh-tae   # Ledger Hardware Wallet principal, from the NNS FE dapp https://nns.ic0.app/
NEURON_ID=13419667327548602649  # Coming from the NNS FE dapp https://nns.ic0.app/
./ic-admin \
        --nns-url https://nns.ic0.app \
        -s ~/.config/dfx/identity/node-provider-hotkey/identity.pem \
    propose-to-add-or-remove-node-provider add \
        --proposer $NEURON_ID \
        --proposal-title "Register a node provider '${NODE_PROVIDER_NAME}'" \
        --summary "Register a node provider '${NODE_PROVIDER_NAME}', in line with the announcement and discussion at https://forum.dfinity.org/t/..." \
        --node-provider-pid "$NODE_PROVIDER_PRINCIPAL"

Find the proposal on https://dashboard.internetcomputer.org/governance and wait until it is executed before proceeding to next step.

VIII. Ensure that your datacenter is registered in the network

Search for your data center on https://dashboard.internetcomputer.org/centers.
- If you found the datacenter that is hosting your nodes, remember its ID, and skip the following section. Otherwise, proceed with the registration of a new DC.

Create a data center record for a new DC

In the next block of code:

Replace the --proposer argument value with your Neuron ID from the NNS Frontend Dapp.
Replace the JSON fields from –data-centers-to-add argument and their corresponding values in --summary with: "id"
The ID should be combination of two letters representing a city that your datacenter is in, and an incrementing number. Search data center IDs on https://dashboard.internetcomputer.org, and find a combination of two letters and a number that’s not yet registered. Examples:
- dl1 (Dallas, no IDs with “dl” prefix)
- zh10 (Zurich, numbers 0-9 are already registered)

"region" represents the local region of a datacenter and is formulated as a three-part string divided by commas. The three parts making the string are continent, country code, and region, in the given order. Examples:
- North America,US,Florida
- Europe,DE,Bavaria
- Asia,SG,Singapore

"owner" The entity that provides your datacenter facilities.
- Search https://dashboard.internetcomputer.org for existing data center providers.
- If there’s match, make sure you use the same exact some name for your datacenter.
- Otherwise, name the data center owner to your best knowledge.

"gps" GPS coordinates.
- Find your datacenter on https://www.google.com/maps/.
- Right click on location, and select the GPS coordinates (first item in the menu) in order to copy them.

Create the proposal:

NEURON_ID=13419667327548602649  # Coming from the NNS FE dapp https://nns.ic0.app/
$ ./ic-admin \
        --nns-url https://nns.ic0.app \
        -s ~/.config/dfx/identity/node-provider-hotkey/identity.pem \
    propose-to-add-or-remove-data-centers \
        --summary "Register a Flexential datacenter as dl1 in North America,US,Texas" \
        --skip-confirmation \
        --proposer $NEURON_ID \
        --data-centers-to-add '{
            "id": "dl1",
            "region": "North America,US,Texas",
            "owner": "Flexential",
            "gps": [
                33.00803, -96.66614
            ]
        }'

Find the proposal on https://dashboard.internetcomputer.org/governance and wait until it's executed before proceeding to next step.

IX. Create a node operator record

In the next codeblock:

Replace the NEURON_ID variable value with your neuron ID obtained from the NNS frontend dapp.
Replace the NODE_PROVIDER_PRINCIPAL variable value with the Ledger Hardware Wallet principal obtained from the NNS frontend dapp.
Replace the DC_ID variable value with id of your datacenter.
Replace the NODE_ALLOWANCE variable value with number of nodes you are providing.

Create the proposal:

NEURON_ID=13419667327548602649  # Coming from the NNS FE dapp https://nns.ic0.app/
NODE_PROVIDER_PRINCIPAL=fharn-5vyi2-4xb4a-64yyi-3jpmj-pga23-mxy25-d5uim-fqcro-eoefh-tae   # Ledger Hardware Wallet principal, from the NNS FE dapp https://nns.ic0.app/
NODE_OPERATOR_PRINCIPAL=$(DFX_HSM_PIN=358138 dfx --identity node-operator-hsm identity get-principal)
NODE_PROVIDER_NAME="My Company"
NODE_ALLOWANCE=28
DC_ID=dl1

./ic-admin \
        --nns-url https://nns.ic0.app \
        -s ~/.config/dfx/identity/node-provider-hotkey/identity.pem \
    propose-to-add-node-operator \
        $NODE_PROVIDER_PRINCIPAL \
        --summary "Node provider '$NODE_PROVIDER_NAME' is adding $NODE_ALLOWANCE nodes in the $DC_ID data center" \
        --proposer $NEURON_ID \
        --node-operator-principal-id $NODE_OPERATOR_PRINCIPAL \
        --node-allowance $NODE_ALLOWANCE \
        --dc-id $DC_ID

Find the proposal on https://dashboard.internetcomputer.org/governance and wait until it's executed before proceeding to next step.

X. Configure firewall rules

Note: It is only necessary to update the firewall rules if you are adding a new DC, with a new IPv6 prefix, to the Internet Computer. If you need to do this:

Replace the NEURON_ID variable value with your neuron ID obtained from the NNS frontend dapp.

Replace the NODES_IPV6_PREFIX variable value with IPv6 prefix of the network of your nodes.

NEURON_ID=13419667327548602649  # Coming from the NNS FE dapp https://nns.ic0.app/
NODES_IPV6_PREFIX=2001:4d78:700:10a::/64
NODE_OPERATOR_PRINCIPAL=$(DFX_HSM_PIN=358138 dfx --identity node-operator-hsm identity get-principal)

NFTABLES=$(./ic-admin --nns-url "https://nns.ic0.app" get-firewall-config | grep "firewall_config" | cut -d':' -f2 | cut -c2- | rev | cut -c2- | rev | xargs printf)
IPV6_PREFIXES=$(./ic-admin --nns-url "https://nns.ic0.app" get-firewall-config | tr -d '\n' | grep -oE 'ipv6_prefixes: \[[^]]+' | cut -d'[' -f2 | tr -d '"' | tr -d ' ' | tr -d '\n'; echo $NODES_IPV6_PREFIX)

./ic-admin \
        --nns-url https://nns.ic0.app \
        -s ~/.config/dfx/identity/node-provider-hotkey/identity.pem \
    propose-to-set-firewall-config \
        --proposer $NEURON_ID \
        --summary "Set the firewall rules for node operator ${NODE_OPERATOR_PRINCIPAL}" \
        <(echo $NFTABLES) \
        - \
        $IPV6_PREFIXES

Find the proposal on https://dashboard.internetcomputer.org/governance and wait until it’s executed before proceeding to next step.

XI. Onboard nodes

Follow the instructions to onboard new nodes.
- IC OS Installation Runbook - PowerEdge R6525
- IC OS Installation Runbook - Supermicro
Verify that all the nodes were successfully onboarded by checking their status on the dashboard is set to either “Up” or “Unassigned”, or by checking the output from ic-admin get-topology command.
- The internal dashboard can be searched by your provider principal.

XII. Set the reward configuration for your nodes

In the next codeblock:

Replace the NEURON_ID variable value with your neuron ID obtained from the NNS frontend dapp.
Replace the <NODE_X_PRINCIPAL> placeholders with your node principals.
Replace the <number-of-nodes> placeholder with the number of nodes you listed.
Note: The current maximum number of nodes per node operator are 28.

NEURON_ID=13419667327548602649  # Coming from the NNS FE dapp https://nns.ic0.app/
NODE_OPERATOR_PRINCIPAL=$(DFX_HSM_PIN=358138 dfx --identity node-operator-hsm identity get-principal)

./ic-admin \
        --nns-url https://nns.ic0.app \
        -s ~/.config/dfx/identity/node-provider-hotkey/identity.pem \
    propose-to-update-node-operator-config \
        --proposer $NEURON_ID \
        --summary "Set rewards for the following nodes:

        * <NODE_1_PRINCIPAL>
        * <NODE_2_PRINCIPAL>
        * ...
        " \
        --node-operator-id $NODE_OPERATOR_PRINCIPAL \
        --rewardable-nodes '{"type0": <number-of-nodes>}'

@@ Line 78: / Line 78: @@
 #:
 #: [[File:dissolve_delay.png|480px|neuron id]]
-# You will now see a Neuron listed with its ID. Copy the Neuron ID, since you will need the it in the next steps to place the necessary proposals.
+# You will now see a Neuron listed with its ID. Copy the Neuron ID, since you will need it in the next steps to place the necessary proposals.
 #: [[File:Neuron id.png|1024px]]