Internet Computer Wiki - User contributions [en]

Node Provider Networking Guide

2024-02-20T18:26:52Z

Gary.mcelroy: /* Appendix 1: Number of IPv4 Addresses Required */ Clarify table - #'s imply nothing about ipv4 configuration

This guide is designed to provide an overview of the networking requirements and guide Node Providers through setting up their servers into a rack with functioning networking.

Configuring networks is not trivial. You should be familiar with IP networking, network equipment and network cabling.

Resources to learn about networking:

* [https://learningnetwork.cisco.com/s/article/200-301-ccna-study-materials CCNA Study Materials]
* Kevin Wallace [https://www.youtube.com/@kwallaceccie YouTube Training Videos]

'''DFINITY does not provide support for network configuration.'''

If you hire technical assistance, keep decentralization and security in mind. Use a local technician you personally know and carefully monitor their work.

== Requirements ==
To join your servers to the Internet Computer (IC) you will need:

* 10G Network equipment
** [[wikipedia:Small_Form-factor_Pluggable|SFP+]] or [[wikipedia:10_Gigabit_Ethernet|Ethernet]]
** Switch(es)
** Cabling
** Quantity determined by number of nodes deployed
* Rackspace in a data center
* Internet connection
** Bandwidth
*** ~300Mbps per node
*** Ingress/egress ratio is currently 1:1. We expect egress (serving responses to client queries) to increase faster than ingress in the future.
*** This should guide how many servers to deploy and the appropriate ISP connection speed
*** E.g. a 1Gbps connection will support up to 3 IC nodes.
** One IPv6 /64 subnet - each node gets multiple IPv6 addresses
** '''(For Gen2 Node Providers - Gen1 Node Providers will receive different requirements)''' One IPv4 address allocated for every 4 nodes in a given data center per node provider (IPv4 addresses cannot be shared between node providers). See [[Node Provider Networking Guide#Appendix 1: Number of IPv4 Addresses Required|Appendix 1]] for table.
***Additionally, one domain name for each node configured with an IPv4 address. See [[Node Provider Domain Name Guide]] for details.
**'''All IP addresses are assigned statically''' and automatically by IC-OS
***This is configured in the [[IC-OS Installation Runbook#6. Add configuration|IC-OS Installation Runbook]]

==Network Cabling==
When racking and stacking your servers, ensure the '''at least one 10G network port''' on each server is connected to the 10G switch. SFP+ and Ethernet are supported.

[[File:Supermicro 1124US-TNRP 1U server rear photo diagram.png|480px|screenshot]]

For example, on a Supermicro 1U server, the 10G ports are in a cluster as seen above. Vendors differ.

Connect the 10G switch to the ISP endpoint - this could be the Top Of Rack (TOR) switch or other box.

==Network Configuration==
Node machines require:

*The ability to acquire a public static IPv6 address on a /64 subnet
* An IPv6 gateway to communicate with other nodes on the broad internet
*Unfiltered internet access

'''(For Gen2 Node Providers - Gen1 Node Providers will receive different requirements)''' One of every four nodes requires:

*The ability to acquire a public static IPv4 address
* An IPv4 gateway to communicate with other nodes on the broad internet
*Unfiltered internet access

There are many many ways to configure the network and some details depend on the ISP and data center. Here are some [[Example Network Configuration Scenarios]].

See the [[Node Provider Networking Troubleshooting Guide]] for help.

==BMC Setup Recommendations==

===What’s a BMC?===
The [[wikipedia:Intelligent_Platform_Management_Interface#Baseboard_management_controller|Baseboard Management Controller (BMC)]] grants control of the underlying server hardware.

BMC’s have notoriously poor security. Vendors may name their implementation differently (Dell -> iDRAC, HPE -> iLO, etc.).

=== Recommendations===

====Change the password====
BMC’s usually come with a common password. Log in via crash cart, KVM or the web interface and change it to something [https://krebsonsecurity.com/password-dos-and-donts/ strong].

====No broad internet access====
It is highly recommended: '''do not expose your BMC''' to the broad internet. This is a safety precaution against attackers.

Options:

*Don’t connect the BMC to the internet.
**Maintenance or node recovery will require physical access in this case.
**Any BMC activities occur via SSH on the host (unreliable on many mainboard vendors) or via crash cart.
* Connect the BMC to a separate dumb switch, and the dumb switch connects to a Rack Mounted Unit (RMU).
*Connect the BMC to a managed switch, and create a separate VLAN

This can get complicated. It’s outside the scope of this document to explain how to do this.

Resources:

*[https://security.stackexchange.com/questions/46351/best-practice-for-accessing-management-port-of-firewall StackExchange - Best practice for accessing management port of firewall]
*[https://www.supermicro.com/products/nfo/files/IPMI/Best_Practices_BMC_Security.pdf Supermicro Guidance]
*[https://www.unicomengineering.com/blog/ipmi-best-practices/ Unicom Guidance]

== Network monitoring==

===SNMP-based Network Monitoring: ===

* Device Compatibility: Make sure your network devices support and enable SNMP agents. Choose the version of SNMP that aligns with your security needs as different versions offer varying security levels and functionality.
*Secure Configuration: Implement SNMPv3 to enhance security through authentication and encryption, protecting against unauthorized access and data interception.
*Monitoring Points: Select specific network parameters critical for performance. (such as bandwidth utilization, CPU usage, and memory usage). Set up SNMP polling for these parameters.
*Thresholds and Alerts: Predefine alerts when monitored parameters exceed limits to identify issues proactively and take corrective actions.
* Data Retention: Establish data retention policies for storing SNMP data for trend and capacity analysis.
*Regular Review: It is important to regularly review SNMP monitoring configurations and thresholds to ensure that they are up-to-date and aligned with the changing network environment..

===GNMI/gRPC-based Network Monitoring:===

*Protocol Familiarity: Get familiar with GNMI data models for your network devices and understand how they use gRPC (Remote Procedure Call) for network management.
* Device Support: Verify that your network devices support GNMI, which is more commonly found in modern networking equipment that supports programmability.
*Authentication and Encryption: Implement TLS for gRPC security to protect communication between the monitoring system and devices.
*Model Definitions: Make sure you either have access to or create GNMI data models for the devices you're monitoring. These models define the structure and hierarchy of the data that is accessible through GNMI.
*Data Subscription: GNMI allows for real-time updates through subscriptions. Set up subscriptions for relevant data points to receive continuous updates without frequent polling.
*Streaming Mode: Use gNMI's streaming mode for efficient real-time data transfer.

== Server monitoring==

===SNMP-based Server Hardware Monitoring:===

*Determine SNMP Compatibility: Before configuring SNMP monitoring, make sure you enable SNMP agents on your servers. Also, verify the compatibility of the SNMP version with your monitoring system.
*Secure Configuration: Implement SNMPv3 to enhance security through authentication and encryption, protecting against unauthorized access and data interception.
* Monitoring Parameters: It's important to monitor the CPU utilization to ensure that the performance is optimal and to identify any potential bottlenecks. Keeping track of the memory usage is crucial to prevent resource exhaustion. It's also important to check the network interface traffic to identify any bandwidth bottlenecks. Finally, monitoring the server temperatures and hardware health indicators can help detect any hardware issues.
*SNMP Polling: SNMP polling should be set up regularly to collect data on critical parameters.- Thresholds and Alerts: Set Thresholds: Define appropriate thresholds for each monitored parameter. These thresholds determine when alerts should be triggered.
*Data Retention and Trend Analysis: Retain historical SNMP data for trend analysis, capacity planning and performance identification.
*Regular Review: It is important to regularly check the SNMP monitoring configurations and thresholds in order to ensure that they are appropriate for the environment. This helps to maintain proper alignment and accuracy in monitoring.

==What NOT to do==

===Don’t use external firewalls, packet filters, rate limiters===
Don’t block or interfere with any traffic to the node machines. This can disrupt node machine functionality. Occasionally ports are opened for incoming (and outgoing) connections when new versions of node software are deployed.

====What about network security?====
IC-OS manages its own software firewalls and rate limiters strictly and is designed with security as a primary principle.

===Don't configure the switch to use LACP bonding===
This feature is on the roadmap for investigation but IC nodes do not support LACP bonding at the moment. Configuring it on the switch may cause problems with nodes.

==How DFINITY manages its servers==
See reference DFINITY [[Gen-2 Data Center runbook|data center runbook]].

== Final Checklist==

*Did you deploy a 10G switch?
*Is at least '''one 10G port''' on each server plugged into the 10G switch?
* Do you have '''one IPv6 /64 prefix''' allocated from your ISP?
*'''(Gen2)''' Do you have at least '''one IPv4 address for every four nodes''' allocated?
*Does each node have ~300Mbps bandwidth?
*Is your '''BMC inaccessible''' from the broad internet?

==References==

*[[Gen-2 Network Requirements|Gen2 Network Requirements]] - more detailed, possibly out of date.

==Appendix 1: Number of IPv4 Addresses Required ==
'''(For Gen2 Node Providers - Gen1 Node Providers will receive different requirements)'''

This table refers to quantities - the same IPv4 address should not be reused on different nodes.
{| class="wikitable"
|'''How many nodes you have'''
|'''IPv4 Addresses needed for the whole DC'''
|-
|1 to 4
| 1
|-
|5 to 8
|2
|-
|9 to 12
|3
|-
|13 to 16
|4
|-
|17 to 20
|5
|-
|21 to 24
|6
|-
|25 to 28
|7
|}

Node Provider Networking Guide

2024-02-20T18:25:03Z

Gary.mcelroy: /* Requirements */ Distinguish Gen1 vs Gen2 ipv4 req's

This guide is designed to provide an overview of the networking requirements and guide Node Providers through setting up their servers into a rack with functioning networking.

Configuring networks is not trivial. You should be familiar with IP networking, network equipment and network cabling.

Resources to learn about networking:

* [https://learningnetwork.cisco.com/s/article/200-301-ccna-study-materials CCNA Study Materials]
* Kevin Wallace [https://www.youtube.com/@kwallaceccie YouTube Training Videos]

'''DFINITY does not provide support for network configuration.'''

If you hire technical assistance, keep decentralization and security in mind. Use a local technician you personally know and carefully monitor their work.

== Requirements ==
To join your servers to the Internet Computer (IC) you will need:

* 10G Network equipment
** [[wikipedia:Small_Form-factor_Pluggable|SFP+]] or [[wikipedia:10_Gigabit_Ethernet|Ethernet]]
** Switch(es)
** Cabling
** Quantity determined by number of nodes deployed
* Rackspace in a data center
* Internet connection
** Bandwidth
*** ~300Mbps per node
*** Ingress/egress ratio is currently 1:1. We expect egress (serving responses to client queries) to increase faster than ingress in the future.
*** This should guide how many servers to deploy and the appropriate ISP connection speed
*** E.g. a 1Gbps connection will support up to 3 IC nodes.
** One IPv6 /64 subnet - each node gets multiple IPv6 addresses
** '''(For Gen2 Node Providers - Gen1 Node Providers will receive different requirements)''' One IPv4 address allocated for every 4 nodes in a given data center per node provider (IPv4 addresses cannot be shared between node providers). See [[Node Provider Networking Guide#Appendix 1: Number of IPv4 Addresses Required|Appendix 1]] for table.
***Additionally, one domain name for each node configured with an IPv4 address. See [[Node Provider Domain Name Guide]] for details.
**'''All IP addresses are assigned statically''' and automatically by IC-OS
***This is configured in the [[IC-OS Installation Runbook#6. Add configuration|IC-OS Installation Runbook]]

==Network Cabling==
When racking and stacking your servers, ensure the '''at least one 10G network port''' on each server is connected to the 10G switch. SFP+ and Ethernet are supported.

[[File:Supermicro 1124US-TNRP 1U server rear photo diagram.png|480px|screenshot]]

For example, on a Supermicro 1U server, the 10G ports are in a cluster as seen above. Vendors differ.

Connect the 10G switch to the ISP endpoint - this could be the Top Of Rack (TOR) switch or other box.

==Network Configuration==
Node machines require:

*The ability to acquire a public static IPv6 address on a /64 subnet
* An IPv6 gateway to communicate with other nodes on the broad internet
*Unfiltered internet access

'''(For Gen2 Node Providers - Gen1 Node Providers will receive different requirements)''' One of every four nodes requires:

*The ability to acquire a public static IPv4 address
* An IPv4 gateway to communicate with other nodes on the broad internet
*Unfiltered internet access

There are many many ways to configure the network and some details depend on the ISP and data center. Here are some [[Example Network Configuration Scenarios]].

See the [[Node Provider Networking Troubleshooting Guide]] for help.

==BMC Setup Recommendations==

===What’s a BMC?===
The [[wikipedia:Intelligent_Platform_Management_Interface#Baseboard_management_controller|Baseboard Management Controller (BMC)]] grants control of the underlying server hardware.

BMC’s have notoriously poor security. Vendors may name their implementation differently (Dell -> iDRAC, HPE -> iLO, etc.).

=== Recommendations===

====Change the password====
BMC’s usually come with a common password. Log in via crash cart, KVM or the web interface and change it to something [https://krebsonsecurity.com/password-dos-and-donts/ strong].

====No broad internet access====
It is highly recommended: '''do not expose your BMC''' to the broad internet. This is a safety precaution against attackers.

Options:

*Don’t connect the BMC to the internet.
**Maintenance or node recovery will require physical access in this case.
**Any BMC activities occur via SSH on the host (unreliable on many mainboard vendors) or via crash cart.
* Connect the BMC to a separate dumb switch, and the dumb switch connects to a Rack Mounted Unit (RMU).
*Connect the BMC to a managed switch, and create a separate VLAN

This can get complicated. It’s outside the scope of this document to explain how to do this.

Resources:

*[https://security.stackexchange.com/questions/46351/best-practice-for-accessing-management-port-of-firewall StackExchange - Best practice for accessing management port of firewall]
*[https://www.supermicro.com/products/nfo/files/IPMI/Best_Practices_BMC_Security.pdf Supermicro Guidance]
*[https://www.unicomengineering.com/blog/ipmi-best-practices/ Unicom Guidance]

== Network monitoring==

===SNMP-based Network Monitoring: ===

* Device Compatibility: Make sure your network devices support and enable SNMP agents. Choose the version of SNMP that aligns with your security needs as different versions offer varying security levels and functionality.
*Secure Configuration: Implement SNMPv3 to enhance security through authentication and encryption, protecting against unauthorized access and data interception.
*Monitoring Points: Select specific network parameters critical for performance. (such as bandwidth utilization, CPU usage, and memory usage). Set up SNMP polling for these parameters.
*Thresholds and Alerts: Predefine alerts when monitored parameters exceed limits to identify issues proactively and take corrective actions.
* Data Retention: Establish data retention policies for storing SNMP data for trend and capacity analysis.
*Regular Review: It is important to regularly review SNMP monitoring configurations and thresholds to ensure that they are up-to-date and aligned with the changing network environment..

===GNMI/gRPC-based Network Monitoring:===

*Protocol Familiarity: Get familiar with GNMI data models for your network devices and understand how they use gRPC (Remote Procedure Call) for network management.
* Device Support: Verify that your network devices support GNMI, which is more commonly found in modern networking equipment that supports programmability.
*Authentication and Encryption: Implement TLS for gRPC security to protect communication between the monitoring system and devices.
*Model Definitions: Make sure you either have access to or create GNMI data models for the devices you're monitoring. These models define the structure and hierarchy of the data that is accessible through GNMI.
*Data Subscription: GNMI allows for real-time updates through subscriptions. Set up subscriptions for relevant data points to receive continuous updates without frequent polling.
*Streaming Mode: Use gNMI's streaming mode for efficient real-time data transfer.

== Server monitoring==

===SNMP-based Server Hardware Monitoring:===

*Determine SNMP Compatibility: Before configuring SNMP monitoring, make sure you enable SNMP agents on your servers. Also, verify the compatibility of the SNMP version with your monitoring system.
*Secure Configuration: Implement SNMPv3 to enhance security through authentication and encryption, protecting against unauthorized access and data interception.
* Monitoring Parameters: It's important to monitor the CPU utilization to ensure that the performance is optimal and to identify any potential bottlenecks. Keeping track of the memory usage is crucial to prevent resource exhaustion. It's also important to check the network interface traffic to identify any bandwidth bottlenecks. Finally, monitoring the server temperatures and hardware health indicators can help detect any hardware issues.
*SNMP Polling: SNMP polling should be set up regularly to collect data on critical parameters.- Thresholds and Alerts: Set Thresholds: Define appropriate thresholds for each monitored parameter. These thresholds determine when alerts should be triggered.
*Data Retention and Trend Analysis: Retain historical SNMP data for trend analysis, capacity planning and performance identification.
*Regular Review: It is important to regularly check the SNMP monitoring configurations and thresholds in order to ensure that they are appropriate for the environment. This helps to maintain proper alignment and accuracy in monitoring.

==What NOT to do==

===Don’t use external firewalls, packet filters, rate limiters===
Don’t block or interfere with any traffic to the node machines. This can disrupt node machine functionality. Occasionally ports are opened for incoming (and outgoing) connections when new versions of node software are deployed.

====What about network security?====
IC-OS manages its own software firewalls and rate limiters strictly and is designed with security as a primary principle.

===Don't configure the switch to use LACP bonding===
This feature is on the roadmap for investigation but IC nodes do not support LACP bonding at the moment. Configuring it on the switch may cause problems with nodes.

==How DFINITY manages its servers==
See reference DFINITY [[Gen-2 Data Center runbook|data center runbook]].

== Final Checklist==

*Did you deploy a 10G switch?
*Is at least '''one 10G port''' on each server plugged into the 10G switch?
* Do you have '''one IPv6 /64 prefix''' allocated from your ISP?
*'''(Gen2)''' Do you have at least '''one IPv4 address for every four nodes''' allocated?
*Does each node have ~300Mbps bandwidth?
*Is your '''BMC inaccessible''' from the broad internet?

==References==

*[[Gen-2 Network Requirements|Gen2 Network Requirements]] - more detailed, possibly out of date.

==Appendix 1: Number of IPv4 Addresses Required ==
'''(For Gen2 Node Providers - Gen1 Node Providers will receive different requirements)'''
{| class="wikitable"
|'''# Nodes'''
|'''# IPv4 Addresses'''
|-
|1 to 4
| 1
|-
|5 to 8
|2
|-
|9 to 12
|3
|-
|13 to 16
|4
|-
|17 to 20
|5
|-
|21 to 24
|6
|-
|25 to 28
|7
|}

Removing a Node From the Registry

2024-02-05T23:08:53Z

Gary.mcelroy: /* Steps */ Add note to alert the matrix channel

= Removing a node from the registry via DFX =

== When is this necessary? ==
Node operator records allow for some number of nodes to be joined to the network. This is called the '''node allowance'''. Redeploying the same machine will result in the node generating the same IPv6 address. When joining the network any old node-id associated with this address will be removed - and the node allowance will not be affected.

But if the BMC changes - because of a motherboard replacement for example - a new IPv6 address will be generated. In this case the node will appear new to the network upon joining. If the number of nodes deployed matches the node allowance, no new nodes can join.

In this case - the '''old node-id must be removed.'''

== Steps ==
# Ensure that the node does not exist in any subnet.
## If the node is in a subnet - request help on the [[Node Provider Matrix channel|matrix channel]].
## As a last resort, unplugging the node will work. Please alert the matrix channel you are doing this.
# If applicable, get the HSM (NodeOperator Key) from the data center. Skip if you used a `pem` file to create the node operator record.
## Insert it into a computer that has DFX installed and has internet access.
# Using the terminal, execute the following command (Where NODE_ID is the principal as shown on the dashboard of the node to remove):

=== IF USING PHYSICAL HSM: ===
<code>$ dfx identity new node-operator-hsm --hsm-key-id 01 --hsm-pkcs11-lib-path /usr/lib/x86_64-linux-gnu/opensc-pkcs11.so

$ dfx canister --network ic --identity node-operator-hsm call rwlgt-iiaaa-aaaaa-aaaaa-cai remove_node_directly '(record { node_id = principal "NODE_ID" })'</code>

=== IF USING NO HSM: ===
<code>$ dfx identity import node_operator node_operator_private_key.pem --storage-mode=plaintext

$ dfx canister --network ic --identity node_operator call rwlgt-iiaaa-aaaaa-aaaaa-cai remove_node_directly '(record { node_id = principal "NODE_ID" })'</code>

=== When the node is ready to be re-registered: ===
* If it was healthy, has had no problems, and does not need firmware or anything else updated, then simply re-insert the HSM and reboot the server, and it will rejoin with the same node ID principal it was deployed with.
* If it had problems, had hardware replaced, had firmware updated, etc., then please do a fresh redeployment.

[[Node Provider Documentation|Return to Node Provider Documentation]]

[[Node Provider Troubleshooting|Return to Node Provider Troubleshooting]]

Removing a Node From the Registry

2024-02-05T23:05:04Z

Gary.mcelroy: /* Steps */ Added details about 'ensure the node does not exist in any subnet'

= Removing a node from the registry via DFX =

== When is this necessary? ==
Node operator records allow for some number of nodes to be joined to the network. This is called the '''node allowance'''. Redeploying the same machine will result in the node generating the same IPv6 address. When joining the network any old node-id associated with this address will be removed - and the node allowance will not be affected.

But if the BMC changes - because of a motherboard replacement for example - a new IPv6 address will be generated. In this case the node will appear new to the network upon joining. If the number of nodes deployed matches the node allowance, no new nodes can join.

In this case - the '''old node-id must be removed.'''

== Steps ==
# Ensure that the node does not exist in any subnet.
## If the node is in a subnet - request help on the [[Node Provider Matrix channel|matrix channel]]. Alternatively just unplug the node.
# If applicable, get the HSM (NodeOperator Key) from the data center. Skip if you used a `pem` file to create the node operator record.
## Insert it into a computer that has DFX installed and has internet access.
# Using the terminal, execute the following command (Where NODE_ID is the principal as shown on the dashboard of the node to remove):

=== IF USING PHYSICAL HSM: ===
<code>$ dfx identity new node-operator-hsm --hsm-key-id 01 --hsm-pkcs11-lib-path /usr/lib/x86_64-linux-gnu/opensc-pkcs11.so

$ dfx canister --network ic --identity node-operator-hsm call rwlgt-iiaaa-aaaaa-aaaaa-cai remove_node_directly '(record { node_id = principal "NODE_ID" })'</code>

=== IF USING NO HSM: ===
<code>$ dfx identity import node_operator node_operator_private_key.pem --storage-mode=plaintext

$ dfx canister --network ic --identity node_operator call rwlgt-iiaaa-aaaaa-aaaaa-cai remove_node_directly '(record { node_id = principal "NODE_ID" })'</code>

=== When the node is ready to be re-registered: ===
* If it was healthy, has had no problems, and does not need firmware or anything else updated, then simply re-insert the HSM and reboot the server, and it will rejoin with the same node ID principal it was deployed with.
* If it had problems, had hardware replaced, had firmware updated, etc., then please do a fresh redeployment.

[[Node Provider Documentation|Return to Node Provider Documentation]]

[[Node Provider Troubleshooting|Return to Node Provider Troubleshooting]]

Removing a Node From the Registry

2024-02-05T22:09:01Z

Gary.mcelroy: Add context

= Removing a node from the registry via DFX =

== When is this necessary? ==
Node operator records allow for some number of nodes to be joined to the network. This is called the '''node allowance'''. Redeploying the same machine will result in the node generating the same IPv6 address. When joining the network any old node-id associated with this address will be removed - and the node allowance will not be affected.

But if the BMC changes - because of a motherboard replacement for example - a new IPv6 address will be generated. In this case the node will appear new to the network upon joining. If the number of nodes deployed matches the node allowance, no new nodes can join.

In this case - the '''old node-id must be removed.'''

== Steps ==
# Ensure that the node does not exist in any subnet.
# If applicable, get the HSM (NodeOperator Key) from the data center. Skip if you used a `pem` file to create the node operator record.
## Insert it into a computer that has DFX installed and has internet access.
# Using the terminal, execute the following command (Where NODE_ID is the principal as shown on the dashboard of the node to remove):

=== IF USING PHYSICAL HSM: ===
<code>$ dfx identity new node-operator-hsm --hsm-key-id 01 --hsm-pkcs11-lib-path /usr/lib/x86_64-linux-gnu/opensc-pkcs11.so

$ dfx canister --network ic --identity node-operator-hsm call rwlgt-iiaaa-aaaaa-aaaaa-cai remove_node_directly '(record { node_id = principal "NODE_ID" })'</code>

=== IF USING NO HSM: ===
<code>$ dfx identity import node_operator node_operator_private_key.pem --storage-mode=plaintext

$ dfx canister --network ic --identity node_operator call rwlgt-iiaaa-aaaaa-aaaaa-cai remove_node_directly '(record { node_id = principal "NODE_ID" })'</code>

=== When the node is ready to be re-registered: ===
* If it was healthy, has had no problems, and does not need firmware or anything else updated, then simply re-insert the HSM and reboot the server, and it will rejoin with the same node ID principal it was deployed with.
* If it had problems, had hardware replaced, had firmware updated, etc., then please do a fresh redeployment.

[[Node Provider Documentation|Return to Node Provider Documentation]]

[[Node Provider Troubleshooting|Return to Node Provider Troubleshooting]]

Node Deployment Guide

2024-02-05T17:05:27Z

Gary.mcelroy: /* 5. Create Bootable USB Stick */ Add status progress to dd commands

This runbook covers all steps necessary to install the Internet Computer Operating System (IC-OS).

The physical machine is expected to be racked and stacked according to its respective manual.

To complete these steps, you are expected to by physically present with your machine(s). Once you successfully onboarded your first node, you can bring up the other nodes in parallel.

If you encounter issues through any of these steps, check the [[Node Provider Troubleshooting]] page. If that does not solve your problem, you are encouraged to ask for assistance in the [[Node Provider Matrix channel]].

==1. Choose onboarding path (HSM vs. no HSM)==
If you chose the [[Node Provider Onboarding#5. Choose onboarding path .28HSM vs no HSM.29|HSM Node Provider Onboarding Path]], follow the [[NitroKey HSM installation runbook]] to onboard your nodes.

If you chose to onboard '''without''' a Nitrokey HSM, '''continue to the next step.'''

==2. Obtain requirements ==
*A USB (3.0 speed that can hold at least 4GB) to put the image file on.
**Faster USBs will allow the process to go much faster.
*The <code>node_operator_private_key.pem</code> for your data center (Acquired from [[Node Provider Onboarding#6. Setup the Node Operator keys|Node Provider Onboarding step 6]])
* It is recommended that each server have a label with the BMC's MAC address for ease of identification in future dashboard upgrades.

== 3. Download installation image==
Download the latest release of the '''IC-OS USB Installer Image''' and the '''corresponding checksum''' from the [https://dashboard.internetcomputer.org/releases Internet Computer Dashboard Releases].
*Note that you should always use a release that is less than 6 weeks old in order to ensure that your node can correctly connect to the network.

== 4. Verify checksum and unarchive file==
===Mac OS X ===
#Open the Terminal and type:
#:<syntaxhighlight lang="shell">shasum -a 256 ~/Downloads/disk-img.tar.gz</syntaxhighlight>
#Compare the calculated checksum with the '''IC-OS installation image checksum''' file downloaded in the previous step. '''Warning:''' Only continue if they are identical, otherwise please post your issue in the [[Node Provider Matrix channel]].
#:Open the Terminal and type: <syntaxhighlight lang="shell">tar xzvf ~/Downloads/disk-img.tar.gz</syntaxhighlight>

===Linux / Ubuntu===
#Open the Terminal and type:
#:<syntaxhighlight lang="shell">sha256sum ~/Downloads/disk-img.tar.gz</syntaxhighlight>
#Compare the calculated checksum with the '''IC-OS installation image checksum''' file downloaded in the previous step. '''Warning:''' Only continue if they are identical, otherwise please post your issue in the [[Node Provider Matrix channel]].
#:Open the Terminal and type: <syntaxhighlight lang="shell">tar xzvf ~/Downloads/disk-img.tar.gz</syntaxhighlight>

===Windows===
#Open PowerShell and type:
#:<syntaxhighlight lang="shell">Get-FileHash -Algorithm SHA256 .\Downloads\disk-img.tar.gz</syntaxhighlight>
#Compare the calculated checksum with the '''IC-OS installation image checksum''' file downloaded in the previous step. '''Warning:''' Only continue if they are identical, otherwise please post your issue in the [[Node Provider Matrix channel]].
#:Open PowerShell and type: <syntaxhighlight lang="shell">tar xzvf .\Downloads\disk-img.tar.gz</syntaxhighlight>

==5. Create Bootable USB Stick ==
===Mac OS X===
# Open the Terminal and type:
#:<syntaxhighlight lang="shell">diskutil list</syntaxhighlight>
#All available drives should be shown. Identify which device corresponds to your USB stick. You may need to unmount the USB drive:
#:<syntaxhighlight lang="shell">sudo diskutil unmount /dev/YOUR_USB_DEVICE_MOUNTED_PARTITION # E.g. /dev/disk4s1</syntaxhighlight>
#Replace ''/dev/YOUR_USB_DEVICE'' with the device that corresponds to your USB stick. Additionally, replace the path to your downloaded IC-OS ''disk.img'' file. '''Warning:''' You risk losing your own data if you specify a wrong drive.
#:<syntaxhighlight lang="shell">sudo dd if=/Users/YOUR_USER_NAME/Downloads/disk.img of=/dev/YOUR_USB_DEVICE bs=1M status=progress</syntaxhighlight>If you get a “device is busy” error from the dd command, you can try running the following command to unmount all of the partitions on the disk, then re-run the dd command:
#:<syntaxhighlight lang="shell">sudo diskutil unmountDisk /dev/YOUR_USB_DEVICE # E.g. /dev/disk4</syntaxhighlight>

=== Linux / Ubuntu===
#Open the Terminal and type
#:<syntaxhighlight lang="shell">blkid</syntaxhighlight>
# All available drives should be shown. Identify which device corresponds to your USB stick. You may need to unmount the USB drive:
#:<syntaxhighlight lang="shell">sudo diskutil unmount /dev/YOUR_USB_DEVICE_MOUNTED_PARTITION # E.g. /dev/sdb1</syntaxhighlight>
#Replace ''/dev/YOUR_USB_DEVICE'' with the device that corresponds to your USB stick. Additionally, replace the path to your downloaded IC-OS ''disk.img'' file. '''Warning:''' You risk losing your own data if you specify a wrong drive.
#:<syntaxhighlight lang="shell">sudo dd if=/home/YOUR_USER_NAME/Downloads/disk.img of=/dev/YOUR_USB_DEVICE bs=1M status=progress</syntaxhighlight>

===Windows===
#Download and install [https://rufus.ie/en/ Rufus Portable]
#Start Rufus
#Select the USB stick under device and select the previously downloaded IC-OS disk image and press start
#:[[File:05.png|480px|screenshot]]
#You may see some warnings. Make sure you don't have any other USBs in your computer and chose OK
#:[[File:06.png|480px|screenshot]]
#:[[File:07.png|480px|screenshot]]
# The "Ready" bar will go from left to right as it completes.

==6. Add configuration==

===A. Open Config.ini in a text editor===

===='''Mac OS X'''====

#Open Finder. You should now be able to see the CONFIG partition. If it's not visible, remove the USB and insert it again.
#:[[File:mac_01.png|580px|screenshot]]
#Double-click <code>config.ini</code> to open it in TextEdit.

===='''Linux'''====

#Open the File Manager. You should now be able to see the CONFIG partition. If it's not visible, remove the USB and insert it again.
#:[[File:linux_01.png|580px|screenshot]]
#Double-click <code>config.ini</code> to open it in KWrite.

===='''Windows'''====

#Open the Disk Management utility with a right click on the Start menu
#:[[File:09-b.png|300px|screenshot]]#:
#Right click the CONFIG partition
# Select Change drive letter or paths...
#:[[File:10-b.png|780px|screenshot]]
# Select any letter from the drop-down list
#:[[File:11-b.png|480px|screenshot]]
#Click OK.
#You should now be able to see the CONFIG partition in your Windows Explorer. Select the <code>config.ini</code> configuration file
#:[[File:12-b.png|780px|screenshot]]
#Click on Edit to open it.

===B. Edit Config.ini===

#Insert your IPv6 prefix and gateway.
#:[[File:Config.ini ipv6 info.png|780px|screenshot]]
#:*The IPv6 prefix should consist of four groups of hexadecimal digits, separated by colons (':'). Each group can contain up to four hex digits.
#:* For example, a valid prefix could look like this: <code>2a00:fb01:400:200</code>
#:*'''Important:'''
#:**The prefix should not have a trailing ':'
#:**IPv6 CIDR notation allows for a double colon ('::') to represent consecutive groups of zeroes in an address. However, the prefix configuration in this context does '''not''' support '::'. The '::' shorthand should '''not''' be used. Even if some groups are all zeros, they must be explicitly written out.
#[Optional] Insert your IPv4 info and domain name.
#:[[File:Config.ini ipv4 info.png|780px|screenshot]]
#:*Configuring your node with IPv4 settings is optional, but if you do configure your node with IPv4 settings, you must also define the domain name for your node.
#:*'''Important:'''
#:**Please note that you must reconfigure the IC-OS image for every IPv4 node you deploy. This means that you cannot use a single IC-OS image to configure multiple nodes like you were able to do when just configuring IPv6 nodes. '''After each IPv4 node deployment, you must plug your USB stick back into your laptop and return to [[Node Deployment Guide#6. Add configuration|step 6]] in the node deployment guide to reconfigure your installation image.'''
#Save the changes.
#:*If you have trouble saving this file directly, you may need to save to a known location first, then copy the file into place.
#:*If you need help, please do not hesitate to post your issue in the [[Node Provider Matrix channel]].
#:*:[[File:mac_03.png|580px|screenshot]]

=== C. Copy Node Operator private key to config partition===

#Copy <code>node_operator_private_key.pem</code> (created in [[Node Provider Onboarding#6. Setup the Node Operator keys|Node Provider Onboarding step 6]]) to the <code>CONFIG</code> partition. This file should have the name <code>node_operator_private_key.pem</code>, and sit next to <code>config.ini</code>, NOT inside the <code>ssh_authorized_keys</code> folder.

==7. Connect Crash Cart==
#In order to configure the UEFI and initiate the installation of the IC-OS, please connect a crash cart to the physical machine.
#Plug-in the VGA/Video, keyboard and IC-OS USB stick
#:[[File:08.png|580px|screenshot]]

==8. UEFI Setup and Boot Menu==

Make sure that server date/time is set to UTC (Universal Time Coordinated)

Use the related page below to set up the BIOS/UEFI according to your hardware vendor.

*[[Node Provider Machine Hardware Guide#Gen 2 Node Machine requirements|Gen2 hardware]]
**[[IC-OS Installation - UEFI Configuration - Gen2 Dell]]
** [[IC-OS Installation - UEFI Configuration - Gen2 Supermicro]]
** [[IC-OS Installation - UEFI Configuration - Gen2 Gigabyte]]
**[[IC-OS Installation - UEFI Configuration - Gen2 ASUS]]
*[[Node Provider Machine Hardware Guide#Gen 1 Node Machine requirements|Gen1 hardware]]
**[[IC-OS Installation - UEFI Configuration - Gen1 Dell|IC-OS Installation - UEFI Configuration - Gen1 Dell (Poweredge R6525)]]
**[[IC-OS Installation - UEFI Configuration - Gen1 Supermicro]]
***
'''Important:''' Do NOT enable the RAID bios setting. Doing so will cause issues with the IC-OS installation.

Resume from this point when you are finished configuring the BIOS.

==9. IC-OS Installation==
#Please wait while the USB Installer is booting up. This process can take up to 3 minutes.
#:[[File:35-sm.png|580px|screenshot]]
#The IC-OS installation starts. Please keep an eye on the progress. This part can take up to 10 minutes. Please remember to check the [[Possible Node Onboarding Errors]] page if you encounter any errors.
#:[[File:36-sm.png|580px|screenshot]]
#If the installation finished successfully, it will initiate a reboot.
#:[[File:38-sm.png|580px|screenshot]] 
==10. First Boot==
Please remember to check the [[Possible Node Onboarding Errors]] page if you encounter any errors onboarding.
#Once you see this message, you may unplug the USB stick and VGA/Video.
#:[[File:Node join message.png|580px|screenshot]]

Congratulations! Your machine successfully joined the Internet Computer! The machine has joined the IC and the Node Provider will start receiving rewards!

==11. Verify node onboarding==

#Verify that your node was successfully onboarded by checking its status on the [https://dashboard.internetcomputer.org/ dashboard] is set to either “Awaiting Subnet” or “Active in Subnet”.
#*The dashboard can be searched by your Node Provider principal. There, you should see the Node ID of your node (Node ID is outputted in step 10).
#*If the status of your node is not either “Awaiting Subnet” or “Active in Subnet”, or if it is not listed under your Node Provider principal, you should contact the [[Node Provider Matrix channel]] for assistance.
#*:[[File:Node onboarding verification.png|680px|screenshot]]

Node Deployment Guide (with an HSM)

2024-02-05T17:04:03Z

Gary.mcelroy: /* 4. Verify checksum and unarchive file */ Add status progress to dd commands

This runbook covers all steps necessary to install the Internet Computer Operating System (IC-OS) using the legacy NitroKey HSM instructions. To use the non-HSM onboarding instructions, follow the [[IC-OS Installation Runbook]].

The physical machine is expected to be racked and stacked according to its respective manual.

To complete these steps, you are expected to be physically present in the data center your machine(s) reside(s). Once you successfully onboard your first node, you can bring up the others in parallel.

If you encounter issues through any of these steps, check the [[Node Provider Troubleshooting]] page. If that does not solve your problem, you are encouraged to ask for assistance in the [[Node Provider Matrix channel]].

==1. Choose onboarding path (HSM vs no HSM)==
If you chose the [[Node Provider Onboarding#5. Choose onboarding path .28HSM vs no HSM.29|HSM Node Provider Onboarding Path]], '''continue to the next step.'''

If you chose to onboard '''without''' a Nitrokey HSM, follow the [[IC-OS Installation Runbook]] to onboard your nodes.

==2. Obtain requirements==
*A USB (3.0 speed that can hold at least 4GB) to put the image file on.
** Faster USBs will allow the process to go much faster.
*The NitroKey HSM for your data center.
*[Optional] A USB hub
** This is helpful at some data centers for simultaneously connecting keyboard, mouse, Nitrokey, etc.
*It is recommended that each server has a label with the BMC's MAC address for ease of identification in future dashboard upgrades.

==3. Download installation image==
Download the latest release of the '''IC-OS USB Installer Image''' and the '''corresponding checksum''' from the [https://dashboard.internetcomputer.org/releases Internet Computer Dashboard Releases].
*'''Note that you should always use a release from the last 6 weeks (newer is better) in order to ensure that your node can correctly correct to the network.'''

==4. Verify checksum and unarchive file==
===Mac OS X===
#Open the Terminal and type:
#:<syntaxhighlight lang="shell">shasum -a 256 ~/Downloads/disk-img.tar.gz</syntaxhighlight>
#Compare the calculated checksum with the '''IC-OS installation image checksum''' file downloaded in the previous step. '''Warning:''' Only continue if they are identical, otherwise please post your issue in the [[Node Provider Matrix channel]].
#:Open the Terminal and type: <syntaxhighlight lang="shell">tar xzvf ~/Downloads/disk-img.tar.gz</syntaxhighlight>

===Linux / Ubuntu===
#Open the Terminal and type:
#:<syntaxhighlight lang="shell">sha256sum ~/Downloads/disk-img.tar.gz</syntaxhighlight>
#Compare the calculated checksum with the '''IC-OS installation image checksum''' file downloaded in the previous step. '''Warning:''' Only continue if they are identical, otherwise please post your issue in the [[Node Provider Matrix channel]].
#:Open the Terminal and type: <syntaxhighlight lang="shell">tar xzvf ~/Downloads/disk-img.tar.gz</syntaxhighlight>

===Windows===
#Open PowerShell and type:
#:<syntaxhighlight lang="shell">Get-FileHash -Algorithm SHA256 .\Downloads\disk-img.tar.gz</syntaxhighlight>
#Compare the calculated checksum with the '''IC-OS installation image checksum''' file downloaded in the previous step. '''Warning:''' Only continue if they are identical, otherwise please post your issue in the [[Node Provider Matrix channel]].
#:Open PowerShell and type: <syntaxhighlight lang="shell">tar xzvf .\Downloads\disk-img.tar.gz</syntaxhighlight>

==5. Create Bootable USB Stick==
===Mac OS X===
#Open the Terminal and type:
#:<syntaxhighlight lang="shell">diskutil list</syntaxhighlight>
#All available drives should be shown. Identify which device corresponds to your USB stick. You may need to unmount the USB drive:
#:<syntaxhighlight lang="shell">sudo diskutil unmount /dev/YOUR_USB_DEVICE_MOUNTED_PARTITION # E.g. /dev/disk4s1</syntaxhighlight>
#Replace ''/dev/YOUR_USB_DEVICE'' with the device that corresponds to your USB stick. Additionally, replace the path to your downloaded IC-OS ''disk.img'' file. '''Warning:''' You risk losing your own data if you specify a wrong drive.
#:<syntaxhighlight lang="shell">sudo dd if=/Users/YOUR_USER_NAME/Downloads/disk.img of=/dev/YOUR_USB_DEVICE bs=1M status=progress</syntaxhighlight>If you get a “device is busy” error from the dd command, you can try running the following command to unmount all of the partitions on the disk, then re-run the dd command:
#:<syntaxhighlight lang="shell">sudo diskutil unmountDisk /dev/YOUR_USB_DEVICE # E.g. /dev/disk4</syntaxhighlight>

===Linux / Ubuntu===
#Open the Terminal and type
#:<syntaxhighlight lang="shell">blkid</syntaxhighlight>
#All available drives should be shown. Identify which device corresponds to your USB stick. You may need to unmount the USB drive:
#:<syntaxhighlight lang="shell">sudo diskutil unmount /dev/YOUR_USB_DEVICE_MOUNTED_PARTITION # E.g. /dev/sdb1</syntaxhighlight>
#Replace ''/dev/YOUR_USB_DEVICE'' with the device that corresponds to your USB stick. Additionally, replace the path to your downloaded IC-OS ''disk.img'' file. '''Warning:''' You risk losing your own data if you specify a wrong drive.
#:<syntaxhighlight lang="shell">sudo dd if=/home/YOUR_USER_NAME/Downloads/disk.img of=/dev/YOUR_USB_DEVICE bs=1M status=progress</syntaxhighlight>

===Windows===
#Download and install [https://rufus.ie/en/ Rufus Portable]
#Start Rufus
#Select the USB stick under device and select the previously downloaded IC-OS disk image and press start
#:[[File:05.png|480px|screenshot]]
#You may see some warnings. Make sure you don't have any other USBs in your computer and chose OK
#:[[File:06.png|480px|screenshot]]
#:[[File:07.png|480px|screenshot]]
#The "Ready" bar will go from left to right as it completes.

==6. Add configuration ==

===A. Open Config.ini in a text editor===

===='''Mac OS X'''====

#Open Finder. You should now be able to see the CONFIG partition. If it's not visible, remove the USB and insert it again.
#:[[File:mac_01.png|580px|screenshot]]
#Double-click <code>config.ini</code> to open it in TextEdit.

===='''Linux'''====

#Open the File Manager. You should now be able to see the CONFIG partition. If it's not visible, remove the USB and insert it again.
#:[[File:linux_01.png|580px|screenshot]]
#Double-click <code>config.ini</code> to open it in KWrite.

===='''Windows'''====

#Open the Disk Management utility with a right click on the Start menu.
#:[[File:09-b.png|300px|screenshot]]#:
#Right click the CONFIG partition.
#Select Change drive letter or paths...
#:[[File:10-b.png|780px|screenshot]]
#Select any letter from the drop-down list.
#:[[File:11-b.png|480px|screenshot]]
# Click OK.
# You should now be able to see the CONFIG partition in your Windows Explorer. Select the <code>config.ini</code> configuration file.
#:[[File:12-b.png|780px|screenshot]]
#Click on Edit to open it.

===B. Edit Config.ini===

#Insert your IPv6 prefix and gateway.
#:[[File:Config.ini ipv6 info.png|780px|screenshot]]
#:*The IPv6 prefix should consist of four groups of hexadecimal digits, separated by colons (':'). Each group can contain up to four hex digits.
#:* For example, a valid prefix could look like this: <code>2a00:fb01:400:200</code>
#:*'''Important:'''
#:**The prefix should not have a trailing ':'
#:**IPv6 CIDR notation allows for a double colon ('::') to represent consecutive groups of zeroes in an address. However, the prefix configuration in this context does '''not''' support '::'. The '::' shorthand should '''not''' be used. Even if some groups are all zeros, they must be explicitly written out.
#[Optional] Insert your IPv4 info and domain name.
#:[[File:Config.ini ipv4 info.png|780px|screenshot]]
#:*Configuring your node with IPv4 settings is optional, but if you do configure your node with IPv4 settings, you must also define the domain name for your node.
#:*'''Important:'''
#:**Please note that you must reconfigure the IC-OS image for every IPv4 node you deploy. This means that you cannot use a single IC-OS image to configure multiple nodes like you were able to do when just configuring IPv6 nodes. '''After each IPv4 node deployment, you must plug your USB stick back into your laptop and return to [[Node Deployment Guide (with an HSM)#6. Add configuration|step 6]] in the node deployment guide to reconfigure your installation image.'''
#Save the changes.
#:*If you have trouble saving this file directly, you may need to save to a known location first, then copy the file into place.
#:*If you need help, please do not hesitate to post your issue in the [[Node Provider Matrix channel]].
#:*:[[File:mac_03.png|580px|screenshot]]

==7. Connect Crash Cart==
#In order to configure the UEFI and initiate the installation of the IC-OS, please connect a crash cart to the physical machine.
#Plug-in the VGA/Video, keyboard and IC-OS USB stick
#:[[File:08.png|580px|screenshot]]

==8. UEFI Setup and Boot Menu==
Use the related page below to set up the BIOS/UEFI according to your hardware vendor.

*[[Node Provider Machine Hardware Guide#Gen 2 Node Machine requirements|Gen2 hardware]]
**[[IC-OS Installation - UEFI Configuration - Gen2 Dell]]
**[[IC-OS Installation - UEFI Configuration - Gen2 Supermicro]]
**[[IC-OS Installation - UEFI Configuration - Gen2 Gigabyte]]
**[[IC-OS Installation - UEFI Configuration - Gen2 ASUS]]
*[[Node Provider Machine Hardware Guide#Gen 1 Node Machine requirements|Gen1 hardware]]
**[[IC-OS Installation - UEFI Configuration - Gen1 Dell|IC-OS Installation - UEFI Configuration - Gen1 Dell (Poweredge R6525)]]
**[[IC-OS Installation - UEFI Configuration - Gen1 Supermicro]]
***
'''Important:''' Do NOT enable the RAID bios setting. Doing so will cause issues with the IC-OS installation.

Resume from this point when you are finished configuring the BIOS.

==9. IC-OS Installation ==
#Please wait while the USB Installer is booting up. This process can take up to 3 minutes.
#:[[File:35-sm.png|580px|screenshot]]
#The IC-OS installation starts. Please keep an eye on the progress. This part can take up to 10 minutes. Please remember to check the [[Possible Node Onboarding Errors]] page if you encounter any errors.
#:[[File:36-sm.png|580px|screenshot]]
#Once you get asked to insert the HSM, please remove the keyboard and instead insert the HSM USB device.
#:[[File:37-sm.png|580px|screenshot]]
#If the installation finished successfully, it will initiate a reboot. '''Please do not unplug the USB stick or HSM USB''' device at this point.
#:[[File:38-sm.png|580px|screenshot]]

==10. First Boot==
Please remember to check the [[Possible Node Onboarding Errors]] page if you encounter any errors onboarding.

🚨 '''Do NOT re-try the IC-OS installation after completing this section, as this can cause duplication within the registry.'''
#The first boot of the IC-OS still requires the HSM USB device. Please wait until further instructions. This step can take up to 2 minutes.
#:[[File:39-sm.png|580px|screenshot]]
#Once you see this message, you may unplug the HSM USB device, USB stick and VGA/Video. Your machine successfully joined the Internet Computer! **'''Label the server with the node ID for easy future identification in the dashboard (at least the first 10 characters).***''' Note that each redeployment will assign a new node ID, so you will need to update your label if this was a 2nd redeployment.
#:[[File:Node join message.png|580px|screenshot]]

Congratulations! Your machine successfully joined the Internet Computer! The machine has joined the IC and the Node Provider will start receiving rewards!

🚨 Again: Once you reach this stage and see this message, '''do not attempt to restart the onboarding process.''' Doing so may cause duplicate entries in the registry.
==11. Verify node onboarding ==

#Verify that your node was successfully onboarded by checking its status on the [https://dashboard.internetcomputer.org/ dashboard] is set to either “Awaiting Subnet” or “Active in Subnet”.
#*The dashboard can be searched by your Node Provider principal. There, you should see the Node ID of your node (Node ID outputted in step 10).
#*If the status of your node is not either “Awaiting Subnet” or “Active in Subnet”, or if it is not listed under your Node Provider principal, you should contact the [[Node Provider Matrix channel]] for assistance.
#*:[[File:Node onboarding verification.png|680px|screenshot]]

Node Deployment Guide

2024-01-29T17:19:54Z

Gary.mcelroy: Correct wording

This runbook covers all steps necessary to install the Internet Computer Operating System (IC-OS).

The physical machine is expected to be racked and stacked according to its respective manual.

To complete these steps, you are expected to by physically present with your machine(s). Once you successfully onboarded your first node, you can bring up the other nodes in parallel.

If you encounter issues through any of these steps, check the [[Node Provider Troubleshooting]] page. If that does not solve your problem, you are encouraged to ask for assistance in the [[Node Provider Matrix channel]].

==1. Choose onboarding path (HSM vs. no HSM)==
If you chose the [[Node Provider Onboarding#5. Choose onboarding path .28HSM vs no HSM.29|HSM Node Provider Onboarding Path]], follow the [[NitroKey HSM installation runbook]] to onboard your nodes.

If you chose to onboard '''without''' a Nitrokey HSM, '''continue to the next step.'''

==2. Obtain requirements ==
*A USB (3.0 speed that can hold at least 4GB) to put the image file on.
**Faster USBs will allow the process to go much faster.
*The <code>node_operator_private_key.pem</code> for your data center (Acquired from [[Node Provider Onboarding#6. Setup the Node Operator keys|Node Provider Onboarding step 6]])
* It is recommended that each server have a label with the BMC's MAC address for ease of identification in future dashboard upgrades.

== 3. Download installation image==
Download the latest release of the '''IC-OS USB Installer Image''' and the '''corresponding checksum''' from the [https://dashboard.internetcomputer.org/releases Internet Computer Dashboard Releases].
*Note that you should always use a release that is less than 6 weeks old in order to ensure that your node can correctly connect to the network.

== 4. Verify checksum and unarchive file==
===Mac OS X ===
#Open the Terminal and type:
#:<syntaxhighlight lang="shell">shasum -a 256 ~/Downloads/disk-img.tar.gz</syntaxhighlight>
#Compare the calculated checksum with the '''IC-OS installation image checksum''' file downloaded in the previous step. '''Warning:''' Only continue if they are identical, otherwise please post your issue in the [[Node Provider Matrix channel]].
#:Open the Terminal and type: <syntaxhighlight lang="shell">tar xzvf ~/Downloads/disk-img.tar.gz</syntaxhighlight>

===Linux / Ubuntu===
#Open the Terminal and type:
#:<syntaxhighlight lang="shell">sha256sum ~/Downloads/disk-img.tar.gz</syntaxhighlight>
#Compare the calculated checksum with the '''IC-OS installation image checksum''' file downloaded in the previous step. '''Warning:''' Only continue if they are identical, otherwise please post your issue in the [[Node Provider Matrix channel]].
#:Open the Terminal and type: <syntaxhighlight lang="shell">tar xzvf ~/Downloads/disk-img.tar.gz</syntaxhighlight>

===Windows===
#Open PowerShell and type:
#:<syntaxhighlight lang="shell">Get-FileHash -Algorithm SHA256 .\Downloads\disk-img.tar.gz</syntaxhighlight>
#Compare the calculated checksum with the '''IC-OS installation image checksum''' file downloaded in the previous step. '''Warning:''' Only continue if they are identical, otherwise please post your issue in the [[Node Provider Matrix channel]].
#:Open PowerShell and type: <syntaxhighlight lang="shell">tar xzvf .\Downloads\disk-img.tar.gz</syntaxhighlight>

==5. Create Bootable USB Stick ==
===Mac OS X===
# Open the Terminal and type:
#:<syntaxhighlight lang="shell">diskutil list</syntaxhighlight>
#All available drives should be shown. Identify which device corresponds to your USB stick. You may need to unmount the USB drive:
#:<syntaxhighlight lang="shell">sudo diskutil unmount /dev/YOUR_USB_DEVICE_MOUNTED_PARTITION # E.g. /dev/disk4s1</syntaxhighlight>
#The file path is an example. Use the absolute path to the downloaded image. '''Warning:''' You risk losing your own data if you specify a wrong device.
#:<syntaxhighlight lang="shell">sudo dd if=/Users/YOUR_USER_NAME/Downloads/disk.img of=/dev/YOUR_USB_DEVICE bs=1M</syntaxhighlight>If you get a “device is busy” error from the dd command, you can try running the following command to unmount all of the partitions on the disk, then re-run the dd command:
#:<syntaxhighlight lang="shell">sudo diskutil unmountDisk /dev/YOUR_USB_DEVICE # E.g. /dev/disk4</syntaxhighlight>

=== Linux / Ubuntu===
#Open the Terminal and type
#:<syntaxhighlight lang="shell">blkid</syntaxhighlight>
# All available drives should be shown. Identify which device corresponds to your USB stick. You may need to unmount the USB drive:
#:<syntaxhighlight lang="shell">sudo diskutil unmount /dev/YOUR_USB_DEVICE_MOUNTED_PARTITION # E.g. /dev/sdb1</syntaxhighlight>
#Replace ''/dev/YOUR_USB_DEVICE'' with the device that corresponds to your USB stick. '''Warning:''' You risk losing your own data if you specify a wrong drive.
#:<syntaxhighlight lang="shell">sudo dd if=~/Downloads/disk.img of=/dev/YOUR_USB_DEVICE bs=1M</syntaxhighlight>

===Windows===
#Download and install [https://rufus.ie/en/ Rufus Portable]
#Start Rufus
#Select the USB stick under device and select the previously downloaded IC-OS disk image and press start
#:[[File:05.png|480px|screenshot]]
#You may see some warnings. Make sure you don't have any other USBs in your computer and chose OK
#:[[File:06.png|480px|screenshot]]
#:[[File:07.png|480px|screenshot]]
# The "Ready" bar will go from left to right as it completes.

==6. Add configuration==

===A. Open Config.ini in a text editor===

===='''Mac OS X'''====

#Open Finder. You should now be able to see the CONFIG partition. If it's not visible, remove the USB and insert it again.
#:[[File:mac_01.png|580px|screenshot]]
#Double-click <code>config.ini</code> to open it in TextEdit.

===='''Linux'''====

#Open the File Manager. You should now be able to see the CONFIG partition. If it's not visible, remove the USB and insert it again.
#:[[File:linux_01.png|580px|screenshot]]
#Double-click <code>config.ini</code> to open it in KWrite.

===='''Windows'''====

#Open the Disk Management utility with a right click on the Start menu
#:[[File:09-b.png|300px|screenshot]]#:
#Right click the CONFIG partition
# Select Change drive letter or paths...
#:[[File:10-b.png|780px|screenshot]]
# Select any letter from the drop-down list
#:[[File:11-b.png|480px|screenshot]]
#Click OK.
#You should now be able to see the CONFIG partition in your Windows Explorer. Select the <code>config.ini</code> configuration file
#:[[File:12-b.png|780px|screenshot]]
#Click on Edit to open it.

===B. Edit Config.ini===

#Insert your IPv6 prefix, subnet and gateway.
#:[[File:Edit config ini.png|580px|screenshot]]
#:*The IPv6 prefix should consist of four groups of hexadecimal digits, separated by colons (':'). Each group can contain up to four hex digits.
#:*For example, a valid prefix could look like this: <code>2a00:fb01:400:100</code>
#:*'''Important:'''
#:**The prefix should not have a trailing ':'
#:**IPv6 CIDR notation allows for a double colon ('::') to represent consecutive groups of zeroes in an address. However, the prefix configuration in this context does '''not''' support '::'. Hence, the '::' shorthand should '''not''' be used: even if some groups are all zeros, they must be explicitly written out.
#Save the changes.
#:* If you have trouble saving this file directly, you may need to save to a known location first, then copy the file into place.
#:* If you need help, please do not hesitate to post your issue in the [[Node Provider Matrix channel]].
#:*:[[File:mac_03.png|580px|screenshot]]

=== C. Copy Node Operator private key to config partition===

#Copy <code>node_operator_private_key.pem</code> (created in [[Node Provider Onboarding#6. Setup the Node Operator keys|Node Provider Onboarding step 6]]) to the <code>CONFIG</code> partition. This file should have the name <code>node_operator_private_key.pem</code>, and sit next to <code>config.ini</code>, NOT inside the <code>ssh_authorized_keys</code> folder.

==7. Connect Crash Cart==
#In order to configure the UEFI and initiate the installation of the IC-OS, please connect a crash cart to the physical machine.
#Plug-in the VGA/Video, keyboard and IC-OS USB stick
#:[[File:08.png|580px|screenshot]]

==8. UEFI Setup and Boot Menu==

Make sure that server date/time is set to UTC (Universal Time Coordinated)

Use the related page below to set up the BIOS/UEFI according to your hardware vendor.

*[[Node Provider Machine Hardware Guide#Gen 2 Node Machine requirements|Gen2 hardware]]
**[[IC-OS Installation - UEFI Configuration - Gen2 Dell]]
** [[IC-OS Installation - UEFI Configuration - Gen2 Supermicro]]
** [[IC-OS Installation - UEFI Configuration - Gen2 Gigabyte]]
**[[IC-OS Installation - UEFI Configuration - Gen2 ASUS]]
*[[Node Provider Machine Hardware Guide#Gen 1 Node Machine requirements|Gen1 hardware]]
**[[IC-OS Installation - UEFI Configuration - Gen1 Dell|IC-OS Installation - UEFI Configuration - Gen1 Dell (Poweredge R6525)]]
**[[IC-OS Installation - UEFI Configuration - Gen1 Supermicro]]
***
'''Important:''' Do NOT enable the RAID bios setting. Doing so will cause issues with the IC-OS installation.

Resume from this point when you are finished configuring the BIOS.

==9. IC-OS Installation==
#Please wait while the USB Installer is booting up. This process can take up to 3 minutes.
#:[[File:35-sm.png|580px|screenshot]]
#The IC-OS installation starts. Please keep an eye on the progress. This part can take up to 10 minutes. Please remember to check the [[Possible Node Onboarding Errors]] page if you encounter any errors.
#:[[File:36-sm.png|580px|screenshot]]
#If the installation finished successfully, it will initiate a reboot.
#:[[File:38-sm.png|580px|screenshot]] 
==10. First Boot==
Please remember to check the [[Possible Node Onboarding Errors]] page if you encounter any errors onboarding.
#Once you see this message, you may unplug the USB stick and VGA/Video.
#:[[File:Node join message.png|580px|screenshot]]

Congratulations! Your machine successfully joined the Internet Computer! The machine has joined the IC and the Node Provider will start receiving rewards!

==11. Verify node onboarding==

#Verify that your node was successfully onboarded by checking its status on the [https://dashboard.internetcomputer.org/ dashboard] is set to either “Awaiting Subnet” or “Active in Subnet”.
#*The dashboard can be searched by your Node Provider principal. There, you should see the Node ID of your node (Node ID is outputted in step 10).
#*If the status of your node is not either “Awaiting Subnet” or “Active in Subnet”, or if it is not listed under your Node Provider principal, you should contact the [[Node Provider Matrix channel]] for assistance.
#*:[[File:Node onboarding verification.png|680px|screenshot]]

Node Provider Networking Guide

2023-12-20T22:41:16Z

Gary.mcelroy: Added note about LACP bonding

Node Provider Networking Guide

2023-11-23T13:51:13Z

Gary.mcelroy: /* Network Cabling */ Change 2 ports requirement, other verbiage.

Node Deployment Guide (with an HSM)

2023-07-13T19:34:59Z

Gary.mcelroy: /* 1. Download installation image */

This runbook covers all steps necessary to install the Internet Computer Operating System (IC-OS) on [[Node Provider Machine Hardware Guide#Gen 2 Node Machine requirements|Gen-2 hardware]] using the legacy NitroKey HSM instructions. To use the current instructions, follow the [[IC OS Installation Runbook]].

The physical machine is expected to be racked and stacked according to its respective manual.

In case you encounter any issues during the installation process, check the [[Possible Node Onboarding Errors]] page. Otherwise, post your issue in the [https://app.element.io/#/room/#ic-node-providers:matrix.org IC Node Provider Matrix channel].

Many thanks for your efforts in building the Internet Computer.

== Requirements ==
* A USB (3.0 speed that can hold at least 4GB) to put the image file on (Faster USBs will allow the process to go much faster.)
* The NitroKey HSM for your data center.
* [Optional] A USB hub
** This is helpful at some data centers for simultaneously connecting keyboard, mouse, Nitrokey, etc..

== 1. Download installation image ==
Download the latest release of the '''IC-OS USB Installer Image''' and the '''corresponding checksum''' from the [https://dashboard.internetcomputer.org/releases Internet Computer Dashboard Releases].

== 2. Verify checksum and unarchive file ==
=== Mac OS X ===
# Open the Terminal and type:
#: <syntaxhighlight lang="shell">shasum -a 256 ~/Downloads/disk-img.tar.gz</syntaxhighlight>
# Compare the calculated checksum with the file downloaded in the previous step. '''Warning:''' Only continue if they are identical, otherwise please post your issue in the [https://app.element.io/#/room/#ic-node-providers:matrix.org IC Node Provider Matrix channel].
#: Open the Terminal and type: <syntaxhighlight lang="shell">tar xzvf ~/Downloads/disk-img.tar.gz</syntaxhighlight>

=== Linux / Ubuntu ===
# Open the Terminal and type:
#: <syntaxhighlight lang="shell">sha256sum ~/Downloads/disk-img.tar.gz</syntaxhighlight>
# Compare the calculated checksum with the file downloaded in the previous step. '''Warning:''' Only continue if they are identical, otherwise please post your issue in the [https://app.element.io/#/room/#ic-node-providers:matrix.org IC Node Provider Matrix channel].
#: Open the Terminal and type: <syntaxhighlight lang="shell">tar xzvf ~/Downloads/disk-img.tar.gz</syntaxhighlight>

=== Windows ===
# Open PowerShell and type:
#: <syntaxhighlight lang="shell">Get-FileHash -Algorithm SHA256 .\Downloads\disk-img.tar.gz</syntaxhighlight>
# Compare the calculated checksum with the file downloaded in the previous step. '''Warning:''' Only continue if they are identical, otherwise please post your issue in the [https://app.element.io/#/room/#ic-node-providers:matrix.org IC Node Provider Matrix channel].
#: Open PowerShell and type: <syntaxhighlight lang="shell">tar xzvf .\Downloads\disk-img.tar.gz</syntaxhighlight>

== 3. Create Bootable USB Stick ==
=== Mac OS X ===
# Open the Terminal and type:
#:<syntaxhighlight lang="shell">diskutil list</syntaxhighlight>
# All available drives should be shown. Identify which device corresponds to your USB stick. You may need to unmount the USB drive:
#:<syntaxhighlight lang="shell">sudo diskutil unmount /dev/YOUR_USB_DEVICE_MOUNTED_PARTITION # E.g. /dev/disk4s1</syntaxhighlight>
# The file path is an example. Use the absolute path to the downloaded image. '''Warning:''' You risk losing your own data if you specify a wrong device.
#:<syntaxhighlight lang="shell">sudo dd if=/Users/YOUR_USER_NAME/Downloads/disk.img of=/dev/YOUR_USB_DEVICE bs=1M</syntaxhighlight>

=== Linux / Ubuntu ===
# Open the Terminal and type
#:<syntaxhighlight lang="shell">blkid</syntaxhighlight>
# All available drives should be shown. Identify which device corresponds to your USB stick. You may need to unmount the USB drive:
#:<syntaxhighlight lang="shell">sudo diskutil unmount /dev/YOUR_USB_DEVICE_MOUNTED_PARTITION # E.g. /dev/sdb1</syntaxhighlight>
# Replace ''/dev/YOUR_USB_DEVICE'' with the device that corresponds to your USB stick. '''Warning:''' You risk losing your own data if you specify a wrong drive.
#:<syntaxhighlight lang="shell">sudo dd if=~/Downloads/disk.img of=/dev/YOUR_USB_DEVICE bs=1M</syntaxhighlight>

=== Windows ===
# Download and install [https://rufus.ie/en/ Rufus Portable]
# Start Rufus
# Select the USB stick under device and select the previously downloaded IC-OS disk image and press start
#: [[File:05.png|480px|screenshot]]
# You may see some warnings. Make sure you don't have any other USBs in your computer and chose OK
#: [[File:06.png|480px|screenshot]]
#: [[File:07.png|480px|screenshot]]
# The "Ready" bar will go from left to right as it completes.

== 4. Add configuration ==
=== Mac OS X ===
# Open Finder. You should now be able to see the CONFIG partition. If it's not visible, remove the USB and insert it again.
#: [[File:mac_01.png|580px|screenshot]]
# Double-click to open it in TextEdit.
# Insert your IPv6 prefix, subnet and gateway.
#: [[File:mac_02.png|580px|screenshot]]
# Once done, don’t forget to save the changes. If you need help, please do not hesitate to post your issue in the [https://app.element.io/#/room/#ic-node-providers:matrix.org IC Node Provider Matrix channel].
#:[[File:mac_03.png|580px|screenshot]]

=== Windows ===
# Open the Disk Management utility with a right click on the Start menu
#: [[File:09-b.png|300px|screenshot]]#:
# Right click the CONFIG partition
# Select Change drive letter or paths...
#: [[File:10-b.png|780px|screenshot]]
# Select any letter from the drop-down list
#: [[File:11-b.png|480px|screenshot]]
# Click OK.
# You should now be able to see the CONFIG partition in your Windows Explorer. Select the config.ini configuration file
#: [[File:12-b.png|780px|screenshot]]
# Click on Edit to open it.
# Insert your IPv6 prefix, subnet and gateway.
#: [[File:13-b.png|580px|screenshot]]
# Once done, don’t forget to save the changes. If you need help, please do not hesitate to post your issue in the [https://app.element.io/#/room/#ic-node-providers:matrix.org IC Node Provider Matrix channel].
#:[[File:14-b.png|580px|screenshot]]
# If onboarding without a NitroKey HSM, copy <code>node_operator_private_key.pem</code> (created in [[Node Provider Onboarding#7.%20Setup%20the%20Node%20Operator%20keys|Node Provider Onboarding step 7]]) to the <code>CONFIG</code> partition. This file should have the name <code>node_operator_private_key.pem</code>, and sit next to <code>config.ini</code>, NOT inside the <code>ssh_authorized_keys</code> folder.

=== Linux ===
# Open the File Manager. You should now be able to see the CONFIG partition. If it's not visible, remove the USB and insert it again.
#: [[File:linux_01.png|580px|screenshot]]
# Double-click to open it in KWrite.
# Insert your IPv6 prefix, subnet and gateway.
#: [[File:linux_02.png|580px|screenshot]]
# Once done, don’t forget to save the changes. If you need help, please do not hesitate to post your issue in the [https://app.element.io/#/room/#ic-node-providers:matrix.org IC Node Provider Matrix channel].
#:[[File:linux_03.png|580px|screenshot]]

== 5. Connect Crash Cart ==
# In order to configure the UEFI and initiate the installation of the IC-OS, please connect a crash cart to the physical machine.
# Plug-in the VGA/Video, keyboard and IC-OS USB stick
#: [[File:08.png|580px|screenshot]]

== 6. UEFI Setup and Boot Menu ==
Use the related page below to set up the BIOS/UEFI according to your hardware vendor.

* [[IC OS Installation - UEFI Configuration - Dell]]
* [[IC OS Installation - UEFI Configuration - Supermicro]]


Resume from this point when you are finished.

== 7. IC-OS Installation ==
# Please wait while the USB Installer is booting up. This process can take up to 3 minutes.
#: [[File:35-sm.png|580px|screenshot]]
# The IC-OS installation starts. Please keep an eye on the progress. This part can take up to 10 minutes. Please remember to check the [[Possible Node Onboarding Errors]] page if you encounter any errors.
#:[[File:36-sm.png|580px|screenshot]]
# Once you get asked to insert the HSM, please remove the keyboard and instead insert the HSM USB device.
#:[[File:37-sm.png|580px|screenshot]]
# If the installation finished successfully, it will initiate a reboot. '''Please do not unplug the USB stick or HSM USB''' device at this point.
#:[[File:38-sm.png|580px|screenshot]]

== 8. First Boot ==
Please remember to check the [[Possible Node Onboarding Errors]] page if you encounter any errors onboarding.
'''Do NOT re-try the onboarding after proceeding to this section, as this can cause duplication within the registry.'''
# The first boot of the IC-OS still requires the HSM USB device. Please wait until further instructions. This step can take up to 2 minutes.
#:[[File:39-sm.png|580px|screenshot]]
# Once you see this message, you may unplug the HSM USB device, USB stick and VGA/Video. Your machine successfully joined the Internet Computer!
#:[[File:40-sm.png|580px|screenshot]]

Congratulations! Your machine successfully joined the Internet Computer! Again, once you see this message, '''do NOT re-try the onboarding after proceeding to this section, as this can cause duplication within the registry.''' The machine has joined the IC and the node provider will start receiving rewards!

== 9. Verify node onboarding ==

# Return to the [[Node Provider Onboarding#10. Onboard nodes|Node Provider Onboarding]] instructions to verify that your nodes were successfully onboarded

Troubleshooting Node Deployment Errors

2023-07-13T19:34:10Z

Gary.mcelroy: /* Suggested Solutions */

This page has some error codes that may display as you are onboarding your nodes. Please review the examples, causes, and proposed solutions before contacting support.

If you need Dell to service your machine, then these links will assist in [[Retrieving a Dell TSR Log|retrieving a Dell TSR Log]] and in resetting the iDRAC password.

If you encounter an error not listed here, please capture a screenshot and detail when it happened, which stage in onboarding you were at, the status of any lights on the server, and any other relevant details. Post your issue and accompanying screenshots in the [https://app.element.io/#/room/#ic-node-providers:matrix.org IC Node Provider Matrix channel].

== Orchestrator Started ==
This message is not an error, nor is it confirmation that the node is running properly.

* Check [https://dashboard.internetcomputer.org/ the dashboard] to check the status of that particular node. (Status explanations are [[Node Provider Troubleshooting#Node%20Status%20on%20the%20Dashboard|here]].) Use the principal ID that was assigned to the node when it was onboarded to identify it. Use the principal ID that was assigned to the node when it was onboarded to identify it.
* If the node is not visible on the dashboard then it has not registered with the Internet Computer.
** If you have recently installed a current IC-OS image, then you can try inserting the HSM and/or a reboot to see if it joins. This would work if the IC-OS installation was successful and only the registration and joining was interrupted.
** If you have ''not'' recently installed a current IC-OS image, then do ''not'' insert the HSM. You do not want the node to rejoin with an old IC-OS image, as it will only fail again. Instead, you should consider [[Updating Firmware|upgrading the firmware]] if it is running on old versions, and then redeploy the node with [[Node Provider Documentation|a fresh/current IC-OS image]] (which will assign a new principal to the node so that you can identify it in the dashboard.)

== General Troubleshooting ==
During the IC OS installation, you may hit enter to obtain console access to troubleshoot any issues you are encountering. You can also hit enter at the error page in order to access the console.

Once you have console access, in order to stop the IC OS installation service, enter:

<syntaxhighlight lang="shell">$ systemctl stop setupos</syntaxhighlight>

== Missing Drives ==
==== Example Error ====
--------------------------------------------------------------------------------
INTERNET COMPUTER - SETUP - FAILED
--------------------------------------------------------------------------------

Please contact the Node Provider Matrix channel for support.

--------------------------------------------------------------------------------
ERROR
--------------------------------------------------------------------------------

Not enough drives found. Are all drives correctly installed?

--------------------------------------------------------------------------------
ERROR
--------------------------------------------------------------------------------

'''Another version of it might say "Aggregate Disk size does not meet requirements"'''

==== Common Causes ====
This error means that the IC-OS installation medium could not detect all required drives. This is a common issue, even if you believe that all drives are installed correctly. Some of them may not be functioning properly, or may not be fully seated into the chassis.

==== Suggested Solutions ====
Check that all drives are fully seated and installed correctly, or install the required number of drives. You may be able to check the drives for indication LEDs to see which may not be installed or functioning correctly.

== Invalid CPU Configuration ==
==== Example Error ====

--------------------------------------------------------------------------------
INTERNET COMPUTER - SETUP - FAILED
--------------------------------------------------------------------------------

Please contact the Node Provider Matrix channel for support.

--------------------------------------------------------------------------------
ERROR
--------------------------------------------------------------------------------

Number of threads (16/32) does NOT meet system requirements.

--------------------------------------------------------------------------------
ERROR
--------------------------------------------------------------------------------

==== Common Causes ====
Issues related to CPU capability usually mean that the CPUs are not configured correctly in the system BIOS.

==== Suggested Solutions ====
Please check that BIOS settings are configured correctly. It may be helpful to reset all settings to factory defaults, and go through the BIOS configuration again.
== Unable to Reach Internet ==
==== Example Error ====

--------------------------------------------------------------------------------
INTERNET COMPUTER - SETUP - FAILED
--------------------------------------------------------------------------------

Please contact the Node Provider Matrix channel for support.

--------------------------------------------------------------------------------
ERROR
--------------------------------------------------------------------------------

Unable to ping IPv6 gateway.

--------------------------------------------------------------------------------
ERROR
--------------------------------------------------------------------------------

==== Common Causes ====
This error means that the node is not able to communicate with the network properly. This can be due to a misconfigured network configuration, or due to issues somewhere between the node and the rest of the internet.

==== Suggested Solutions ====
Please try to capture any output that is displayed before this error shows. For example:
* Printing user defined network settings...
IPv6 Prefix : XXX
IPv6 Subnet : XXX
IPv6 Gateway: XXX

* Printing system's network settings...
IPv6 Prefix : XXX
IPv6 Subnet : XXX
IPv6 Gateway: XXX

* Printing IPv6 addresses...
SetupOS: XXX
HostOS : XXX
GuestOS: XXX

Please compare this, and the initial configuration, to what you expect. If this configuration does not match, please update the initial configuration, and try again.

If this does match the expected configuration, please attempt to diagnose any machines between this node and the rest of the internet. This could be due to improper firewall configuration, or an issue with the data center’s network. If all configuration looks correct, please attempt to reboot any machines between this node and the rest of the internet. In most cases, this would be a firewall. Rebooting the firewall - even if it seems to be operating correctly - has resolved this issue many times.

== Unable to setup PV ==
==== Example Error ====

--------------------------------------------------------------------------------
INTERNET COMPUTER - SETUP - FAILED
--------------------------------------------------------------------------------

Please contact the Node Provider Matrix channel for support.

--------------------------------------------------------------------------------
ERROR
--------------------------------------------------------------------------------

Unable to setup PV on drive '/dev/nvme8n1'.

--------------------------------------------------------------------------------
ERROR
--------------------------------------------------------------------------------

==== Common Causes ====
This error means that the node is able to recognize that a drive is installed, but is unable to write to it. This could indicate that there is a hardware issue with the drive.

==== Suggested Solutions ====
Please try to remove and re-install all drives, before attempting to install the node again. It may be helpful to independently verify that each drive is functioning correctly.
== Long Wait on Node Join ==
==== Example Error ====
Orchestrator started.
Starting node registration.
Attaching HSM.
Sending add_node request.

But not:
Join request successful!
You may now safely remove the HSM.

==== Common Causes ====
The node has installed and launched successfully, but is unable to join the network. This could be due to an out-of-date IC-OS installation image, trouble contacting the NNS, or node installation limits on the network.

==== Suggested Solutions ====
Please verify that a recent [https://dashboard.internetcomputer.org/releases IC-OS installation] image version is being used, and check https://dashboard.internetcomputer.org/ to see how many nodes are currently registered under your Node Provider. If there are more nodes listed than expected, or if there are multiple nodes overlapping, please have any extra nodes removed from the network before attempting to install again. This can be caused if multiple installations have been performed on the same hardware, without cleaning up the records from the network.

* [[Internet Computer wiki|Return to Wiki Home]]

Node Deployment Guide

2023-07-13T19:24:43Z

Gary.mcelroy: /* 1. Download installation image */ Replaced link for download - dynamic download links!

This runbook covers all steps necessary to install the Internet Computer Operating System (IC-OS) on [[Node Provider Machine Hardware Guide#Gen 2 Node Machine requirements|Gen-2 hardware]]. Gen-1 hardware Node Providers should use the [[Node Provider Documentation#Onboarding for accepted node providers|Gen-1 runbooks]].

The physical machine is expected to be racked and stacked according to its respective manual.

If you encounter issues through any of these steps, check the [[Node Provider Troubleshooting]] page. If that does not solve your problem, you are encouraged to ask for assistance in the [[Node Provider Matrix channel]].

Note: the following instructions are for onboarding nodes without using a NitroKey HSM. If you wish to follow the legacy procedure to onboard using a NitroKey HSM, follow the [[NitroKey HSM installation runbook]].

== Requirements ==
* A USB (3.0 speed that can hold at least 4GB) to put the image file on (Faster USBs will allow the process to go much faster.)
* The <code>node_operator_private_key.pem</code> for your data center (Acquired from [[Node Provider Onboarding#5. Setup the Node Operator keys|Node Provider Onboarding step 7]])
* It is recommended that each server have a label with the BMC's MAC address for ease of identification in future dashboard upgrades.

== 1. Download installation image ==
Download the latest release of the '''IC-OS USB Installer Image''' and the '''corresponding checksum''' from the [https://dashboard.internetcomputer.org/releases Internet Computer Dashboard Releases].

== 2. Verify checksum and unarchive file ==
=== Mac OS X ===
# Open the Terminal and type:
#: <syntaxhighlight lang="shell">shasum -a 256 ~/Downloads/disk-img.tar.gz</syntaxhighlight>
# Compare the calculated checksum with the '''IC-OS installation image checksum''' file downloaded in the previous step. '''Warning:''' Only continue if they are identical, otherwise please post your issue in the [[Node Provider Matrix channel]].
#: Open the Terminal and type: <syntaxhighlight lang="shell">tar xzvf ~/Downloads/disk-img.tar.gz</syntaxhighlight>

=== Linux / Ubuntu ===
# Open the Terminal and type:
#: <syntaxhighlight lang="shell">sha256sum ~/Downloads/disk-img.tar.gz</syntaxhighlight>
# Compare the calculated checksum with the '''IC-OS installation image checksum''' file downloaded in the previous step. '''Warning:''' Only continue if they are identical, otherwise please post your issue in the [[Node Provider Matrix channel]].
#: Open the Terminal and type: <syntaxhighlight lang="shell">tar xzvf ~/Downloads/disk-img.tar.gz</syntaxhighlight>

=== Windows ===
# Open PowerShell and type:
#: <syntaxhighlight lang="shell">Get-FileHash -Algorithm SHA256 .\Downloads\disk-img.tar.gz</syntaxhighlight>
# Compare the calculated checksum with the '''IC-OS installation image checksum''' file downloaded in the previous step. '''Warning:''' Only continue if they are identical, otherwise please post your issue in the [[Node Provider Matrix channel]].
#: Open PowerShell and type: <syntaxhighlight lang="shell">tar xzvf .\Downloads\disk-img.tar.gz</syntaxhighlight>

== 3. Create Bootable USB Stick ==
=== Mac OS X ===
# Open the Terminal and type:
#:<syntaxhighlight lang="shell">diskutil list</syntaxhighlight>
# All available drives should be shown. Identify which device corresponds to your USB stick. You may need to unmount the USB drive:
#:<syntaxhighlight lang="shell">sudo diskutil unmount /dev/YOUR_USB_DEVICE_MOUNTED_PARTITION # E.g. /dev/disk4s1</syntaxhighlight>
# The file path is an example. Use the absolute path to the downloaded image. '''Warning:''' You risk losing your own data if you specify a wrong device.
#:<syntaxhighlight lang="shell">sudo dd if=/Users/YOUR_USER_NAME/Downloads/disk.img of=/dev/YOUR_USB_DEVICE bs=1M</syntaxhighlight>

=== Linux / Ubuntu ===
# Open the Terminal and type
#:<syntaxhighlight lang="shell">blkid</syntaxhighlight>
# All available drives should be shown. Identify which device corresponds to your USB stick. You may need to unmount the USB drive:
#:<syntaxhighlight lang="shell">sudo diskutil unmount /dev/YOUR_USB_DEVICE_MOUNTED_PARTITION # E.g. /dev/sdb1</syntaxhighlight>
# Replace ''/dev/YOUR_USB_DEVICE'' with the device that corresponds to your USB stick. '''Warning:''' You risk losing your own data if you specify a wrong drive.
#:<syntaxhighlight lang="shell">sudo dd if=~/Downloads/disk.img of=/dev/YOUR_USB_DEVICE bs=1M</syntaxhighlight>

=== Windows ===
# Download and install [https://rufus.ie/en/ Rufus Portable]
# Start Rufus
# Select the USB stick under device and select the previously downloaded IC-OS disk image and press start
#: [[File:05.png|480px|screenshot]]
# You may see some warnings. Make sure you don't have any other USBs in your computer and chose OK
#: [[File:06.png|480px|screenshot]]
#: [[File:07.png|480px|screenshot]]
# The "Ready" bar will go from left to right as it completes.

== 4. Add configuration ==

=== A. Open Config.ini in a text editor ===

==== '''Mac OS X''' ====

# Open Finder. You should now be able to see the CONFIG partition. If it's not visible, remove the USB and insert it again.
#:[[File:mac_01.png|580px|screenshot]]
# Double-click <code>config.ini</code> to open it in TextEdit.

===='''Linux'''====

# Open the File Manager. You should now be able to see the CONFIG partition. If it's not visible, remove the USB and insert it again.
#:[[File:linux_01.png|580px|screenshot]]
# Double-click <code>config.ini</code> to open it in KWrite.

==== '''Windows''' ====

# Open the Disk Management utility with a right click on the Start menu
#:[[File:09-b.png|300px|screenshot]]#:
# Right click the CONFIG partition
# Select Change drive letter or paths...
#:[[File:10-b.png|780px|screenshot]]
# Select any letter from the drop-down list
#:[[File:11-b.png|480px|screenshot]]
# Click OK.
# You should now be able to see the CONFIG partition in your Windows Explorer. Select the <code>config.ini</code> configuration file
#:[[File:12-b.png|780px|screenshot]]
# Click on Edit to open it.

=== B. Edit Config.ini ===

# Insert your IPv6 prefix, subnet and gateway.
#:[[File:Edit config ini.png|580px|screenshot]]
#:* The IPv6 prefix should consist of four groups of hexadecimal digits, separated by colons (':'). Each group can contain up to four hex digits.
#:* For example, a valid prefix could look like this: <code>2a00:fb01:400:100</code>
#:*'''Important:'''
#:** The prefix should not have a trailing ':'
#:** IPv6 CIDR notation allows for a double colon ('::') to represent consecutive groups of zeroes in an address. However, the prefix configuration in this context does '''not''' support '::'. The '::' shorthand should '''not''' be used. Even if some groups are all zeros, they must be explicitly written out.
# Save the changes.
#:* If you have trouble saving this file directly, you may need to save to a known location first, then copy the file into place.
#:* If you need help, please do not hesitate to post your issue in the [[Node Provider Matrix channel]].
#:*:[[File:mac_03.png|580px|screenshot]]

=== C. Copy Node Operator private key to config partition ===

# Copy <code>node_operator_private_key.pem</code> (created in [[Node Provider Onboarding#7. Setup the Node Operator keys|Node Provider Onboarding step 7]]) to the <code>CONFIG</code> partition. This file should have the name <code>node_operator_private_key.pem</code>, and sit next to <code>config.ini</code>, NOT inside the <code>ssh_authorized_keys</code> folder.

== 5. Connect Crash Cart ==
# In order to configure the UEFI and initiate the installation of the IC-OS, please connect a crash cart to the physical machine.
# Plug-in the VGA/Video, keyboard and IC-OS USB stick
#: [[File:08.png|580px|screenshot]]

== 6. UEFI Setup and Boot Menu ==
Use the related page below to set up the BIOS/UEFI according to your hardware vendor.

* [[IC OS Installation - UEFI Configuration - Dell]]
* [[IC OS Installation - UEFI Configuration - Supermicro]]
'''Important:''' Do NOT enable the RAID bios setting. Doing so will cause issues with the IC-OS installation.

Resume from this point when you are finished configuring the BIOS.

== 7. IC-OS Installation ==
# Please wait while the USB Installer is booting up. This process can take up to 3 minutes.
#: [[File:35-sm.png|580px|screenshot]]
# The IC-OS installation starts. Please keep an eye on the progress. This part can take up to 10 minutes. Please remember to check the [[Possible Node Onboarding Errors]] page if you encounter any errors.
#:[[File:36-sm.png|580px|screenshot]]
# If the installation finished successfully, it will initiate a reboot.
#:[[File:38-sm.png|580px|screenshot]] 
== 8. First Boot ==
Please remember to check the [[Possible Node Onboarding Errors]] page if you encounter any errors onboarding.
'''Do NOT re-try the onboarding after proceeding to this section, as this can cause duplication within the registry.'''
# Once you see this message, you may unplug the USB stick and VGA/Video. You can ignore the message to remove the HSM, as you did not use an HSM to onboard your nodes.
#:[[File:40-sm.png|580px|screenshot]]

Congratulations! Your machine successfully joined the Internet Computer! Again, once you see this message, '''do NOT re-try the onboarding after proceeding to this section, as this can cause duplication within the registry.''' The machine has joined the IC and the Node Provider will start receiving rewards!

== 9. Verify node onboarding ==

# Verify that your node was successfully onboarded by checking its status on the [https://dashboard.internetcomputer.org/ dashboard] is set to either “Awaiting Subnet” or “Active in Subnet”.
#* The dashboard can be searched by your Node Provider principal. There, you should see the Node ID of your node (Node ID outputted in step 8).
#* If the status of your node is not either “Awaiting Subnet” or “Active in Subnet”, or if it is not listed under your Node Provider principal, you should contact the [[Node Provider Matrix channel]] for assistance.
#*:[[File:Dashboard-node-verification.png|thumb|998x998px]]

Node Provider Machine Hardware Guide

2023-07-11T20:10:19Z

Gary.mcelroy: /* Purchasing Hardware Guide */ Edits to "purchase hardware"

Node Providers operate one or more node machines than run in the Internet Computer network.

Gen1 hardware requirements have been used by Node Providers to set up node machines during the Genesis launch.

The Gen2 hardware requirements have been defined for the further growth of the Internet Computer network. The specifications for the Gen2 node machines are generic (instead of vendor specific) and support VM memory encryption and attestation, which will be needed in future features on the Internet Computer.

While Gen2 Node Providers are only strictly required to follow the generic specifications '''(the IC-OS installation will fail if the generic specifications are not met)''', it is strongly recommended for Node Providers to purchase one of the validated configurations listed below.
== Gen 2 Node Machine requirements ==
==== Generic specification ====
{| class="wikitable"
|-
| Dual Socket AMD EPYC Milan CPU - Recommended: [https://en.wikichip.org/wiki/amd/epyc/7313 7313] (16C/32T 3 Ghz)
[https://en.wikichip.org/wiki/amd/epyc#7003_Series_.28Zen_3.29 optionally] 7343, 7373, 73F3
|-
| 16x 32GB RDIMM, 3200MT/s, Dual Rank
|-
| 5x 6.4TB NVMe Mixed Mode (DWPD >= 3)
|-
| Dual Port 10G SFP or BASE-T
|-
| TPM 2.0
|}
'''Important:''' Do NOT order a RAID controller for your node machine. If a RAID controller is included in your setup, it will cause issues with the installation of IC-OS on your machine.

=== Dfinity-validated configurations ===
DFINITY has [https://forum.dfinity.org/t/draft-motion-proposal-new-hardware-specification-and-remuneration-for-ic-nodes/14202/14?u=garym validated] the following Gen2 hardware configurations.
==== Dell ====
{| class="wikitable"
|-
| 2 || AMD EPYC 7343 3.2GHz, 16C/32T, 128M Cache (190W)
|-
| 16 || 32GB RDIMM, 3200MT/s, Dual Rank 16Gb BASE x8
|-
| 5 || 6.4TB Enterprise NVMe Mixed Use AG Drive U.2 Gen4 with carrier
|-
| 1 || PowerEdge R6525 Motherboard, with 2 x 1Gb Onboard LOM (BCM5720)MLK V2
|-
| 2 || Dual, Hot-plug, Redundant Power Supply (1+1) 1100W, Mixed Mode Titanium
|-
| 1 || Intel X710 Dual Port 10GbE SFP+, OCP NIC 3.0
|-
| 1 || Trusted Platform Module 2.0 V3
|}

==== ASUS ====
{| class="wikitable"
|-
| 2 || AMD EPYC 7313 (3,00 GHz, 16-Core, 128 MB)
|-
| 16 || 32GB ECC Reg ATP DDR4 3200 RAM
|-
| 5 || 6.4 TB NVMe Kioxia SSD 3D-NAND TLC U.3 (Kioxia CM6-V)
|-
| 1 || Asus Mainboard KMPP-D32 Series (without OCP 3.0, without Pike)
|-
| 2 || 1600 Watt redundant PSU
|-
| 1 || Broadcom 25 Gigabit P225P SFP28 Dual Port Network Card
|-
|0
|TPM 2.0*
|}
<nowiki>*</nowiki> Note Gen2 machines require TPM 2.0 hardware but this specific test machine did not include it.

==== Supermicro ====
{| class="wikitable"
|-
| 2 || AMD EPYC 7313 (3,00 GHz, 16-Core, 128 MB)
|-
| 16 || 32GB ECC Reg DDR4 3200 RAM
|-
| 5 || 6.4 TB NVMe TLC SSD, PCIe 4.0 x4, U.3 2.5", 3 DWPD
|-
| 1 || SuperMicro H12DSU-iN
|-
| 2 || 1200 Watt redundant PSU
|-
| 1 || Dual Port 10GBase-T Network Adapter Intel® X710-TM4*
|-
|1
|Dual Port 10GbE Network Adapter Intel® X710-TM4, SFP+*
|-
| 1 || Trusted Platform Module 2.0
|}
<nowiki>*</nowiki> Only one dual port network adaptor is required.

==== Currently pending validation: Gigabyte ====
{| class="wikitable"
|-
| 2 || AMD EPYC 7313 (3,00 GHz, 16-Core, 128 MB)
|-
| 16 || 32GB ECC Reg DDR4 3200 RAM
|-
| 5 || 6.4 TB NVMe TLC SSD, PCIe 4.0 x4, U.3 2.5", 3 DWPD
|-
| 1 || Gigabyte MZ92-FS1 Rev. 3.0
|-
| 2 || 1200 Watt redundant PSU
|-
| 1 || Dual Port 1000Base-T Network Adapter Intel® I350-AM2*
|-
|1
|Dual Port 10GbE Network Adapter, Broadcom NetXtreme E-Series*
|-
| 1 || Trusted Platform Module 2.0
|}
<nowiki>*</nowiki> Only one dual port network adaptor is required.

=== Community-validated configurations ===
Node Providers have validated the following Gen2 hardware configurations:

==== Dell PowerEdge R7525 ====
{| class="wikitable"
|2
|AMD 7313 3.0GHz,16C/32T,128M,155W,3200
|-
|16
|32GB RDIMM, 3200MT/s, Dual Rank 16Gb BASE x8
|-
|5
|6.4TB Enterprise NVMe Mixed Use AG Drive U.2 Gen4 with carrier
|-
|1
|PowerEdge R7525 Motherboard, with 2 x 1Gb Onboard LOM (BCM5720)MLK V2
|-
|2
|Dual, Hot-plug, Power Supply Redundant (1+1), 800W, Mixed Mode, NAF
|-
|1
|Intel X710 Dual Port 10GbE SFP+, OCP NIC 3.0
|-
|1
|Trusted Platform Module 2.0 V3
|}

== Gen 1 Node Machine requirements ==
If you're a Node Provider acquiring machines to join the IC, use the Gen2 specifications listed above. This section is for Gen1 Node Providers who joined before 2022.

==== SuperMicro ====
{| class="wikitable"
|-
| 1 || AS - 1023US - TR4
|-
| 2 || Rome 7302 DP/UP 16C/32T 3.0
|-
| 16 || 32GB DDR4-3200 2Rx4 ECC REG DIMM
|-
| 5 || Samsung PM983 3.2TB NVMe PCIE/SATA Hybrid M.2 & 1 PCIE
|-
| 2 || 800W Power Supply
|-
| 1 || Std LP 2-port 10G RJ45, Intel x540
|-
| 5 || Micron 5300 PRO 7.4TB, SATA, 2.5', 3D TLC, .6DWPD (with Caddie)
|-
| 1 || C13/C14 13A Power Cord
|}

==== Dell - type 1 ====
{| class="wikitable"
|-
| 1 || R6525
|-
| 1 || Chassis - Supports Up to 10 NVMe drives, 12 drives total
|-
| 1 || Dual 1 GB on Motherboard
|-
| 3 || Low Profile PCIe Slots
|-
| - || 3 Year Basic NBD Support
|-
| - || iDrac Enterprise
|-
| 1 || Dual port 10GbE Base - T Adapter Broadcom, PCIe Low Profile
|-
| 10 || 3.2TB NVMe, Mixed Use, 2.5" with Carrier
|-
| 16 || 32GB RDIMM (3200MT/s)
|-
| 2 || AMD 7302 3GHz, 16C/32T, 128M, 155W, 3200
|-
| 1 || Single Power Supply (800W)
|-
| 1 || C13-C14, 3M, 125V 15A Power Cored
|-
|}

==== Dell - type 2 ====
{| class="wikitable"
|-
| 1 || R6515
|-
| 1 || 3.5" Chassis with up to 4 Hot-Plug Hard Drives and OS RAID
|-
| 1 || Dual 1 Gb on Motherboard
|-
| 3 || Low Profile PCIe Slots
|-
| 1 || Standard Fan
|-
| - || 3 Year Basic NBD Support
|-
| - || iDrac Enterprise
|-
| 2 || Dual Port 10GbE Base - T Adapter Broadcom, PCIe LOw Profile
|-
| 2 || 480GB SSD SATA Mix Use 6Gbps 512 2.5in Hot-Plug AG Drive, 3.5in
|-
| 4 || 8GB RDIMM, 3200 MT/s, Single Rank
|-
| 1 || AMD EPYC 7232P 3.10GHz, 8C/16T, 64M Cache (120W) DDR4-3200
|-
| 1 || Dual Hot-Plug Redundant Power Supply (1+1), 550W
|-
| 2 || Jumper Cord - C13/C14, .6M, 250V, 13A
|}

== Purchasing Hardware ==
# '''Choose your node machine configuration'''
#* Stick with the configurations above. Anything else is unsupported.
# '''Where to buy?'''
#* [https://www.supermicro.com/en/wheretobuy Supermicro]
#* [https://www.dell.com/en-us/shop/dell-poweredge-servers/sc/servers Dell]
#* [https://www.gigabyte.com/WheretoBuy/Enterprise Gigabyte]
#* [https://servers.asus.com/support/wheretobuy/117 ASUS]
# '''Place your order'''
#* Double check the components meet the generic specs above.

Lead times can be weeks to months. Plan accordingly with respect to [[Node Provider Onboarding|onboarding]].

Node Provider Networking Guide

2023-06-29T20:51:25Z

Gary.mcelroy: /* Requirements */ Added network configuration guidance and explicit limits to guidance

Node Provider Networking Guide

2023-06-29T20:07:54Z

Gary.mcelroy: Added note about dfinity support

Node Provider Networking Guide

2023-06-29T20:06:01Z

Gary.mcelroy: Added CCNA resources

Node Deployment Guide

2023-06-29T19:54:17Z

Gary.mcelroy: /* 1. Download installation image */ Instructions for downloading clarified.

This runbook covers all steps necessary to install the Internet Computer Operating System (IC-OS) on [[Node Provider Machine Hardware Guide#Gen 2 Node Machine requirements|Gen-2 hardware]]. Gen-1 hardware Node Providers should use the [[Node Provider Documentation#Onboarding for accepted node providers|Gen-1 runbooks]].

The physical machine is expected to be racked and stacked according to its respective manual.

If you encounter issues through any of these steps, check the [[Node Provider Troubleshooting]] page. If that does not solve your problem, you are encouraged to ask for assistance in the [[Node Provider Matrix channel]].

Note: the following instructions are for onboarding nodes without using a NitroKey HSM. If you wish to follow the legacy procedure to onboard using a NitroKey HSM, follow the [[NitroKey HSM installation runbook]].

== Requirements ==
* A USB (3.0 speed that can hold at least 4GB) to put the image file on (Faster USBs will allow the process to go much faster.)
* The <code>node_operator_private_key.pem</code> for your data center (Acquired from [[Node Provider Onboarding#5. Setup the Node Operator keys|Node Provider Onboarding step 5. Setup the Node Operator keys]])

== 1. Download installation image ==
Download both the '''IC-OS USB Installer Image''' and the '''corresponding checksum''' from the [[IC OS Download|IC-OS Download Page]]

== 2. Verify checksum and unarchive file ==
=== Mac OS X ===
# Open the Terminal and type:
#: <syntaxhighlight lang="shell">shasum -a 256 ~/Downloads/disk-img.tar.gz</syntaxhighlight>
# Compare the calculated checksum with the file downloaded in the previous step. '''Warning:''' Only continue if they are identical, otherwise please post your issue in the [[Node Provider Matrix channel]].
#: Open the Terminal and type: <syntaxhighlight lang="shell">tar xzvf ~/Downloads/disk-img.tar.gz</syntaxhighlight>

=== Linux / Ubuntu ===
# Open the Terminal and type:
#: <syntaxhighlight lang="shell">sha256sum ~/Downloads/disk-img.tar.gz</syntaxhighlight>
# Compare the calculated checksum with the file downloaded in the previous step. '''Warning:''' Only continue if they are identical, otherwise please post your issue in the [[Node Provider Matrix channel]].
#: Open the Terminal and type: <syntaxhighlight lang="shell">tar xzvf ~/Downloads/disk-img.tar.gz</syntaxhighlight>

=== Windows ===
# Open PowerShell and type:
#: <syntaxhighlight lang="shell">Get-FileHash -Algorithm SHA256 .\Downloads\disk-img.tar.gz</syntaxhighlight>
# Compare the calculated checksum with the file downloaded in the previous step. '''Warning:''' Only continue if they are identical, otherwise please post your issue in the [[Node Provider Matrix channel]].
#: Open PowerShell and type: <syntaxhighlight lang="shell">tar xzvf .\Downloads\disk-img.tar.gz</syntaxhighlight>

== 3. Create Bootable USB Stick ==
=== Mac OS X ===
# Open the Terminal and type:
#:<syntaxhighlight lang="shell">diskutil list</syntaxhighlight>
# All available drives should be shown. Identify which device corresponds to your USB stick. You may need to unmount the USB drive:
#:<syntaxhighlight lang="shell">sudo diskutil unmount /dev/YOUR_USB_DEVICE_MOUNTED_PARTITION # E.g. /dev/disk4s1</syntaxhighlight>
# The file path is an example. Use the absolute path to the downloaded image. '''Warning:''' You risk losing your own data if you specify a wrong device.
#:<syntaxhighlight lang="shell">sudo dd if=/Users/YOUR_USER_NAME/Downloads/disk.img of=/dev/YOUR_USB_DEVICE bs=1M</syntaxhighlight>

=== Linux / Ubuntu ===
# Open the Terminal and type
#:<syntaxhighlight lang="shell">blkid</syntaxhighlight>
# All available drives should be shown. Identify which device corresponds to your USB stick. You may need to unmount the USB drive:
#:<syntaxhighlight lang="shell">sudo diskutil unmount /dev/YOUR_USB_DEVICE_MOUNTED_PARTITION # E.g. /dev/sdb1</syntaxhighlight>
# Replace ''/dev/YOUR_USB_DEVICE'' with the device that corresponds to your USB stick. '''Warning:''' You risk losing your own data if you specify a wrong drive.
#:<syntaxhighlight lang="shell">sudo dd if=~/Downloads/disk.img of=/dev/YOUR_USB_DEVICE bs=1M</syntaxhighlight>

=== Windows ===
# Download and install [https://rufus.ie/en/ Rufus Portable]
# Start Rufus
# Select the USB stick under device and select the previously downloaded IC-OS disk image and press start
#: [[File:05.png|480px|screenshot]]
# You may see some warnings. Make sure you don't have any other USBs in your computer and chose OK
#: [[File:06.png|480px|screenshot]]
#: [[File:07.png|480px|screenshot]]
# The "Ready" bar will go from left to right as it completes.

== 4. Add configuration ==
=== Mac OS X ===
# Open Finder. You should now be able to see the CONFIG partition. If it's not visible, remove the USB and insert it again.
#: [[File:mac_01.png|580px|screenshot]]
# Double-click to open it in TextEdit.
# Insert your IPv6 prefix, subnet and gateway.
#:[[File:mac_02.png|580px|screenshot]]
#:* The IPv6 prefix should consist of four groups of hexadecimal digits, separated by colons (':'). Each group can contain up to four hex digits.
#:* For example, a valid prefix could look like this: <code>2a00:fb01:400:100</code>
#:* '''Important:'''
#:** The prefix should not have a trailing ':'
#:** IPv6 CIDR notation allows for a double colon ('::') to represent consecutive groups of zeroes in an address. However, the prefix configuration in this context does '''not''' support '::'. The '::' shorthand should '''not''' be used. Even if some groups are all zeros, they must be explicitly written out.
# Once done, don’t forget to save the changes. If you need help, please do not hesitate to post your issue in the [[Node Provider Matrix channel]].
#:[[File:mac_03.png|580px|screenshot]]
# Copy <code>node_operator_private_key.pem</code> from [[Node Provider Onboarding#5.%20Setup%20the%20Node%20Operator%20keys|Node Provider Onboarding step 5. Setup the Node Operator keys]] to the <code>CONFIG</code> partition. This file should have the name <code>node_operator_private_key.pem</code>, and sit next to <code>config.ini</code>, NOT inside the <code>ssh_authorized_keys</code> folder.

=== Windows ===
# Open the Disk Management utility with a right click on the Start menu
#: [[File:09-b.png|300px|screenshot]]#:
# Right click the CONFIG partition
# Select Change drive letter or paths...
#: [[File:10-b.png|780px|screenshot]]
# Select any letter from the drop-down list
#: [[File:11-b.png|480px|screenshot]]
# Click OK.
# You should now be able to see the CONFIG partition in your Windows Explorer. Select the config.ini configuration file
#: [[File:12-b.png|780px|screenshot]]
# Click on Edit to open it.
# Insert your IPv6 prefix, subnet and gateway.
#:[[File:13-b.png|580px|screenshot]]
#:* The IPv6 prefix should consist of four groups of hexadecimal digits, separated by colons (':'). Each group can contain up to four hex digits.
#:* For example, a valid prefix could look like this: <code>2a00:fb01:400:100</code>
#:*'''Important:'''
#:** The prefix should not have a trailing ':'
#:** IPv6 CIDR notation allows for a double colon ('::') to represent consecutive groups of zeroes in an address. However, the prefix configuration in this context does '''not''' support '::'. The '::' shorthand should '''not''' be used. Even if some groups are all zeros, they must be explicitly written out.
# Once done, don’t forget to save the changes. If you need help, please do not hesitate to post your issue in the [[Node Provider Matrix channel]].
#:[[File:14-b.png|580px|screenshot]]
# Copy <code>node_operator_private_key.pem</code> from [[Node Provider Onboarding#5. Setup the Node Operator keys|Node Provider Onboarding step 5. Setup the Node Operator keys]] to the <code>CONFIG</code> partition. This file should have the name <code>node_operator_private_key.pem</code>, and sit next to <code>config.ini</code>, NOT inside the <code>ssh_authorized_keys</code> folder.

=== Linux ===
# Open the File Manager. You should now be able to see the CONFIG partition. If it's not visible, remove the USB and insert it again.
#: [[File:linux_01.png|580px|screenshot]]
# Double-click to open it in KWrite.
# Insert your IPv6 prefix, subnet and gateway.
#:[[File:linux_02.png|580px|screenshot]]
#:* The IPv6 prefix should consist of four groups of hexadecimal digits, separated by colons (':'). Each group can contain up to four hex digits.
#:* For example, a valid prefix could look like this: <code>2a00:fb01:400:100</code>
#:*'''Important:'''
#:** The prefix should not have a trailing ':'
#:** IPv6 CIDR notation allows for a double colon ('::') to represent consecutive groups of zeroes in an address. However, the prefix configuration in this context does '''not''' support '::'. The '::' shorthand should '''not''' be used. Even if some groups are all zeros, they must be explicitly written out.
# Once done, don’t forget to save the changes. If you need help, please do not hesitate to post your issue in the [[Node Provider Matrix channel]].
#:[[File:linux_03.png|580px|screenshot]]
# Copy <code>node_operator_private_key.pem</code> from [[Node Provider Onboarding#5. Setup the Node Operator keys|Node Provider Onboarding step 5. Setup the Node Operator keys]] to the <code>CONFIG</code> partition. This file should have the name <code>node_operator_private_key.pem</code>, and sit next to <code>config.ini</code>, NOT inside the <code>ssh_authorized_keys</code> folder.

== 5. Connect Crash Cart ==
# In order to configure the UEFI and initiate the installation of the IC-OS, please connect a crash cart to the physical machine.
# Plug-in the VGA/Video, keyboard and IC-OS USB stick
#: [[File:08.png|580px|screenshot]]

== 6. UEFI Setup and Boot Menu ==
Use the related page below to set up the BIOS/UEFI according to your hardware vendor.

* [[IC OS Installation - UEFI Configuration - Dell]]
* [[IC OS Installation - UEFI Configuration - Supermicro]]
'''Important:''' Do NOT enable the RAID bios setting. Doing so will cause issues with the IC-OS installation.

Resume from this point when you are finished configuring the BIOS.

== 7. IC-OS Installation ==
# Please wait while the USB Installer is booting up. This process can take up to 3 minutes.
#: [[File:35-sm.png|580px|screenshot]]
# The IC-OS installation starts. Please keep an eye on the progress. This part can take up to 10 minutes. Please remember to check the [[Possible Node Onboarding Errors]] page if you encounter any errors.
#:[[File:36-sm.png|580px|screenshot]]
# If the installation finished successfully, it will initiate a reboot.
#:[[File:38-sm.png|580px|screenshot]] 
== 8. First Boot ==
Please remember to check the [[Possible Node Onboarding Errors]] page if you encounter any errors onboarding.
'''Do NOT re-try the onboarding after proceeding to this section, as this can cause duplication within the registry.'''
# Once you see this message, you may unplug the USB stick and VGA/Video. You can ignore the message to remove the HSM, as you did not use an HSM to onboard your nodes.
#:[[File:40-sm.png|580px|screenshot]]

Congratulations! Your machine successfully joined the Internet Computer! Again, once you see this message, '''do NOT re-try the onboarding after proceeding to this section, as this can cause duplication within the registry.''' The machine has joined the IC and the Node Provider will start receiving rewards!

== 9. Verify node onboarding ==

# Verify that your node was successfully onboarded by checking its status on the [https://dashboard.internetcomputer.org/ dashboard] is set to either “Awaiting Subnet” or “Active in Subnet”.
#* The dashboard can be searched by your Node Provider principal. There, you should see the Node ID of your node (Node ID outputted in step 8).
#* If the status of your node is not either “Awaiting Subnet” or “Active in Subnet”, or if it is not listed under your Node Provider principal, you should contact the [[Node Provider Matrix channel]] for assistance.
#*:[[File:Dashboard-node-verification.png|thumb|998x998px]]

Node Provider Networking Guide

2023-06-26T17:09:54Z

Gary.mcelroy: Clarified language

This guide is designed to provide an overview of the networking requirements and guide Node Providers through setting up their servers into a rack with functioning networking.

To follow this guide, you should be familiar with IP networking, network equipment and network cabling.

== Requirements ==
To join your servers to the Internet Computer (IC) you will need:

* 10G Network equipment
** [[wikipedia:Small_Form-factor_Pluggable|SFP+]] or [[wikipedia:10_Gigabit_Ethernet|Ethernet]]
** Switch(es)
** Cabling
** Quantity determined by number of nodes deployed
* Rackspace in a data center
* Internet connection
** Bandwidth
*** ~300Mbps per node
*** Ingress/egress ratio is currently 1:1. We expect egress (serving responses to client queries) to increase faster than ingress in the future.
*** This should guide how many servers to deploy and the appropriate ISP connection speed
*** E.g. a 1Gbps connection will support up to 3 IC nodes.
** One IPv6 /64 subnet - each node gets multiple IPv6 addresses
** One IPv4 address for every 4 nodes in a given data center per node provider (IPv4 addresses cannot be shared between node providers). See [[Node Provider Networking Guide#Appendix 1: Number of IPv4 Addresses Required|Appendix 1]] for table.
** '''All IP addresses are assigned statically''' and automatically by IC-OS
*** This is configured in the [[IC OS Installation Runbook#4.%20Add%20configuration|IC-OS Installation Runbook]]

== Network Cabling ==
When racking and stacking your servers, ensure the '''first two 10G network ports''' on each server are connected to the 10G switch.

[[File:Supermicro 1124US-TNRP 1U server rear photo diagram.png]]

For example, on a Supermicro 1U server, the bottom two ports are considered ports 1 and 2 and will be enumerated by Linux in this order. Connect the bottom two ports to the switch.

Servers from other vendors will differ! See the server documentation for guidance.

This is subject to change - the IC-OS network configuration logic is undergoing improvements to make it more flexible.

Connect the 10G switch to the ISP endpoint - this could be the Top Of Rack (TOR) switch or other box.

== BMC Setup Recommendations ==

=== What’s a BMC? ===
The [[wikipedia:Intelligent_Platform_Management_Interface#Baseboard_management_controller|Baseboard Management Controller (BMC)]] grants control of the underlying server hardware.

BMC’s have notoriously poor security. Vendors may name their implementation differently (Dell -> iDRAC, HPE -> iLO, etc.).

=== Recommendations ===

==== Change the password ====
BMC’s usually come with a common password. Log in via crash cart, KVM or the web interface and change it to something [https://krebsonsecurity.com/password-dos-and-donts/ strong].

==== No broad internet access ====
It is highly recommended: '''do not expose your BMC''' to the broad internet. This is a safety precaution against attackers.

Options:

* Don’t connect the BMC to the internet.
** Maintenance or node recovery will require physical access in this case.
** Any BMC activities occur via SSH on the host (unreliable on many mainboard vendors) or via crash cart.
* Connect the BMC to a separate dumb switch, and the dumb switch connects to a Rack Mounted Unit (RMU).
* Connect the BMC to a managed switch, and create a separate VLAN

This can get complicated. It’s outside the scope of this document to explain how to do this.

Resources:

* [https://security.stackexchange.com/questions/46351/best-practice-for-accessing-management-port-of-firewall StackExchange - Best practice for accessing management port of firewall]
* [https://www.supermicro.com/products/nfo/files/IPMI/Best_Practices_BMC_Security.pdf Supermicro Guidance]
* [https://www.unicomengineering.com/blog/ipmi-best-practices/ Unicom Guidance]

== What NOT to do ==

=== Don’t use external firewalls, packet filters, rate limiters ===
Don’t block or interfere with any traffic to the node machines. This can disrupt node machine functionality. Occasionally ports are opened for incoming (and outgoing) connections when new versions of node software are deployed.

==== What about network security? ====
IC-OS manages its own firewall(s) and rate limiters very strictly and is designed with security as a primary principle.

== How DFINITY manages its servers ==
See reference DFINITY [[Gen-2 Data Center runbook|data center runbook]].

== Final Checklist ==

* Did you deploy a 10G switch?
* Do the '''first and second 10G ports''' on each server plug into the 10G switch?
* Do you have '''one IPv6 /64 prefix''' allocated from your ISP?
* Do you have at least '''one IPv4 address for every four nodes''' allocated?
* Does each node have ~300Mbps bandwidth?
* Is your '''BMC inaccessible''' from the broad internet?

== References ==

* [[Gen-2 Network Requirements|Gen2 Network Requirements]] - more detailed, possibly out of date.

== Appendix 1: Number of IPv4 Addresses Required ==
{| class="wikitable"
|# Nodes
|# IPv4 Addresses
|-
|1
|1
|-
|2
|1
|-
|3
|1
|-
|4
|1
|-
|5
|2
|-
|6
|2
|-
|7
|2
|-
|8
|2
|-
|9
|3
|-
|10
|3
|-
|11
|3
|-
|12
|3
|-
|13
|4
|-
|14
|4
|-
|15
|4
|-
|16
|4
|-
|17
|5
|-
|18
|5
|-
|19
|5
|-
|20
|5
|-
|21
|6
|-
|22
|6
|-
|23
|6
|-
|24
|6
|-
|25
|7
|-
|26
|7
|-
|27
|7
|-
|28
|7
|}

Node Provider Onboarding

2023-06-15T13:24:46Z

Gary.mcelroy: /* V. Setup the Node Operator keys */ Add upgrade command

Learn how to participate in the Internet Computer network as a Node Provider and to receive rewards for supporting the network.

== Requirements ==

* [https://wiki.internetcomputer.org/wiki/Node_provider_hardware Node Hardware]
* Rack space with a 10Gb connectivity, RJ45 terminated on the nodes
* [[Node Provider Network Setup Guide#The Bare Minimum Network Requirements|Public IP addresses]]:
** One /64 IPv6 range
** One IPv4 address for every 4 node machines
* [https://www.ledger.com/ Hardware wallet]
* [https://shop.nitrokey.com/shop/product/nkhs2-nitrokey-hsm-2-7/ NitroKey HSM] (Optional)
* 11 ICP (10 of which are to be staked for the NNS proposal deposit)
* Basic understanding of [[Neurons 101|neurons]], [https://internetcomputer.org/docs/current/tokenomics/nns/nns-staking-voting-rewards staking], and [[Governance of the Internet Computer|governance]] proposals. For instance, understanding what it means to stake a neuron for 8 years.
* The technical knowledge to understand some minor steps that are not explicitly mentioned in these instructions. For instance, when to insert an HSM.

'''Note:''' Please allocate at least 0.5 day for going through the first part, i.e., the registration of a new NP. It may even take a couple of days, depending on how quickly the community votes for the proposals. There is a also fair amount of complexity and the technical knowledge that needs to be absorbed in order to complete the steps. But this only needs to be done once. 
The next step, going to the DC and bringing up and onboarding the machines, is much quicker. Estimate to spend 10-15 minutes per machine. This time should go down to ~5 minutes as you gain experience. Also, multiple machines can be brought up in parallel.

== I. Install the required tools ==
===''' A. Install ic-admin '''===

<code>ic-admin</code> is the tool used to create and submit NNS proposals.

==== MacOS ====
# Retrieve the file <syntaxhighlight lang="shell">
curl "https://download.dfinity.systems/ic/7445081734e6d896d090295967d50710975c4f25/openssl-static-binaries/x86_64-darwin/ic-admin.gz" -o - | gunzip > ./ic-admin
chmod +x ./ic-admin
</syntaxhighlight>
# Verify the binary <syntaxhighlight lang="shell">
diff <(shasum -a 256 ./ic-admin | cut -d' ' -f1) <(echo 3f75026d2f28f171068e332a42c82a2795c93fbf5ab351baef30b30eb901cdba) && echo "ic-admin checksum matches" || echo "***ERROR***: ic-admin checksum does not match"
</syntaxhighlight>

==== Linux ====
NOTE: The instructions below have been tested with the Ubuntu 20.04 release.
# Retrieve the file <syntaxhighlight lang="shell">
curl "https://download.dfinity.systems/ic/7445081734e6d896d090295967d50710975c4f25/openssl-static-binaries/x86_64-linux/ic-admin.gz" -o - | gunzip > ./ic-admin
chmod +x ./ic-admin
</syntaxhighlight>
# Verify the binary <syntaxhighlight lang="shell">
diff <(shasum -a 256 ./ic-admin | cut -d' ' -f1) <(echo e29bb9cc462e800b8b960ad49c412e5f5fdbb5ae2ae9fde0c13058422ba32802) && echo "ic-admin checksum matches" || echo "***ERROR***: ic-admin checksum does not match"
</syntaxhighlight>

===''' B. Install dfx '''===

# <code>dfx</code> is used to generate neuron hotkeys, among other things <syntaxhighlight lang="shell">
$ sh -ci "$(curl -fsSL https://smartcontracts.org/install.sh)"

</syntaxhighlight>
# Verify that dfx is up to date. <syntaxhighlight lang="shell">
$ export PATH=$HOME/bin:$PATH
$ dfx upgrade
$ dfx --version
</syntaxhighlight>
# Create an identity for the Node Provider '''Hotkey''' <syntaxhighlight lang="shell">
$ dfx identity new node-provider-hotkey
Creating identity: "node-provider-hotkey".
Created identity: "node-provider-hotkey".
$ dfx --identity node-provider-hotkey identity get-principal
wuyst-x5tpn-g5wri-mp3ps-vjtba-de3xs-w5xgb-crvek-tucbe-o5rqi-mae
</syntaxhighlight>
# '''Note:''' The node provider hotkey is NOT the node provider principal. This is the hotkey that is used for the NNS proposal submissions only.

== II. Create and Manage Neuron via NNS Frontend Dapp and Internet Identity ==

# Setup your hardware wallet: https://medium.com/dfinity/integrating-ledger-nano-with-the-nns-front-end-dapp-user-manual-9c5600925e16
# Send at least 11 ICPs to the hardware wallet address.
# Navigate to Neurons tab and create a Neuron by staking at least 10 ICP from your hardware wallet. Staking more ICP works as well, but 10 is the minimum.
# IMPORTANT! Confirm the transaction on your hardware wallet.
#: [[File:-docs-stake_neuron_1.png|1024px|stake neuron]]
#:
# After the Neuron has been created successfully, confirm to add NNS Dapp as hotkey in the dialogue and on your hardware wallet, and close the dialog after the action completes.
#: [[File:-docs-stake_neuron_2.png|1024px|neuron id]]
# Set the dissolve delay to at least 6 months, and confirm the choice in the dialogue and on your hardware wallet. After the action completes, you can close the "Follow Neurons".
#:
#: [[File:dissolve_delay.png|480px|neuron id]]
# You will now see a Neuron listed with its ID. Copy the Neuron ID, since you will need it in the next steps to place the necessary proposals.
#: [[File:Neuron id.png|1024px]]

== III. Add hotkeys ==

# Select the Neuron you just created to open Neuron management view and press “Add hotkey” button.
#: [[File:Hotkey 1.png|800px]]
# A dialog will pop up where you can enter the principal you generated in step 2 (output from command <code>dfx --identity node-provider-hotkey identity get-principal</code>). This will allow you to submit NNS proposals using <code>ic-admin</code> and will not be used for anything else. 
#:Press the confirm button and confirm the transactions on your hardware wallet. 
#: [[File:Hotkey 2.png|800px]]
#:
# Get the Ledger Hardware Wallet Principal Id: Navigate back to ICP page and select your Ledger hardware wallet account. You will need to use this Ledger Hardware Wallet principal as the Node Provider principal in order to get the rewards directly into the secure hardware wallet.
[[File:Node provider principal 1.png|1024px]]
[[File:Node provider principal 2.png|800px]]
# Copy and save this Node Provider principal by clicking on the copy icon after the principal id. You'll need it in the next steps.

<syntaxhighlight lang="shell">
NODE_PROVIDER_PRINCIPAL=fharn-5vyi2-4xb4a-64yyi-3jpmj-pga23-mxy25-d5uim-fqcro-eoefh-tae # Ledger Hardware Wallet principal, from the NNS FrontEnd dapp https://nns.ic0.app/
</syntaxhighlight>

== IV. Configure your HSM ==

=== Onboarding without a NitroKey HSM ===
If you will not be using a NitroKey HSM, continue to the next step.

=== Using a NitroKey HSM ===
It's first necessary to install the necessary tools.

=== MacOS ===
# Download this OpenSC binary: https://github.com/OpenSC/OpenSC/releases/download/0.22.0/OpenSC-0.22.0.dmg
# Double click the DMG image that you downloaded and then double click the OpenSC PKG file.
# If your system doesn't allow the installation software from an unidentified developer please follow these steps or contact your system administrator:
#* Choose the Apple menu > System Preferences > click Security and Privacy.
#* Click the lock Icon to unlock it, then enter an administrator name and password.
#* Ensure that you're on the tab named “General”.
#* You should see the OpenSC app and you should be able to enable its installation by choosing “Open anyway”.
# Click continue and install until the installation is complete.

=== Linux ===

NOTE: The instructions below have been tested with the Ubuntu 20.04 release.

# Install pcscd and opensc
#: <syntaxhighlight lang="shell">
sudo add-apt-repository universe
sudo apt update
sudo apt install pcscd opensc
</syntaxhighlight>

== V. Setup the Node Operator keys ==

=== Onboarding without a NitroKey HSM ===

# '''''Ensure dfx is at least version 0.14.''''' Node Operator keys created with older versions of dfx '''will fail to join the IC'''. Run:<syntaxhighlight lang="bash">
dfx upgrade
</syntaxhighlight>
# Create a new principal with dfx:<syntaxhighlight lang="shell">
dfx identity new --storage-mode=plaintext node_operator
</syntaxhighlight>
# Confirm <code>node_operator</code> identity was created successfully:<syntaxhighlight lang="shell">
dfx identity list
</syntaxhighlight>This list ''should'' contain <code>node_operator</code>.
# Copy new key to a known location:<syntaxhighlight lang="shell">
cp ~/.config/dfx/identity/node_operator/identity.pem ./node_operator_private_key.pem
</syntaxhighlight>

=== Using a NitroKey HSM ===
# Initialize the HSM. <syntaxhighlight lang="shell">
sc-hsm-tool --initialize --so-pin 3537363231383830 --pin 358138
</syntaxhighlight>
# Change the HSM so-pin.
#* '''WARNING:''' The new HSM so pin must have 16 hexadecimal digits. This is not very well known, and some HSM users have lost access to a Nitrokey HSM because they tried using regular characters and the command below accepted it.
#* '''Do NOT change the user pin. It must remain as the default for the onboarding scripts to work'''<syntaxhighlight lang="shell">
pkcs11-tool --login --login-type so --so-pin 3537363231383830 --change-pin
</syntaxhighlight>
# Create a keypair on the HSM. Enter the default pin 358138 when prompted.<syntaxhighlight lang="shell">
pkcs11-tool -k --key-type EC:prime256v1 --login -d 01
</syntaxhighlight>
#*'''Note:''' Key backup may be possible with [https://github.com/OpenSC/OpenSC/wiki/SmartCardHSM#using-key-backup-and-restore these instructions].

== VI. Get the node operator principal ==

=== Onboarding without a NitroKey HSM ===

# Confirm <code>node_operator</code> identity was created successfully in step V.:<syntaxhighlight lang="shell">
dfx identity list
</syntaxhighlight>This list ''should'' contain <code>node_operator</code>.
# Get the principal:<syntaxhighlight lang="shell">
NODE_OPERATOR_PRINCIPAL=$(dfx --identity node_operator identity get-principal)
echo $NODE_OPERATOR_PRINCIPAL

uqquy-76uhn-2mys5-xxxxx-xxxxx-xxxxx-xxxxx-xxxxx-xxxxx-xxxxx-xxx
</syntaxhighlight>

=== Using a NitroKey HSM ===
# Configure dfx identity (skip this step if you already configured it for an other HSM).
#*'''Note:''' Depending on your installation, the path to the <code>--hsm-pkcs11-lib-path</code> might be different on your platform. You can locate the correct path with the following command: <syntaxhighlight lang="shell">
find / -name opensc-pkcs11.so 2> /dev/null
</syntaxhighlight>
#* MacOS <syntaxhighlight lang="shell">
dfx identity new node-operator-hsm --hsm-key-id 01 --hsm-pkcs11-lib-path /Library/OpenSC/lib/opensc-pkcs11.so
</syntaxhighlight>
#* Linux <syntaxhighlight lang="shell">
dfx identity new node-operator-hsm --hsm-key-id 01 --hsm-pkcs11-lib-path /usr/lib/x86_64-linux-gnu/opensc-pkcs11.so
</syntaxhighlight>
# Get the principal. <syntaxhighlight lang="shell">
$ NODE_OPERATOR_PRINCIPAL=$(DFX_HSM_PIN=358138 dfx --identity node-operator-hsm identity get-principal)
$ echo $NODE_OPERATOR_PRINCIPAL

uqquy-76uhn-2mys5-xxxxx-xxxxx-xxxxx-xxxxx-xxxxx-xxxxx-xxxxx-xxx
</syntaxhighlight>

== VII. Register your NP principal to the network ==

In the next codeblock:
* Replace the <code>NEURON_ID</code> value with your neuron ID from the NNS Frontend Dapp
* Replace the <code>NODE_PROVIDER_PRINCIPAL</code> value with the Ledger Hardware Wallet principal that you got from the NNS Frontend DAPP.
* Replace the <code>NODE_PROVIDER_NAME</code> value with the name of the entity that will provide the nodes. 
* '''''IMPORTANT:''''' Please make sure that you also update the <code>--summary</code> and include a link to the forum discussion, your company's web page, and/or to another place that can convince the voting community that you are making a legitimate request. This way you will avoid the community voting NO to your proposal and you losing the staked ICPs.

# Create the Proposal <syntaxhighlight lang="shell">
NODE_PROVIDER_NAME="My Company"
NODE_PROVIDER_PRINCIPAL=fharn-5vyi2-4xb4a-64yyi-3jpmj-pga23-mxy25-d5uim-fqcro-eoefh-tae # Ledger Hardware Wallet principal, from the NNS FE dapp https://ic0.app/
NEURON_ID=13419667327548602649 # Coming from the NNS FrontEnd dapp https://nns.ic0.app/
./ic-admin \
--nns-url https://ic0.app \
-s ~/.config/dfx/identity/node-provider-hotkey/identity.pem \
propose-to-add-or-remove-node-provider add \
--proposer $NEURON_ID \
--proposal-title "Register a node provider '${NODE_PROVIDER_NAME}'" \
--summary "Register a node provider '${NODE_PROVIDER_NAME}', in line with the announcement and discussion at https://forum.dfinity.org/t/..." \
--node-provider-pid "$NODE_PROVIDER_PRINCIPAL"
</syntaxhighlight>
# Find the proposal on https://dashboard.internetcomputer.org/governance and '''wait until it is executed before proceeding to next step.'''

== VIII. Ensure that your datacenter is registered in the network ==
# Search for your data center on https://dashboard.internetcomputer.org/centers.
#* If you found the datacenter that is hosting your nodes, remember its ID, and skip the following section. Otherwise, proceed with the registration of a new DC. [[File:dc_id.png|1024px]]

=== Create a data center record for a new DC ===
In the next block of code:
* Replace the <code>--proposer</code> argument value with your Neuron ID from the NNS Frontend Dapp.
* Replace the JSON fields from <code>–data-centers-to-add</code> argument and their corresponding values in <code>--summary</code> with: <code>"id"</code>
* The ID should be combination of two letters representing a city that your datacenter is in, and an incrementing number. Search data center IDs on https://dashboard.internetcomputer.org, and find a combination of two letters and a number that’s not yet registered. Examples:
** dl1 (Dallas, no IDs with “dl” prefix)
** zh10 (Zurich, numbers 0-9 are already registered)
[[File:dc_id.png|1024px]]

* <code>"region"</code> represents the local region of a datacenter and is formulated as a three-part string divided by commas. The three parts making the string are continent, country code, and region, in the given order. Examples:
** North America,US,Florida
** Europe,DE,Bavaria
** Asia,SG,Singapore
[[File:datacenter_region.png|1024px]]

* <code>"owner"</code> The entity that provides your datacenter facilities.
** Search https://dashboard.internetcomputer.org for existing data center providers.
** If there’s match, make sure you use the same exact some name for your datacenter.
** Otherwise, name the data center owner to your best knowledge. [[File:datacenter_owner.png|1024px]]

* <code>"gps"</code> GPS coordinates.
** Find your datacenter on https://www.google.com/maps/.
** Right click on location, and select the GPS coordinates (first item in the menu) in order to copy them.
[[File:maps.png|480px|alt=Getting GPS coordinates|Getting GPS coordinates]]

# Create the proposal: <syntaxhighlight lang="shell">
NEURON_ID=13419667327548602649 # Coming from the NNS FE dapp https://nns.ic0.app/
$ ./ic-admin \
--nns-url https://ic0.app \
-s ~/.config/dfx/identity/node-provider-hotkey/identity.pem \
propose-to-add-or-remove-data-centers \
--summary "Register a Flexential datacenter as dl1 in North America,US,Texas" \
--skip-confirmation \
--proposer $NEURON_ID \
--data-centers-to-add '{
"id": "dl1",
"region": "North America,US,Texas",
"owner": "Flexential",
"gps": [
33.00803, -96.66614
]
}'
</syntaxhighlight>
# Find the proposal on https://dashboard.internetcomputer.org/governance and wait until it's executed before proceeding to next step.

== IX. Create a node operator record ==
In the next codeblock:
* Replace the <code>NEURON_ID</code> variable value with your neuron ID obtained from the NNS frontend dapp.
* Replace the <code>NODE_PROVIDER_PRINCIPAL</code> variable value with the principal obtained in step VI.
* Replace the <code>DC_ID</code> variable value with id of your datacenter.
* Replace the <code>NODE_ALLOWANCE</code> variable value with number of nodes you are providing.

# Create the proposal: <syntaxhighlight lang="shell">
NEURON_ID=13419667327548602649 # Coming from the NNS FE dapp https://nns.ic0.app/
NODE_PROVIDER_PRINCIPAL=fharn-5vyi2-4xb4a-64yyi-3jpmj-pga23-mxy25-d5uim-fqcro-eoefh-tae # Ledger Hardware Wallet principal, from the NNS FE dapp https://nns.ic0.app/
NODE_OPERATOR_PRINCIPAL=uqquy-76uhn-2mys5-xxxxx-xxxxx-xxxxx-xxxxx-xxxxx-xxxxx-xxxxx-xxx # Node Operator principal from step VI
NODE_PROVIDER_NAME="My Company"
NODE_ALLOWANCE=8
DC_ID=dl1

./ic-admin \
--nns-url https://ic0.app \
-s ~/.config/dfx/identity/node-provider-hotkey/identity.pem \
propose-to-add-node-operator \
$NODE_PROVIDER_PRINCIPAL \
--summary "Node provider '$NODE_PROVIDER_NAME' is adding $NODE_ALLOWANCE nodes in the $DC_ID data center" \
--proposer $NEURON_ID \
--node-operator-principal-id $NODE_OPERATOR_PRINCIPAL \
--node-allowance $NODE_ALLOWANCE \
--dc-id $DC_ID
</syntaxhighlight>

# Find the proposal on https://dashboard.internetcomputer.org/governance and wait until it's executed before proceeding to next step.

== X. Onboard nodes ==

# Follow the instructions to onboard new nodes:
#* Gen2 - For NP's onboarding in 2023 and later
#** [[IC OS Installation Runbook]]
#* Gen1 - For NP's participating in the IC before 2023
#**[[IC OS Installation Runbook - PowerEdge R6525]]
#**[[IC OS Installation Runbook - Supermicro]]
# Verify that all the nodes were successfully onboarded by checking their status on the dashboard is set to either “Up” or “Unassigned”, or by checking the output from <code>ic-admin get-topology</code> command.
#* The internal dashboard can be searched by your provider principal.
[[File:onboarded_nodes.png|1024px|onboarded nodes]]

== See Also ==
* '''The Internet Computer project website (hosted on the IC): [https://internetcomputer.org/ internetcomputer.org]'''
* '''IC Node Provider Matrix/Element channel:''' https://app.element.io/#/room/#ic-node-providers:matrix.org

Node Provider Onboarding

2023-06-15T13:23:03Z

Gary.mcelroy: /* B. Install dfx */ Added note about dfx version

Learn how to participate in the Internet Computer network as a Node Provider and to receive rewards for supporting the network.

== Requirements ==

* [https://wiki.internetcomputer.org/wiki/Node_provider_hardware Node Hardware]
* Rack space with a 10Gb connectivity, RJ45 terminated on the nodes
* [[Node Provider Network Setup Guide#The Bare Minimum Network Requirements|Public IP addresses]]:
** One /64 IPv6 range
** One IPv4 address for every 4 node machines
* [https://www.ledger.com/ Hardware wallet]
* [https://shop.nitrokey.com/shop/product/nkhs2-nitrokey-hsm-2-7/ NitroKey HSM] (Optional)
* 11 ICP (10 of which are to be staked for the NNS proposal deposit)
* Basic understanding of [[Neurons 101|neurons]], [https://internetcomputer.org/docs/current/tokenomics/nns/nns-staking-voting-rewards staking], and [[Governance of the Internet Computer|governance]] proposals. For instance, understanding what it means to stake a neuron for 8 years.
* The technical knowledge to understand some minor steps that are not explicitly mentioned in these instructions. For instance, when to insert an HSM.

'''Note:''' Please allocate at least 0.5 day for going through the first part, i.e., the registration of a new NP. It may even take a couple of days, depending on how quickly the community votes for the proposals. There is a also fair amount of complexity and the technical knowledge that needs to be absorbed in order to complete the steps. But this only needs to be done once. 
The next step, going to the DC and bringing up and onboarding the machines, is much quicker. Estimate to spend 10-15 minutes per machine. This time should go down to ~5 minutes as you gain experience. Also, multiple machines can be brought up in parallel.

== I. Install the required tools ==
===''' A. Install ic-admin '''===

<code>ic-admin</code> is the tool used to create and submit NNS proposals.

==== MacOS ====
# Retrieve the file <syntaxhighlight lang="shell">
curl "https://download.dfinity.systems/ic/7445081734e6d896d090295967d50710975c4f25/openssl-static-binaries/x86_64-darwin/ic-admin.gz" -o - | gunzip > ./ic-admin
chmod +x ./ic-admin
</syntaxhighlight>
# Verify the binary <syntaxhighlight lang="shell">
diff <(shasum -a 256 ./ic-admin | cut -d' ' -f1) <(echo 3f75026d2f28f171068e332a42c82a2795c93fbf5ab351baef30b30eb901cdba) && echo "ic-admin checksum matches" || echo "***ERROR***: ic-admin checksum does not match"
</syntaxhighlight>

==== Linux ====
NOTE: The instructions below have been tested with the Ubuntu 20.04 release.
# Retrieve the file <syntaxhighlight lang="shell">
curl "https://download.dfinity.systems/ic/7445081734e6d896d090295967d50710975c4f25/openssl-static-binaries/x86_64-linux/ic-admin.gz" -o - | gunzip > ./ic-admin
chmod +x ./ic-admin
</syntaxhighlight>
# Verify the binary <syntaxhighlight lang="shell">
diff <(shasum -a 256 ./ic-admin | cut -d' ' -f1) <(echo e29bb9cc462e800b8b960ad49c412e5f5fdbb5ae2ae9fde0c13058422ba32802) && echo "ic-admin checksum matches" || echo "***ERROR***: ic-admin checksum does not match"
</syntaxhighlight>

===''' B. Install dfx '''===

# <code>dfx</code> is used to generate neuron hotkeys, among other things <syntaxhighlight lang="shell">
$ sh -ci "$(curl -fsSL https://smartcontracts.org/install.sh)"

</syntaxhighlight>
# Verify that dfx is up to date. <syntaxhighlight lang="shell">
$ export PATH=$HOME/bin:$PATH
$ dfx upgrade
$ dfx --version
</syntaxhighlight>
# Create an identity for the Node Provider '''Hotkey''' <syntaxhighlight lang="shell">
$ dfx identity new node-provider-hotkey
Creating identity: "node-provider-hotkey".
Created identity: "node-provider-hotkey".
$ dfx --identity node-provider-hotkey identity get-principal
wuyst-x5tpn-g5wri-mp3ps-vjtba-de3xs-w5xgb-crvek-tucbe-o5rqi-mae
</syntaxhighlight>
# '''Note:''' The node provider hotkey is NOT the node provider principal. This is the hotkey that is used for the NNS proposal submissions only.

== II. Create and Manage Neuron via NNS Frontend Dapp and Internet Identity ==

# Setup your hardware wallet: https://medium.com/dfinity/integrating-ledger-nano-with-the-nns-front-end-dapp-user-manual-9c5600925e16
# Send at least 11 ICPs to the hardware wallet address.
# Navigate to Neurons tab and create a Neuron by staking at least 10 ICP from your hardware wallet. Staking more ICP works as well, but 10 is the minimum.
# IMPORTANT! Confirm the transaction on your hardware wallet.
#: [[File:-docs-stake_neuron_1.png|1024px|stake neuron]]
#:
# After the Neuron has been created successfully, confirm to add NNS Dapp as hotkey in the dialogue and on your hardware wallet, and close the dialog after the action completes.
#: [[File:-docs-stake_neuron_2.png|1024px|neuron id]]
# Set the dissolve delay to at least 6 months, and confirm the choice in the dialogue and on your hardware wallet. After the action completes, you can close the "Follow Neurons".
#:
#: [[File:dissolve_delay.png|480px|neuron id]]
# You will now see a Neuron listed with its ID. Copy the Neuron ID, since you will need it in the next steps to place the necessary proposals.
#: [[File:Neuron id.png|1024px]]

== III. Add hotkeys ==

# Select the Neuron you just created to open Neuron management view and press “Add hotkey” button.
#: [[File:Hotkey 1.png|800px]]
# A dialog will pop up where you can enter the principal you generated in step 2 (output from command <code>dfx --identity node-provider-hotkey identity get-principal</code>). This will allow you to submit NNS proposals using <code>ic-admin</code> and will not be used for anything else. 
#:Press the confirm button and confirm the transactions on your hardware wallet. 
#: [[File:Hotkey 2.png|800px]]
#:
# Get the Ledger Hardware Wallet Principal Id: Navigate back to ICP page and select your Ledger hardware wallet account. You will need to use this Ledger Hardware Wallet principal as the Node Provider principal in order to get the rewards directly into the secure hardware wallet.
[[File:Node provider principal 1.png|1024px]]
[[File:Node provider principal 2.png|800px]]
# Copy and save this Node Provider principal by clicking on the copy icon after the principal id. You'll need it in the next steps.

<syntaxhighlight lang="shell">
NODE_PROVIDER_PRINCIPAL=fharn-5vyi2-4xb4a-64yyi-3jpmj-pga23-mxy25-d5uim-fqcro-eoefh-tae # Ledger Hardware Wallet principal, from the NNS FrontEnd dapp https://nns.ic0.app/
</syntaxhighlight>

== IV. Configure your HSM ==

=== Onboarding without a NitroKey HSM ===
If you will not be using a NitroKey HSM, continue to the next step.

=== Using a NitroKey HSM ===
It's first necessary to install the necessary tools.

=== MacOS ===
# Download this OpenSC binary: https://github.com/OpenSC/OpenSC/releases/download/0.22.0/OpenSC-0.22.0.dmg
# Double click the DMG image that you downloaded and then double click the OpenSC PKG file.
# If your system doesn't allow the installation software from an unidentified developer please follow these steps or contact your system administrator:
#* Choose the Apple menu > System Preferences > click Security and Privacy.
#* Click the lock Icon to unlock it, then enter an administrator name and password.
#* Ensure that you're on the tab named “General”.
#* You should see the OpenSC app and you should be able to enable its installation by choosing “Open anyway”.
# Click continue and install until the installation is complete.

=== Linux ===

NOTE: The instructions below have been tested with the Ubuntu 20.04 release.

# Install pcscd and opensc
#: <syntaxhighlight lang="shell">
sudo add-apt-repository universe
sudo apt update
sudo apt install pcscd opensc
</syntaxhighlight>

== V. Setup the Node Operator keys ==

=== Onboarding without a NitroKey HSM ===

# '''''Ensure dfx is at least version 0.14.''''' Node Operator keys created with older versions of dfx '''will fail to join the IC'''.
# Create a new principal with dfx:<syntaxhighlight lang="shell">
dfx identity new --storage-mode=plaintext node_operator
</syntaxhighlight>
# Confirm <code>node_operator</code> identity was created successfully:<syntaxhighlight lang="shell">
dfx identity list
</syntaxhighlight>This list ''should'' contain <code>node_operator</code>.
# Copy new key to a known location:<syntaxhighlight lang="shell">
cp ~/.config/dfx/identity/node_operator/identity.pem ./node_operator_private_key.pem
</syntaxhighlight>

=== Using a NitroKey HSM ===
# Initialize the HSM. <syntaxhighlight lang="shell">
sc-hsm-tool --initialize --so-pin 3537363231383830 --pin 358138
</syntaxhighlight>
# Change the HSM so-pin.
#* '''WARNING:''' The new HSM so pin must have 16 hexadecimal digits. This is not very well known, and some HSM users have lost access to a Nitrokey HSM because they tried using regular characters and the command below accepted it.
#* '''Do NOT change the user pin. It must remain as the default for the onboarding scripts to work'''<syntaxhighlight lang="shell">
pkcs11-tool --login --login-type so --so-pin 3537363231383830 --change-pin
</syntaxhighlight>
# Create a keypair on the HSM. Enter the default pin 358138 when prompted.<syntaxhighlight lang="shell">
pkcs11-tool -k --key-type EC:prime256v1 --login -d 01
</syntaxhighlight>
#*'''Note:''' Key backup may be possible with [https://github.com/OpenSC/OpenSC/wiki/SmartCardHSM#using-key-backup-and-restore these instructions].

== VI. Get the node operator principal ==

=== Onboarding without a NitroKey HSM ===

# Confirm <code>node_operator</code> identity was created successfully in step V.:<syntaxhighlight lang="shell">
dfx identity list
</syntaxhighlight>This list ''should'' contain <code>node_operator</code>.
# Get the principal:<syntaxhighlight lang="shell">
NODE_OPERATOR_PRINCIPAL=$(dfx --identity node_operator identity get-principal)
echo $NODE_OPERATOR_PRINCIPAL

uqquy-76uhn-2mys5-xxxxx-xxxxx-xxxxx-xxxxx-xxxxx-xxxxx-xxxxx-xxx
</syntaxhighlight>

=== Using a NitroKey HSM ===
# Configure dfx identity (skip this step if you already configured it for an other HSM).
#*'''Note:''' Depending on your installation, the path to the <code>--hsm-pkcs11-lib-path</code> might be different on your platform. You can locate the correct path with the following command: <syntaxhighlight lang="shell">
find / -name opensc-pkcs11.so 2> /dev/null
</syntaxhighlight>
#* MacOS <syntaxhighlight lang="shell">
dfx identity new node-operator-hsm --hsm-key-id 01 --hsm-pkcs11-lib-path /Library/OpenSC/lib/opensc-pkcs11.so
</syntaxhighlight>
#* Linux <syntaxhighlight lang="shell">
dfx identity new node-operator-hsm --hsm-key-id 01 --hsm-pkcs11-lib-path /usr/lib/x86_64-linux-gnu/opensc-pkcs11.so
</syntaxhighlight>
# Get the principal. <syntaxhighlight lang="shell">
$ NODE_OPERATOR_PRINCIPAL=$(DFX_HSM_PIN=358138 dfx --identity node-operator-hsm identity get-principal)
$ echo $NODE_OPERATOR_PRINCIPAL

uqquy-76uhn-2mys5-xxxxx-xxxxx-xxxxx-xxxxx-xxxxx-xxxxx-xxxxx-xxx
</syntaxhighlight>

== VII. Register your NP principal to the network ==

In the next codeblock:
* Replace the <code>NEURON_ID</code> value with your neuron ID from the NNS Frontend Dapp
* Replace the <code>NODE_PROVIDER_PRINCIPAL</code> value with the Ledger Hardware Wallet principal that you got from the NNS Frontend DAPP.
* Replace the <code>NODE_PROVIDER_NAME</code> value with the name of the entity that will provide the nodes. 
* '''''IMPORTANT:''''' Please make sure that you also update the <code>--summary</code> and include a link to the forum discussion, your company's web page, and/or to another place that can convince the voting community that you are making a legitimate request. This way you will avoid the community voting NO to your proposal and you losing the staked ICPs.

# Create the Proposal <syntaxhighlight lang="shell">
NODE_PROVIDER_NAME="My Company"
NODE_PROVIDER_PRINCIPAL=fharn-5vyi2-4xb4a-64yyi-3jpmj-pga23-mxy25-d5uim-fqcro-eoefh-tae # Ledger Hardware Wallet principal, from the NNS FE dapp https://ic0.app/
NEURON_ID=13419667327548602649 # Coming from the NNS FrontEnd dapp https://nns.ic0.app/
./ic-admin \
--nns-url https://ic0.app \
-s ~/.config/dfx/identity/node-provider-hotkey/identity.pem \
propose-to-add-or-remove-node-provider add \
--proposer $NEURON_ID \
--proposal-title "Register a node provider '${NODE_PROVIDER_NAME}'" \
--summary "Register a node provider '${NODE_PROVIDER_NAME}', in line with the announcement and discussion at https://forum.dfinity.org/t/..." \
--node-provider-pid "$NODE_PROVIDER_PRINCIPAL"
</syntaxhighlight>
# Find the proposal on https://dashboard.internetcomputer.org/governance and '''wait until it is executed before proceeding to next step.'''

== VIII. Ensure that your datacenter is registered in the network ==
# Search for your data center on https://dashboard.internetcomputer.org/centers.
#* If you found the datacenter that is hosting your nodes, remember its ID, and skip the following section. Otherwise, proceed with the registration of a new DC. [[File:dc_id.png|1024px]]

=== Create a data center record for a new DC ===
In the next block of code:
* Replace the <code>--proposer</code> argument value with your Neuron ID from the NNS Frontend Dapp.
* Replace the JSON fields from <code>–data-centers-to-add</code> argument and their corresponding values in <code>--summary</code> with: <code>"id"</code>
* The ID should be combination of two letters representing a city that your datacenter is in, and an incrementing number. Search data center IDs on https://dashboard.internetcomputer.org, and find a combination of two letters and a number that’s not yet registered. Examples:
** dl1 (Dallas, no IDs with “dl” prefix)
** zh10 (Zurich, numbers 0-9 are already registered)
[[File:dc_id.png|1024px]]

* <code>"region"</code> represents the local region of a datacenter and is formulated as a three-part string divided by commas. The three parts making the string are continent, country code, and region, in the given order. Examples:
** North America,US,Florida
** Europe,DE,Bavaria
** Asia,SG,Singapore
[[File:datacenter_region.png|1024px]]

* <code>"owner"</code> The entity that provides your datacenter facilities.
** Search https://dashboard.internetcomputer.org for existing data center providers.
** If there’s match, make sure you use the same exact some name for your datacenter.
** Otherwise, name the data center owner to your best knowledge. [[File:datacenter_owner.png|1024px]]

* <code>"gps"</code> GPS coordinates.
** Find your datacenter on https://www.google.com/maps/.
** Right click on location, and select the GPS coordinates (first item in the menu) in order to copy them.
[[File:maps.png|480px|alt=Getting GPS coordinates|Getting GPS coordinates]]

# Create the proposal: <syntaxhighlight lang="shell">
NEURON_ID=13419667327548602649 # Coming from the NNS FE dapp https://nns.ic0.app/
$ ./ic-admin \
--nns-url https://ic0.app \
-s ~/.config/dfx/identity/node-provider-hotkey/identity.pem \
propose-to-add-or-remove-data-centers \
--summary "Register a Flexential datacenter as dl1 in North America,US,Texas" \
--skip-confirmation \
--proposer $NEURON_ID \
--data-centers-to-add '{
"id": "dl1",
"region": "North America,US,Texas",
"owner": "Flexential",
"gps": [
33.00803, -96.66614
]
}'
</syntaxhighlight>
# Find the proposal on https://dashboard.internetcomputer.org/governance and wait until it's executed before proceeding to next step.

== IX. Create a node operator record ==
In the next codeblock:
* Replace the <code>NEURON_ID</code> variable value with your neuron ID obtained from the NNS frontend dapp.
* Replace the <code>NODE_PROVIDER_PRINCIPAL</code> variable value with the principal obtained in step VI.
* Replace the <code>DC_ID</code> variable value with id of your datacenter.
* Replace the <code>NODE_ALLOWANCE</code> variable value with number of nodes you are providing.

# Create the proposal: <syntaxhighlight lang="shell">
NEURON_ID=13419667327548602649 # Coming from the NNS FE dapp https://nns.ic0.app/
NODE_PROVIDER_PRINCIPAL=fharn-5vyi2-4xb4a-64yyi-3jpmj-pga23-mxy25-d5uim-fqcro-eoefh-tae # Ledger Hardware Wallet principal, from the NNS FE dapp https://nns.ic0.app/
NODE_OPERATOR_PRINCIPAL=uqquy-76uhn-2mys5-xxxxx-xxxxx-xxxxx-xxxxx-xxxxx-xxxxx-xxxxx-xxx # Node Operator principal from step VI
NODE_PROVIDER_NAME="My Company"
NODE_ALLOWANCE=8
DC_ID=dl1

./ic-admin \
--nns-url https://ic0.app \
-s ~/.config/dfx/identity/node-provider-hotkey/identity.pem \
propose-to-add-node-operator \
$NODE_PROVIDER_PRINCIPAL \
--summary "Node provider '$NODE_PROVIDER_NAME' is adding $NODE_ALLOWANCE nodes in the $DC_ID data center" \
--proposer $NEURON_ID \
--node-operator-principal-id $NODE_OPERATOR_PRINCIPAL \
--node-allowance $NODE_ALLOWANCE \
--dc-id $DC_ID
</syntaxhighlight>

# Find the proposal on https://dashboard.internetcomputer.org/governance and wait until it's executed before proceeding to next step.

== X. Onboard nodes ==

# Follow the instructions to onboard new nodes:
#* Gen2 - For NP's onboarding in 2023 and later
#** [[IC OS Installation Runbook]]
#* Gen1 - For NP's participating in the IC before 2023
#**[[IC OS Installation Runbook - PowerEdge R6525]]
#**[[IC OS Installation Runbook - Supermicro]]
# Verify that all the nodes were successfully onboarded by checking their status on the dashboard is set to either “Up” or “Unassigned”, or by checking the output from <code>ic-admin get-topology</code> command.
#* The internal dashboard can be searched by your provider principal.
[[File:onboarded_nodes.png|1024px|onboarded nodes]]

== See Also ==
* '''The Internet Computer project website (hosted on the IC): [https://internetcomputer.org/ internetcomputer.org]'''
* '''IC Node Provider Matrix/Element channel:''' https://app.element.io/#/room/#ic-node-providers:matrix.org

Node Provider FAQ

2023-05-25T16:06:46Z

Gary.mcelroy: /* Governance/Remuneration */ Added "Node Machine" question

== General ==

== Governance/Remuneration ==

== Technical ==

=== What is a node? Just one machine? ===
Yes.

Sometimes "Node Machine" - a single server participating with the IC - is used to differentiate from "[https://internetcomputer.org/docs/current/references/glossary/#node Node"], which is sometimes used to refer to the software that runs the IC.

=== When purchasing node hardware can I deviate from the [[Node Machine Hardware#Gen 2 Node Machine|node hardware requirements]]? ===
The hardware components must meet the [[Node Machine Hardware#Generic specification|generic specification]] of the Gen2 node hardware. This is verified by the IC-OS installer. The installer will fail if the expected components are not found.

Small details are expected to be different between vendors such as SSD manufacturer, chassis model, etc..

=== Do I need to configure RAID on my node machines? ===
No. RAID (hardware or software) should not be attempted. The IC-OS installer will verify there are 5x independent 6.4TB NVMe SSD’s and prepare them appropriately - formatting all disks as it installs.

IC-OS uses a ‘striped’ LVM volume across all the disks (technically a software RAID 0).

'''What about redundancy?''' Replica nodes provide redundancy at a higher level than disk redundancy.

==See also==

* [[Node Provider Documentation]]

Node Provider Documentation

2023-05-25T13:52:26Z

Gary.mcelroy: /* Onboarding for accepted Node Providers */ Added link to FAQ

==Introduction==

ICP runs on a [[Sovereign Network]] that is a governed by a DAO using [[Deterministic Decentralization | deterministic decentralization]] to maximize its [[Decentralization in ICP: Infrastructure Governance | decentralization]]. To be part of the ICP infrastructure, any potential node providers can submit NNS proposals to the DAO controlling the ICP blockchain. ICP community then votes on whether to include the node provider. Node providers invest in and operate the node hardware which powers the Internet Computer. Running these nodes in data centers provides the high performance and the cost-effectiveness of the Internet Computer. Every node provider is allowed a limited amount of nodes.

This article is the hub for ICP node provider documentation.

==Node Provider Tokenomics & Remuneration==
Node providers receive rewards (remuneration) for operating node machines that run the IC network. The single source of truth for node provider rewards is the NNS, where changes can only be made through NNS proposals adopted by the IC community.

This page summarizes the current node provider rewards and serves to discuss proposals for future reward models: [[Node Provider Remuneration]]

==Node Machine Hardware Requirements==

Node machines on ICP network need to keep up with the requirements of the network, please see: [[Node Machine Hardware#Gen_2 | Gen-2 Node Machine Hardware]].

==Submitting Proposal to Join the Network==

As part of the process to become a node provider, a candidate node provider has to declare their intent and self-identify so the ICP DAO can make an informed decision. Please see [[Node Provider Self-declaration]] for more info.

==Onboarding for accepted Node Providers==

===Gen-2 Documentation===

These articles are for candidate node providers considering becoming node providers or node providers recently accepted by the ICP DAO. The term "Gen-2" refers to "Generation 2", which is the current set of protocols for new node providers.

* [[Node Machine Hardware#Gen_2 |Gen-2 Node Machine Hardware]]
* [[Node Provider Network Setup Guide]]
* [[Node Provider Remuneration]]
* [[Node Provider Self-declaration]]
* [[Node Provider Onboarding]]
* [[IC OS Installation Runbook]] for Gen-2
* [[Node Provider FAQ]]

Users accepted by the ICP DAO to be a node provider, can follow these instructions to add their node: [[Node Provider Onboarding]].

===Node Provider Troubleshooting===
* [[Node Provider Troubleshooting]]
** [[Possible Node Onboarding Errors]]
** [[Unhealthy Nodes]]
** [[Updating Firmware]]
** [[iDRAC access and TSR logs]]

===Gen-1 Documentation===
The first batch of ICP node providers joined under Gen-1 (Generation 1). These documents are for those legacy node providers.

* [[IC OS Installation Runbook - Dell Poweredge]] for Gen-1
* [[IC OS Installation Runbook - Supermicro]] for Gen-1
* [[Storage Runbook]] for Gen-1

Node Provider Documentation

2023-05-25T13:51:33Z

Gary.mcelroy: /* Onboarding for accepted Node Providers */ Fixed formatting

==Introduction==

ICP runs on a [[Sovereign Network]] that is a governed by a DAO using [[Deterministic Decentralization | deterministic decentralization]] to maximize its [[Decentralization in ICP: Infrastructure Governance | decentralization]]. To be part of the ICP infrastructure, any potential node providers can submit NNS proposals to the DAO controlling the ICP blockchain. ICP community then votes on whether to include the node provider. Node providers invest in and operate the node hardware which powers the Internet Computer. Running these nodes in data centers provides the high performance and the cost-effectiveness of the Internet Computer. Every node provider is allowed a limited amount of nodes.

This article is the hub for ICP node provider documentation.

==Node Provider Tokenomics & Remuneration==
Node providers receive rewards (remuneration) for operating node machines that run the IC network. The single source of truth for node provider rewards is the NNS, where changes can only be made through NNS proposals adopted by the IC community.

This page summarizes the current node provider rewards and serves to discuss proposals for future reward models: [[Node Provider Remuneration]]

==Node Machine Hardware Requirements==

Node machines on ICP network need to keep up with the requirements of the network, please see: [[Node Machine Hardware#Gen_2 | Gen-2 Node Machine Hardware]].

==Submitting Proposal to Join the Network==

As part of the process to become a node provider, a candidate node provider has to declare their intent and self-identify so the ICP DAO can make an informed decision. Please see [[Node Provider Self-declaration]] for more info.

==Onboarding for accepted Node Providers==

===Gen-2 Documentation===

These articles are for candidate node providers considering becoming node providers or node providers recently accepted by the ICP DAO. The term "Gen-2" refers to "Generation 2", which is the current set of protocols for new node providers.

* [[Node Machine Hardware#Gen_2 |Gen-2 Node Machine Hardware]]
* [[Node Provider Network Setup Guide]]
* [[Node Provider Remuneration]]
* [[Node Provider Self-declaration]]
* [[Node Provider Onboarding]]
* [[IC OS Installation Runbook]] for Gen-2

Users accepted by the ICP DAO to be a node provider, can follow these instructions to add their node: [[Node Provider Onboarding]].

===Node Provider Troubleshooting===
* [[Node Provider Troubleshooting]]
** [[Possible Node Onboarding Errors]]
** [[Unhealthy Nodes]]
** [[Updating Firmware]]
** [[iDRAC access and TSR logs]]

===Gen-1 Documentation===
The first batch of ICP node providers joined under Gen-1 (Generation 1). These documents are for those legacy node providers.

* [[IC OS Installation Runbook - Dell Poweredge]] for Gen-1
* [[IC OS Installation Runbook - Supermicro]] for Gen-1
* [[Storage Runbook]] for Gen-1

Node Provider FAQ

2023-05-25T13:48:24Z

Gary.mcelroy: Started NP FAQ. Added to items.

== General ==

== Governance/Remuneration ==

== Technical ==

=== When purchasing node hardware can I deviate from the [[Node Machine Hardware#Gen 2 Node Machine|node hardware requirements]]? ===
The hardware components must meet the [[Node Machine Hardware#Generic specification|generic specification]] of the Gen2 node hardware. This is verified by the IC-OS installer. The installer will fail if the expected components are not found.

Small details are expected to be different between vendors such as SSD manufacturer, chassis model, etc..

=== Do I need to configure RAID on my node machines? ===
No. RAID (hardware or software) should not be attempted. The IC-OS installer will verify there are 5x independent 6.4TB NVMe SSD’s and prepare them appropriately - formatting all disks as it installs.

IC-OS uses a ‘striped’ LVM volume across all the disks (technically a software RAID 0).

'''What about redundancy?''' Replica nodes provide redundancy at a higher level than disk redundancy.

Node Provider Machine Hardware Guide

2023-05-25T13:42:17Z

Gary.mcelroy: /* Gen 1 Node Machine */ Clarified which section to use

== What are the Hardware Requirements for Node Machines? ==
Node providers operate one or more node machines than run in the IC network. Gen1 Hardware requirements have been used by Node Providers to set up node machines during Genesis launch.

The Gen2 Hardware requirements have been defined for the further growth of the IC network. The specifications for the Gen2 node machines are generic (instead of vendor specific) and support VM memory encryption and attestation, which will be needed in future features on the IC.

Below are the up-to-date specifications for both the Gen2 node machines and Gen1 node machines.
== Gen 2 Node Machine ==
=== Generic specification ===
{| class="wikitable"
|-
| Dual Socket AMD EPYC Milan CPU - Recommended: [https://en.wikichip.org/wiki/amd/epyc/7313 7313] (16C/32T 3 Ghz)
[https://en.wikichip.org/wiki/amd/epyc#7003_Series_.28Zen_3.29 optionally] 7343, 7373, 73F3
|-
| 16x 32GB RDIMM, 3200MT/s, Dual Rank
|-
| 5x 6.4TB NVMe Mixed Mode (DWPD >= 3)
|-
| Dual Port 10G SFP or BASE-T
|-
| TPM 2.0
|}

=== Validated Configurations ===
DFINITY has [https://forum.dfinity.org/t/draft-motion-proposal-new-hardware-specification-and-remuneration-for-ic-nodes/14202/14?u=garym validated] the following Gen2 hardware configurations.
==== Dell ====
{| class="wikitable"
|-
| 2 || AMD EPYC 7343 3.2GHz, 16C/32T, 128M Cache (190W)
|-
| 16 || 32GB RDIMM, 3200MT/s, Dual Rank 16Gb BASE x8
|-
| 5 || 6.4TB Enterprise NVMe Mixed Use AG Drive U.2 Gen4 with carrier
|-
| 1 || PowerEdge R6525 Motherboard, with 2 x 1Gb Onboard LOM (BCM5720)MLK V2
|-
| 2 || Dual, Hot-plug, Redundant Power Supply (1+1) 1100W, Mixed Mode Titanium
|-
| 1 || Intel X710 Dual Port 10GbE SFP+, OCP NIC 3.0
|-
| 1 || Trusted Platform Module 2.0 V3
|}

==== ASUS ====
{| class="wikitable"
|-
| 2 || AMD EPYC 7313 (3,00 GHz, 16-Core, 128 MB)
|-
| 16 || 32GB ECC Reg ATP DDR4 3200 RAM
|-
| 5 || 6.4 TB NVMe Kioxia SSD 3D-NAND TLC U.3 (Kioxia CM6-V)
|-
| 1 || Asus Mainboard KMPP-D32 Series (without OCP 3.0, without Pike)
|-
| 2 || 1600 Watt redundant PSU
|-
| 1 || Broadcom 25 Gigabit P225P SFP28 Dual Port Network Card
|-
|0
|TPM 2.0*
|}
<nowiki>*</nowiki> Note Gen2 machines require TPM 2.0 hardware but this specific test machine did not include it.

==== Supermicro ====
{| class="wikitable"
|-
| 2 || AMD EPYC 7313 (3,00 GHz, 16-Core, 128 MB)
|-
| 16 || 32GB ECC Reg DDR4 3200 RAM
|-
| 5 || 6.4 TB NVMe TLC SSD, PCIe 4.0 x4, U.3 2.5", 3 DWPD
|-
| 1 || SuperMicro H12DSU-iN
|-
| 2 || 1200 Watt redundant PSU
|-
| 1 || Dual Port 10GBase-T Network Adapter Intel® X710-TM4*
|-
|1
|Dual Port 10GbE Network Adapter Intel® X710-TM4, SFP+*
|-
| 1 || Trusted Platform Module 2.0
|}
==== Currently pending validation: Gigabyte ====
{| class="wikitable"
|-
| 2 || AMD EPYC 7313 (3,00 GHz, 16-Core, 128 MB)
|-
| 16 || 32GB ECC Reg DDR4 3200 RAM
|-
| 5 || 6.4 TB NVMe TLC SSD, PCIe 4.0 x4, U.3 2.5", 3 DWPD
|-
| 1 || Gigabyte MZ92-FS1 Rev. 3.0
|-
| 2 || 1200 Watt redundant PSU
|-
| 1 || Dual Port 1000Base-T Network Adapter Intel® I350-AM2*
|-
|1
|Dual Port 10GbE Network Adapter, Broadcom NetXtreme E-Series*
|-
| 1 || Trusted Platform Module 2.0
|}
<nowiki>*</nowiki> Only one dual port network adaptor is required.

== Gen 1 Node Machine ==
If you're a Node Provider acquiring machines to join the IC, use the Gen2 section above. This section is for Gen1 Node Providers who joined before 2022.

=== Node Machine Type 1 - Dell ===
{| class="wikitable"
|-
| 1 || R6525
|-
| 1 || Chassis - Supports Up to 10 NVMe drives, 12 drives total
|-
| 1 || Dual 1 GB on Motherboard
|-
| 3 || Low Profile PCIe Slots
|-
| - || 3 Year Basic NBD Support
|-
| - || iDrac Enterprise
|-
| 1 || Dual port 10GbE Base - T Adapter Broadcom, PCIe Low Profile
|-
| 10 || 3.2TB NVMe, Mixed Use, 2.5" with Carrier
|-
| 16 || 32GB RDIMM (3200MT/s)
|-
| 2 || AMD 7302 3GHz, 16C/32T, 128M, 155W, 3200
|-
| 1 || Single Power Supply (800W)
|-
| 1 || C13-C14, 3M, 125V 15A Power Cored
|-
|}

=== Node Machine Type 1 - SuperMicro ===
{| class="wikitable"
|-
| 1 || AS - 1023US - TR4
|-
| 2 || Rome 7302 DP/UP 16C/32T 3.0
|-
| 16 || 32GB DDR4-3200 2Rx4 ECC REG DIMM
|-
| 5 || Samsung PM983 3.2TB NVMe PCIE/SATA Hybrid M.2 & 1 PCIE
|-
| 2 || 800W Power Supply
|-
| 1 || Std LP 2-port 10G RJ45, Intel x540
|-
| 5 || Micron 5300 PRO 7.4TB, SATA, 2.5', 3D TLC, .6DWPD (with Caddie)
|-
| 1 || C13/C14 13A Power Cord
|}

=== Node Machine Type 2 - Dell ===
{| class="wikitable"
|-
| 1 || R6515
|-
| 1 || 3.5" Chassis with up to 4 Hot-Plug Hard Drives and OS RAID
|-
| 1 || Dual 1 Gb on Motherboard
|-
| 3 || Low Profile PCIe Slots
|-
| 1 || Standard Fan
|-
| - || 3 Year Basic NBD Support
|-
| - || iDrac Enterprise
|-
| 2 || Dual Port 10GbE Base - T Adapter Broadcom, PCIe LOw Profile
|-
| 2 || 480GB SSD SATA Mix Use 6Gbps 512 2.5in Hot-Plug AG Drive, 3.5in
|-
| 4 || 8GB RDIMM, 3200 MT/s, Single Rank
|-
| 1 || AMD EPYC 7232P 3.10GHz, 8C/16T, 64M Cache (120W) DDR4-3200
|-
| 1 || Dual Hot-Plug Redundant Power Supply (1+1), 550W
|-
| 2 || Jumper Cord - C13/C14, .6M, 250V, 13A
|}

Node Provider Machine Hardware Guide

2023-05-23T15:55:46Z

Gary.mcelroy: /* Validated configuration: ASUS */ Fixed line spacing for real this time

== What are the Hardware Requirements for Node Machines? ==
Node providers operate one or more node machines than run in the IC network. Gen1 Hardware requirements have been used by Node Providers to set up node machines during Genesis launch.

The Gen2 Hardware requirements have been defined for the further growth of the IC network. The specifications for the Gen2 node machines are generic (instead of vendor specific) and supports VM memory encryption and attestation which will be needed in future features on the IC.

Below are the up-to-date specifications for both the Gen2 node machines and Gen1 node machines.

== Gen 2 Node Machine ==
=== Generic specification Gen2 Node Machine ===
{| class="wikitable"
|-
| Dual Socket AMD EPYC Milan CPU - Recommended: [https://en.wikichip.org/wiki/amd/epyc/7313 7313] (16C/32T 3 Ghz)
[https://en.wikichip.org/wiki/amd/epyc#7003_Series_.28Zen_3.29 optionally] 7343, 7373, 73F3
|-
| 16x 32GB RDIMM, 3200MT/s, Dual Rank
|-
| 5x 6.4TB NVMe Mixed Mode (DWPD >= 3)
|-
| Dual Port 10G SFP or BASE-T
|-
| TPM 2.0
|}
 
Note the NVMe drives should be recognized by Linux as NVMe (i.e., show up as `/dev/nvme*` devices). SATA backplanes or any other hardware which prevents this should not be used.
 
=== Validated Configurations ===
DFINITY has [https://forum.dfinity.org/t/draft-motion-proposal-new-hardware-specification-and-remuneration-for-ic-nodes/14202/14?u=garym validated] the following Gen2 hardware configurations.
 
==== Validated configuration: Dell ====
{| class="wikitable"
|-
| 2 || AMD EPYC 7343 3.2GHz, 16C/32T, 128M Cache (190W)
|-
| 16 || 32GB RDIMM, 3200MT/s, Dual Rank 16Gb BASE x8
|-
| 5 || 6.4TB Enterprise NVMe Mixed Use AG Drive U.2 Gen4 with carrier
|-
| 1 || PowerEdge R6525 Motherboard, with 2 x 1Gb Onboard LOM (BCM5720)MLK V2
|-
| 2 || Dual, Hot-plug, Redundant Power Supply (1+1) 1100W, Mixed Mode Titanium
|-
| 1 || Intel X710 Dual Port 10GbE SFP+, OCP NIC 3.0
|-
| 1 || Trusted Platform Module 2.0 V3
|}

==== Validated configuration: ASUS ====
{| class="wikitable"
|-
| 2 || AMD EPYC 7313 (3,00 GHz, 16-Core, 128 MB)
|-
| 16 || 32GB ECC Reg ATP DDR4 3200 RAM
|-
| 5 || 6.4 TB NVMe Kioxia SSD 3D-NAND TLC U.3 (Kioxia CM6-V)
|-
| 1 || Asus Mainboard KMPP-D32 Series (without OCP 3.0, without Pike)
|-
| 2 || 1600 Watt redundant PSU
|-
| 1 || Broadcom 25 Gigabit P225P SFP28 Dual Port Network Card
|-
|0
|TPM 2.0*
|}
<nowiki>*</nowiki> Note Gen2 machines require TPM 2.0 hardware but this specific test machine did not include it. 
==== Validated configurations: Supermicro & Gigabyte ====
Validation is being re-run on Supermicro and Gigabyte machines which match the spec. This section will be updated when those results are ready.

== Gen 1 Node Machine ==
=== Node Machine Type 1 - Dell ===
{| class="wikitable"
|-
| 1 || R6525
|-
| 1 || Chassis - Supports Up to 10 NVMe drives, 12 drives total
|-
| 1 || Dual 1 GB on Motherboard
|-
| 3 || Low Profile PCIe Slots
|-
| - || 3 Year Basic NBD Support
|-
| - || iDrac Enterprise
|-
| 1 || Dual port 10GbE Base - T Adapter Broadcom, PCIe Low Profile
|-
| 10 || 3.2TB NVMe, Mixed Use, 2.5" with Carrier
|-
| 16 || 32GB RDIMM (3200MT/s)
|-
| 2 || AMD 7302 3GHz, 16C/32T, 128M, 155W, 3200
|-
| 1 || Single Power Supply (800W)
|-
| 1 || C13-C14, 3M, 125V 15A Power Cored
|-
|}
 
=== Node Machine Type 1 - SuperMicro ===
{| class="wikitable"
|-
| 1 || AS - 1023US - TR4
|-
| 2 || Rome 7302 DP/UP 16C/32T 3.0
|-
| 16 || 32GB DDR4-3200 2Rx4 ECC REG DIMM
|-
| 5 || Samsung PM983 3.2TB NVMe PCIE/SATA Hybrid M.2 & 1 PCIE
|-
| 2 || 800W Power Supply
|-
| 1 || Std LP 2-port 10G RJ45, Intel x540
|-
| 5 || Micron 5300 PRO 7.4TB, SATA, 2.5', 3D TLC, .6DWPD (with Caddie)
|-
| 1 || C13/C14 13A Power Cord
|}
 
=== Node Machine Type 2 - Dell ===
{| class="wikitable"
|-
| 1 || R6515
|-
| 1 || 3.5" Chassis with up to 4 Hot-Plug Hard Drives and OS RAID
|-
| 1 || Dual 1 Gb on Motherboard
|-
| 3 || Low Profile PCIe Slots
|-
| 1 || Standard Fan
|-
| - || 3 Year Basic NBD Support
|-
| - || iDrac Enterprise
|-
| 2 || Dual Port 10GbE Base - T Adapter Broadcom, PCIe LOw Profile
|-
| 2 || 480GB SSD SATA Mix Use 6Gbps 512 2.5in Hot-Plug AG Drive, 3.5in
|-
| 4 || 8GB RDIMM, 3200 MT/s, Single Rank
|-
| 1 || AMD EPYC 7232P 3.10GHz, 8C/16T, 64M Cache (120W) DDR4-3200
|-
| 1 || Dual Hot-Plug Redundant Power Supply (1+1), 550W
|-
| 2 || Jumper Cord - C13/C14, .6M, 250V, 13A
|}

Node Provider Machine Hardware Guide

2023-05-23T15:55:02Z

Gary.mcelroy: /* Gen 2 Node Machine */ Added note about missing TPM on ASUS

== What are the Hardware Requirements for Node Machines? ==
Node providers operate one or more node machines than run in the IC network. Gen1 Hardware requirements have been used by Node Providers to set up node machines during Genesis launch.

The Gen2 Hardware requirements have been defined for the further growth of the IC network. The specifications for the Gen2 node machines are generic (instead of vendor specific) and supports VM memory encryption and attestation which will be needed in future features on the IC.

Below are the up-to-date specifications for both the Gen2 node machines and Gen1 node machines.

== Gen 2 Node Machine ==
=== Generic specification Gen2 Node Machine ===
{| class="wikitable"
|-
| Dual Socket AMD EPYC Milan CPU - Recommended: [https://en.wikichip.org/wiki/amd/epyc/7313 7313] (16C/32T 3 Ghz)
[https://en.wikichip.org/wiki/amd/epyc#7003_Series_.28Zen_3.29 optionally] 7343, 7373, 73F3
|-
| 16x 32GB RDIMM, 3200MT/s, Dual Rank
|-
| 5x 6.4TB NVMe Mixed Mode (DWPD >= 3)
|-
| Dual Port 10G SFP or BASE-T
|-
| TPM 2.0
|}
 
Note the NVMe drives should be recognized by Linux as NVMe (i.e., show up as `/dev/nvme*` devices). SATA backplanes or any other hardware which prevents this should not be used.
 
=== Validated Configurations ===
DFINITY has [https://forum.dfinity.org/t/draft-motion-proposal-new-hardware-specification-and-remuneration-for-ic-nodes/14202/14?u=garym validated] the following Gen2 hardware configurations.
 
==== Validated configuration: Dell ====
{| class="wikitable"
|-
| 2 || AMD EPYC 7343 3.2GHz, 16C/32T, 128M Cache (190W)
|-
| 16 || 32GB RDIMM, 3200MT/s, Dual Rank 16Gb BASE x8
|-
| 5 || 6.4TB Enterprise NVMe Mixed Use AG Drive U.2 Gen4 with carrier
|-
| 1 || PowerEdge R6525 Motherboard, with 2 x 1Gb Onboard LOM (BCM5720)MLK V2
|-
| 2 || Dual, Hot-plug, Redundant Power Supply (1+1) 1100W, Mixed Mode Titanium
|-
| 1 || Intel X710 Dual Port 10GbE SFP+, OCP NIC 3.0
|-
| 1 || Trusted Platform Module 2.0 V3
|}

==== Validated configuration: ASUS ====
{| class="wikitable"
|-
| 2 || AMD EPYC 7313 (3,00 GHz, 16-Core, 128 MB)
|-
| 16 || 32GB ECC Reg ATP DDR4 3200 RAM
|-
| 5 || 6.4 TB NVMe Kioxia SSD 3D-NAND TLC U.3 (Kioxia CM6-V)
|-
| 1 || Asus Mainboard KMPP-D32 Series (without OCP 3.0, without Pike)
|-
| 2 || 1600 Watt redundant PSU
|-
| 1 || Broadcom 25 Gigabit P225P SFP28 Dual Port Network Card
|-
|0
|TPM 2.0*
|}
<nowiki>*</nowiki> Note Gen2 machines require TPM 2.0 hardware but this specific test machine did not include it. 
==== Validated configurations: Supermicro & Gigabyte ====
Validation is being re-run on Supermicro and Gigabyte machines which match the spec. This section will be updated when those results are ready.

== Gen 1 Node Machine ==
=== Node Machine Type 1 - Dell ===
{| class="wikitable"
|-
| 1 || R6525
|-
| 1 || Chassis - Supports Up to 10 NVMe drives, 12 drives total
|-
| 1 || Dual 1 GB on Motherboard
|-
| 3 || Low Profile PCIe Slots
|-
| - || 3 Year Basic NBD Support
|-
| - || iDrac Enterprise
|-
| 1 || Dual port 10GbE Base - T Adapter Broadcom, PCIe Low Profile
|-
| 10 || 3.2TB NVMe, Mixed Use, 2.5" with Carrier
|-
| 16 || 32GB RDIMM (3200MT/s)
|-
| 2 || AMD 7302 3GHz, 16C/32T, 128M, 155W, 3200
|-
| 1 || Single Power Supply (800W)
|-
| 1 || C13-C14, 3M, 125V 15A Power Cored
|-
|}
 
=== Node Machine Type 1 - SuperMicro ===
{| class="wikitable"
|-
| 1 || AS - 1023US - TR4
|-
| 2 || Rome 7302 DP/UP 16C/32T 3.0
|-
| 16 || 32GB DDR4-3200 2Rx4 ECC REG DIMM
|-
| 5 || Samsung PM983 3.2TB NVMe PCIE/SATA Hybrid M.2 & 1 PCIE
|-
| 2 || 800W Power Supply
|-
| 1 || Std LP 2-port 10G RJ45, Intel x540
|-
| 5 || Micron 5300 PRO 7.4TB, SATA, 2.5', 3D TLC, .6DWPD (with Caddie)
|-
| 1 || C13/C14 13A Power Cord
|}
 
=== Node Machine Type 2 - Dell ===
{| class="wikitable"
|-
| 1 || R6515
|-
| 1 || 3.5" Chassis with up to 4 Hot-Plug Hard Drives and OS RAID
|-
| 1 || Dual 1 Gb on Motherboard
|-
| 3 || Low Profile PCIe Slots
|-
| 1 || Standard Fan
|-
| - || 3 Year Basic NBD Support
|-
| - || iDrac Enterprise
|-
| 2 || Dual Port 10GbE Base - T Adapter Broadcom, PCIe LOw Profile
|-
| 2 || 480GB SSD SATA Mix Use 6Gbps 512 2.5in Hot-Plug AG Drive, 3.5in
|-
| 4 || 8GB RDIMM, 3200 MT/s, Single Rank
|-
| 1 || AMD EPYC 7232P 3.10GHz, 8C/16T, 64M Cache (120W) DDR4-3200
|-
| 1 || Dual Hot-Plug Redundant Power Supply (1+1), 550W
|-
| 2 || Jumper Cord - C13/C14, .6M, 250V, 13A
|}

Node Provider Onboarding

2023-05-19T16:22:18Z

Gary.mcelroy: /* V. Setup the HSM */ Removed dead link

Learn how to participate in the Internet Computer network as a Node Provider and to receive rewards for supporting the network.

== Requirements ==

* [https://wiki.internetcomputer.org/wiki/Node_provider_hardware Node Hardware]
* Rack space with a 10Gb connectivity, RJ45 terminated on the nodes
* [[Node Provider Network Setup Guide#The Bare Minimum Network Requirements|Public IP addresses]]:
** One /64 IPv6 range
** One IPv4 address for every 4 node machines
* [https://www.ledger.com/ Hardware wallet]
* [https://shop.nitrokey.com/shop/product/nkhs2-nitrokey-hsm-2-7/ NitroKey HSM]
* 11 ICP (10 of which are to be staked for the NNS proposal deposit)
* Basic understanding of [[Neurons 101|neurons]], [https://internetcomputer.org/docs/current/tokenomics/nns/nns-staking-voting-rewards staking], and [[Governance of the Internet Computer|governance]] proposals. For instance, understanding what it means to stake a neuron for 8 years.
* The technical knowledge to understand some minor steps that are not explicitly mentioned in these instructions. For instance, when to insert an HSM.

'''Note:''' Please allocate at least 0.5 day for going through the first part, i.e., the registration of a new NP. It may even take a couple of days, depending on how quickly the community votes for the proposals. There is a also fair amount of complexity and the technical knowledge that needs to be absorbed in order to complete the steps. But this only needs to be done once. 
The next step, going to the DC and bringing up and onboarding the machines, is much quicker. Estimate to spend 10-15 minutes per machine. This time should go down to ~5 minutes as you gain experience. Also, multiple machines can be brought up in parallel.

== I. Install the required tools ==
===''' A. Install ic-admin '''===

<code>ic-admin</code> is the tool used to create and submit NNS proposals.

==== MacOS ====
# Retrieve the file <syntaxhighlight lang="shell">
curl "https://download.dfinity.systems/ic/7445081734e6d896d090295967d50710975c4f25/openssl-static-binaries/x86_64-darwin/ic-admin.gz" -o - | gunzip > ./ic-admin
chmod +x ./ic-admin
</syntaxhighlight>
# Verify the binary <syntaxhighlight lang="shell">
diff <(shasum -a 256 ./ic-admin | cut -d' ' -f1) <(echo 3f75026d2f28f171068e332a42c82a2795c93fbf5ab351baef30b30eb901cdba) && echo "ic-admin checksum matches" || echo "***ERROR***: ic-admin checksum does not match"
</syntaxhighlight>

==== Linux ====
NOTE: The instructions below have been tested with the Ubuntu 20.04 release.
# Retrieve the file <syntaxhighlight lang="shell">
curl "https://download.dfinity.systems/ic/7445081734e6d896d090295967d50710975c4f25/openssl-static-binaries/x86_64-linux/ic-admin.gz" -o - | gunzip > ./ic-admin
chmod +x ./ic-admin
</syntaxhighlight>
# Verify the binary <syntaxhighlight lang="shell">
diff <(shasum -a 256 ./ic-admin | cut -d' ' -f1) <(echo e29bb9cc462e800b8b960ad49c412e5f5fdbb5ae2ae9fde0c13058422ba32802) && echo "ic-admin checksum matches" || echo "***ERROR***: ic-admin checksum does not match"
</syntaxhighlight>

===''' B. Install dfx '''===

# <code>dfx</code> allows generating a neuron hotkey, among other things <syntaxhighlight lang="shell">
$ DFX_VERSION=0.9.3 sh -ci "$(curl -fsSL https://sdk.dfinity.org/install.sh)"
</syntaxhighlight>
# Verify that the version is 0.9.3 <syntaxhighlight lang="shell">
$ export PATH=$HOME/bin:$PATH
$ dfx --version dfx 0.9.3
</syntaxhighlight>
# Create an identity for the Node Provider '''Hotkey''' <syntaxhighlight lang="shell">
$ dfx identity new node-provider-hotkey
Creating identity: "node-provider-hotkey".
Created identity: "node-provider-hotkey".
$ dfx --identity node-provider-hotkey identity get-principal
wuyst-x5tpn-g5wri-mp3ps-vjtba-de3xs-w5xgb-crvek-tucbe-o5rqi-mae
</syntaxhighlight>
# '''Note:''' The node provider hotkey is NOT the node provider principal. This is the hotkey that is used for the NNS proposal submissions only.

== II. Create and Manage Neuron via NNS Frontend Dapp and Internet Identity ==

# Setup your hardware wallet: https://medium.com/dfinity/integrating-ledger-nano-with-the-nns-front-end-dapp-user-manual-9c5600925e16
# Send at least 11 ICPs to the hardware wallet address.
# Navigate to Neurons tab and create a Neuron by staking at least 10 ICP from your hardware wallet. Staking more ICP works as well, but 10 is the minimum.
# IMPORTANT! Confirm the transaction on your hardware wallet.
#: [[File:-docs-stake_neuron_1.png|1024px|stake neuron]]
#:
# After the Neuron has been created successfully, confirm to add NNS Dapp as hotkey in the dialogue and on your hardware wallet, and close the dialog after the action completes.
#: [[File:-docs-stake_neuron_2.png|1024px|neuron id]]
# Set the dissolve delay to at least 6 months, and confirm the choice in the dialogue and on your hardware wallet. After the action completes, you can close the "Follow Neurons".
#:
#: [[File:dissolve_delay.png|480px|neuron id]]
# You will now see a Neuron listed with its ID. Copy the Neuron ID, since you will need it in the next steps to place the necessary proposals.
#: [[File:Neuron id.png|1024px]]

== III. Add hotkeys ==

# Select the Neuron you just created to open Neuron management view and press “Add hotkey” button.
#: [[File:Hotkey 1.png|800px]]
# A dialog will pop up where you can enter the principal you generated in step 2 (output from command <code>dfx --identity node-provider-hotkey identity get-principal</code>). This will allow you to submit NNS proposals using <code>ic-admin</code> and will not be used for anything else. 
#:Press the confirm button and confirm the transactions on your hardware wallet. 
#: [[File:Hotkey 2.png|800px]]
#:
# Get the Ledger Hardware Wallet Principal Id: Navigate back to ICP page and select your Ledger hardware wallet account. You will need to use this Ledger Hardware Wallet principal as the Node Provider principal in order to get the rewards directly into the secure hardware wallet.
[[File:Node provider principal 1.png|1024px]]
[[File:Node provider principal 2.png|800px]]
# Copy and save this Node Provider principal by clicking on the copy icon after the principal id. You'll need it in the next steps.

<syntaxhighlight lang="shell">
NODE_PROVIDER_PRINCIPAL=fharn-5vyi2-4xb4a-64yyi-3jpmj-pga23-mxy25-d5uim-fqcro-eoefh-tae # Ledger Hardware Wallet principal, from the NNS FrontEnd dapp https://nns.ic0.app/
</syntaxhighlight>

== IV. Configure your HSM ==
It's first necessary to install the necessary tools.

=== MacOS ===
# Download this OpenSC binary: https://github.com/OpenSC/OpenSC/releases/download/0.22.0/OpenSC-0.22.0.dmg
# Double click the DMG image that you downloaded and then double click the OpenSC PKG file.
# If your system doesn't allow the installation software from an unidentified developer please follow these steps or contact your system administrator:
#* Choose the Apple menu > System Preferences > click Security and Privacy.
#* Click the lock Icon to unlock it, then enter an administrator name and password.
#* Ensure that you're on the tab named “General”.
#* You should see the OpenSC app and you should be able to enable its installation by choosing “Open anyway”.
# Click continue and install until the installation is complete.

=== Linux ===

NOTE: The instructions below have been tested with the Ubuntu 20.04 release.

# Install pcscd and opensc
#: <syntaxhighlight lang="shell">
sudo add-apt-repository universe
sudo apt update
sudo apt install pcscd opensc
</syntaxhighlight>

== V. Setup the HSM ==

# Initialize the HSM. <syntaxhighlight lang="shell">
sc-hsm-tool --initialize --so-pin 3537363231383830 --pin 358138
</syntaxhighlight>
# Change the HSM so-pin.
#* '''WARNING:''' The new HSM so pin must have 16 hexadecimal digits. This is not very well known, and some HSM users have lost access to a Nitrokey HSM because they tried using regular characters and the command below accepted it.
#* '''Do NOT change the user pin. It must remain as the default for the onboarding scripts to work'''<syntaxhighlight lang="shell">
pkcs11-tool --login --login-type so --so-pin 3537363231383830 --change-pin
</syntaxhighlight>
# Create a keypair on the HSM. Enter the default pin 358138 when prompted.
#*'''Note:''' Key backup may be possible with [https://github.com/OpenSC/OpenSC/wiki/SmartCardHSM#using-key-backup-and-restore these instructions].

== VI. Get the node operator principal from the HSM ==
# Configure dfx identity (skip this step if you already configured it for an other HSM).
#* '''Note:''' Depending on your installation, the path to the <code>--hsm-pkcs11-lib-path</code> might be different on your platform. You can locate the correct path with the following command: <syntaxhighlight lang="shell">
find / -name opensc-pkcs11.so 2> /dev/null
</syntaxhighlight>
#* MacOS <syntaxhighlight lang="shell">
dfx identity new node-operator-hsm --hsm-key-id 01 --hsm-pkcs11-lib-path /Library/OpenSC/lib/opensc-pkcs11.so
</syntaxhighlight>
#* Linux <syntaxhighlight lang="shell">
dfx identity new node-operator-hsm --hsm-key-id 01 --hsm-pkcs11-lib-path /usr/lib/x86_64-linux-gnu/opensc-pkcs11.so
</syntaxhighlight>
# Get the principal. <syntaxhighlight lang="shell">
$ NODE_OPERATOR_PRINCIPAL=$(DFX_HSM_PIN=358138 dfx --identity node-operator-hsm identity get-principal)
$ echo $NODE_OPERATOR_PRINCIPAL

uqquy-76uhn-2mys5-xxxxx-xxxxx-xxxxx-xxxxx-xxxxx-xxxxx-xxxxx-xxx
</syntaxhighlight>

== VII. Register your NP principal to the network ==

In the next codeblock:
* Replace the <code>NEURON_ID</code> value with your neuron ID from the NNS Frontend Dapp
* Replace the <code>NODE_PROVIDER_PRINCIPAL</code> value with the Ledger Hardware Wallet principal that you got from the NNS Frontend DAPP.
* Replace the <code>NODE_PROVIDER_NAME</code> value with the name of the entity that will provide the nodes. 
* '''''IMPORTANT:''''' Please make sure that you also update the <code>--summary</code> and include a link to the forum discussion, your company's web page, and/or to another place that can convince the voting community that you are making a legitimate request. This way you will avoid the community voting NO to your proposal and you losing the staked ICPs.

# Create the Proposal <syntaxhighlight lang="shell">
NODE_PROVIDER_NAME="My Company"
NODE_PROVIDER_PRINCIPAL=fharn-5vyi2-4xb4a-64yyi-3jpmj-pga23-mxy25-d5uim-fqcro-eoefh-tae # Ledger Hardware Wallet principal, from the NNS FE dapp https://ic0.app/
NEURON_ID=13419667327548602649 # Coming from the NNS FrontEnd dapp https://nns.ic0.app/
./ic-admin \
--nns-url https://ic0.app \
-s ~/.config/dfx/identity/node-provider-hotkey/identity.pem \
propose-to-add-or-remove-node-provider add \
--proposer $NEURON_ID \
--proposal-title "Register a node provider '${NODE_PROVIDER_NAME}'" \
--summary "Register a node provider '${NODE_PROVIDER_NAME}', in line with the announcement and discussion at https://forum.dfinity.org/t/..." \
--node-provider-pid "$NODE_PROVIDER_PRINCIPAL"
</syntaxhighlight>
# Find the proposal on https://dashboard.internetcomputer.org/governance and '''wait until it is executed before proceeding to next step.'''

== VIII. Ensure that your datacenter is registered in the network ==
# Search for your data center on https://dashboard.internetcomputer.org/centers.
#* If you found the datacenter that is hosting your nodes, remember its ID, and skip the following section. Otherwise, proceed with the registration of a new DC. [[File:dc_id.png|1024px]]

=== Create a data center record for a new DC ===
In the next block of code:
* Replace the <code>--proposer</code> argument value with your Neuron ID from the NNS Frontend Dapp.
* Replace the JSON fields from <code>–data-centers-to-add</code> argument and their corresponding values in <code>--summary</code> with: <code>"id"</code>
* The ID should be combination of two letters representing a city that your datacenter is in, and an incrementing number. Search data center IDs on https://dashboard.internetcomputer.org, and find a combination of two letters and a number that’s not yet registered. Examples:
** dl1 (Dallas, no IDs with “dl” prefix)
** zh10 (Zurich, numbers 0-9 are already registered)
[[File:dc_id.png|1024px]]

* <code>"region"</code> represents the local region of a datacenter and is formulated as a three-part string divided by commas. The three parts making the string are continent, country code, and region, in the given order. Examples:
** North America,US,Florida
** Europe,DE,Bavaria
** Asia,SG,Singapore
[[File:datacenter_region.png|1024px]]

* <code>"owner"</code> The entity that provides your datacenter facilities.
** Search https://dashboard.internetcomputer.org for existing data center providers.
** If there’s match, make sure you use the same exact some name for your datacenter.
** Otherwise, name the data center owner to your best knowledge. [[File:datacenter_owner.png|1024px]]

* <code>"gps"</code> GPS coordinates.
** Find your datacenter on https://www.google.com/maps/.
** Right click on location, and select the GPS coordinates (first item in the menu) in order to copy them.
[[File:maps.png|480px|alt=Getting GPS coordinates|Getting GPS coordinates]]

# Create the proposal: <syntaxhighlight lang="shell">
NEURON_ID=13419667327548602649 # Coming from the NNS FE dapp https://nns.ic0.app/
$ ./ic-admin \
--nns-url https://ic0.app \
-s ~/.config/dfx/identity/node-provider-hotkey/identity.pem \
propose-to-add-or-remove-data-centers \
--summary "Register a Flexential datacenter as dl1 in North America,US,Texas" \
--skip-confirmation \
--proposer $NEURON_ID \
--data-centers-to-add '{
"id": "dl1",
"region": "North America,US,Texas",
"owner": "Flexential",
"gps": [
33.00803, -96.66614
]
}'
</syntaxhighlight>
# Find the proposal on https://dashboard.internetcomputer.org/governance and wait until it's executed before proceeding to next step.

== IX. Create a node operator record ==
In the next codeblock:
* Replace the <code>NEURON_ID</code> variable value with your neuron ID obtained from the NNS frontend dapp.
* Replace the <code>NODE_PROVIDER_PRINCIPAL</code> variable value with the Ledger Hardware Wallet principal obtained from the NNS frontend dapp.
* Replace the <code>DC_ID</code> variable value with id of your datacenter.
* Replace the <code>NODE_ALLOWANCE</code> variable value with number of nodes you are providing.

# Create the proposal: <syntaxhighlight lang="shell">
NEURON_ID=13419667327548602649 # Coming from the NNS FE dapp https://nns.ic0.app/
NODE_PROVIDER_PRINCIPAL=fharn-5vyi2-4xb4a-64yyi-3jpmj-pga23-mxy25-d5uim-fqcro-eoefh-tae # Ledger Hardware Wallet principal, from the NNS FE dapp https://nns.ic0.app/
NODE_OPERATOR_PRINCIPAL=$(DFX_HSM_PIN=358138 dfx --identity node-operator-hsm identity get-principal)
NODE_PROVIDER_NAME="My Company"
NODE_ALLOWANCE=28
DC_ID=dl1

./ic-admin \
--nns-url https://ic0.app \
-s ~/.config/dfx/identity/node-provider-hotkey/identity.pem \
propose-to-add-node-operator \
$NODE_PROVIDER_PRINCIPAL \
--summary "Node provider '$NODE_PROVIDER_NAME' is adding $NODE_ALLOWANCE nodes in the $DC_ID data center" \
--proposer $NEURON_ID \
--node-operator-principal-id $NODE_OPERATOR_PRINCIPAL \
--node-allowance $NODE_ALLOWANCE \
--dc-id $DC_ID
</syntaxhighlight>

# Find the proposal on https://dashboard.internetcomputer.org/governance and wait until it's executed before proceeding to next step.

== X. Onboard nodes ==

# Follow the instructions to onboard new nodes:
#* Gen2 - For NP's onboarding in 2023 and later
#** [[IC OS Installation Runbook]]
#* Gen1 - For NP's participating in the IC before 2023
#**[[IC OS Installation Runbook - PowerEdge R6525]]
#**[[IC OS Installation Runbook - Supermicro]]
# Verify that all the nodes were successfully onboarded by checking their status on the dashboard is set to either “Up” or “Unassigned”, or by checking the output from <code>ic-admin get-topology</code> command.
#* The internal dashboard can be searched by your provider principal.
[[File:onboarded_nodes.png|1024px|onboarded nodes]]

== See Also ==
* '''The Internet Computer project website (hosted on the IC): [https://internetcomputer.org/ internetcomputer.org]'''

Node Deployment Guide

2023-05-15T22:18:55Z

Gary.mcelroy: /* 1. Download installation image */ Point at a single download page.

This runbook covers all steps necessary to install the Internet Computer Operating System (IC-OS) on [[Node_provider_hardware#Gen_2|Gen-2 hardware]]. Gen-1 hardware Node Providers should use the [[Internet Computer wiki#For Node Providers|Gen-1 runbooks]].

The physical machine is expected to be racked and stacked according to the respective manual.

In case you encounter any issues during the installation process, check the [[Possible Node Onboarding Errors]] page. Otherwise contact the [https://support.internetcomputer.org/hc/en-us Internet Computer Association]

Many thanks for your efforts in building the Internet Computer.

== Requirements ==
* A USB (3.0 speed that can hold at least 4GB) to put the image file on (Faster USBs will allow the process to go much faster.)
* The Nitrokey HSM for your data center.
* [Optional] A USB hub
** This is helpful at some data centers for simultaneously connecting keyboard, mouse, Nitrokey, etc..

== 1. Download installation image ==
Follow the instructions on the [[IC OS Download|IC-OS Download Page]]

== 2. Verify checksum and unarchive file ==
=== Mac OS X ===
* Open the Terminal and type: <syntaxhighlight lang="shell">shasum -a 256 ~/Downloads/disk-img.tar.gz</syntaxhighlight>
* Compare the calculated checksum with the file downloaded in the previous step. '''Warning:''' Only continue if they are identical, otherwise please contact the [https://support.internetcomputer.org/hc/en-us Internet Computer Association].
* Open the Terminal and type: <syntaxhighlight lang="shell">tar xzvf ~/Downloads/disk-img.tar.gz</syntaxhighlight>

=== Linux / Ubuntu ===
* Open the Terminal and type: <syntaxhighlight lang="shell">sha256sum ~/Downloads/disk-img.tar.gz</syntaxhighlight>
* Compare the calculated checksum with the file downloaded in the previous step. '''Warning:''' Only continue if they are identical, otherwise please contact the [https://support.internetcomputer.org/hc/en-us Internet Computer Association].
* Open the Terminal and type: <syntaxhighlight lang="shell">tar xzvf ~/Downloads/disk-img.tar.gz</syntaxhighlight>

=== Windows ===
* Open PowerShell and type: <syntaxhighlight lang="shell">Get-FileHash -Algorithm SHA256 .\Downloads\disk-img.tar.gz</syntaxhighlight>
* Compare the calculated checksum with the file downloaded in the previous step. '''Warning:''' Only continue if they are identical, otherwise please contact the [https://support.internetcomputer.org/hc/en-us Internet Computer Association].
* Open PowerShell and type: <syntaxhighlight lang="shell">tar xzvf .\Downloads\disk-img.tar.gz</syntaxhighlight>

== 3. Create Bootable USB Stick ==
=== Mac OS X ===
# Open the Terminal and type:
#: <syntaxhighlight lang="shell">diskutil list</syntaxhighlight>
# All available drives should be shown. Identify which device corresponds to your USB stick. You may need to unmount the USB drive:
#:<syntaxhighlight lang="shell">sudo diskutil unmount /dev/YOUR_USB_DEVICE_MOUNTED_PARTITION # E.g. /dev/disk4s1</syntaxhighlight>
# The file path is an example. Use the absolute path to the downloaded image. '''Warning:''' You risk losing your own data if you specify a wrong device.
#:<syntaxhighlight lang="shell">sudo dd if=/Users/YOUR_USER_NAME/Downloads/disk.img of=/dev/YOUR_USB_DEVICE bs=1M</syntaxhighlight>

=== Linux / Ubuntu ===
# Open the Terminal and type <syntaxhighlight lang="shell">blkid</syntaxhighlight>
# All available drives should be shown. Identify which device corresponds to your USB stick. You may need to unmount the USB drive:
#:<syntaxhighlight lang="shell">sudo diskutil unmount /dev/YOUR_USB_DEVICE_MOUNTED_PARTITION # E.g. /dev/sdb1</syntaxhighlight>
# Replace ''/dev/YOUR_USB_DEVICE'' with the device that corresponds to your USB stick. '''Warning:''' You risk losing your own data if you specify a wrong drive.
#: <syntaxhighlight lang="shell">sudo dd if=~/Downloads/disk.img of=/dev/YOUR_USB_DEVICE bs=1M</syntaxhighlight>

=== Windows ===
# Download and install [https://rufus.ie/en/ Rufus Portable]
# Start Rufus
# Select the USB stick under device and select the previously downloaded IC-OS disk image and press start
#: [[File:05.png|480px|screenshot]]
# You may see some warnings. Make sure you don't have any other USBs in your computer and chose OK
#: [[File:06.png|480px|screenshot]]
#: [[File:07.png|480px|screenshot]]
# The "Ready" bar will go from left to right as it completes.

== 4. Add configuration ==
=== Mac OS X ===
# Open Finder. You should now be able to see the CONFIG partition. If it's not visible, remove the USB and insert it again.
#: [[File:mac_01.png|580px|screenshot]]
# Double-click to open it in TextEdit.
# Insert your IPv6 prefix, subnet and gateway.
#: [[File:mac_02.png|580px|screenshot]]
# Once done, don’t forget to save the changes. If you need help, please do not hesitate to contact the Internet Computer Association.
#: [[File:mac_03.png|580px|screenshot]]

=== Windows ===
# Open the Disk Management utility with a right click on the Start menu
#: [[File:09-b.png|300px|screenshot]]#:
# Right click the CONFIG partition
# Select Change drive letter or paths...
#: [[File:10-b.png|780px|screenshot]]
# Select any letter from the drop-down list
#: [[File:11-b.png|480px|screenshot]]
# Click OK.
# You should now be able to see the CONFIG partition in your Windows Explorer. Select the config.ini configuration file
#: [[File:12-b.png|780px|screenshot]]
# Click on Edit to open it.
# Insert your IPv6 prefix, subnet and gateway.
#: [[File:13-b.png|580px|screenshot]]
# Once done, don’t forget to save the changes. If you need help, please do not hesitate to contact the Internet Computer Association.
#: [[File:14-b.png|580px|screenshot]]

=== Linux ===
# Open the File Manager. You should now be able to see the CONFIG partition. If it's not visible, remove the USB and insert it again.
#: [[File:linux_01.png|580px|screenshot]]
# Double-click to open it in KWrite.
# Insert your IPv6 prefix, subnet and gateway.
#: [[File:linux_02.png|580px|screenshot]]
# Once done, don’t forget to save the changes. If you need help, please do not hesitate to contact the Internet Computer Association.
#: [[File:linux_03.png|580px|screenshot]]

== 5. Connect Crash Cart ==
# In order to configure the UEFI and initiate the installation of the IC-OS, please connect a crash cart to the physical machine.
# Plug-in the VGA/Video, keyboard and IC-OS USB stick
#: [[File:08.png|580px|screenshot]]

== 6. UEFI Setup and Boot Menu ==

Use the related page below to set up the BIOS/UEFI according to your hardware vendor.

* [[IC OS Installation - UEFI Configuration - Dell]]
* [[IC OS Installation - UEFI Configuration - Supermicro]]


Resume from this point when you are finished.

== 7. IC-OS Installation ==
# Please wait while the USB Installer is booting up. This process can take up to 3 minutes.
#: [[File:35-sm.png|580px|screenshot]]
# The IC-OS installation starts. Please keep an eye on the progress. This part can take up to 8 minutes. Please remember to check the [[Possible Node Onboarding Errors]] page if you encounter any errors.
#: [[File:36-sm.png|580px|screenshot]]
# Once you get asked to insert the HSM, please remove the keyboard and instead insert the HSM USB device.
#: [[File:37-sm.png|580px|screenshot]]
# If the installation finished successfully, it will initiate a reboot. '''Please do not unplug the USB stick or HSM USB''' device at this point.
#: [[File:38-sm.png|580px|screenshot]]

== 8. First Boot ==
Please remember to check the [[Possible Node Onboarding Errors]] page if you encounter any errors onboarding.
'''Do NOT re-try the onboarding after proceeding to this section, as this can cause duplication within the registry.'''
# The first boot of the IC-OS still requires the HSM USB device. Please wait until further instructions. This step can take up to 2 minutes.
#: [[File:39-sm.png|580px|screenshot]]
# Once you see this message, you may unplug the HSM USB device, USB stick and VGA/Video. Your machine successfully joined the Internet Computer.
#: [[File:40-sm.png|580px|screenshot]]

[[Node Provider Onboarding|Return to the Onboarding Document]]

Node Deployment Guide

2023-05-15T22:13:30Z

Gary.mcelroy: /* 1. Download installation image */ Updated to latest release

This runbook covers all steps necessary to install the Internet Computer Operating System (IC-OS) on [[Node_provider_hardware#Gen_2|Gen-2 hardware]]. Gen-1 hardware Node Providers should use the [[Internet Computer wiki#For Node Providers|Gen-1 runbooks]].

The physical machine is expected to be racked and stacked according to the respective manual.

In case you encounter any issues during the installation process, check the [[Possible Node Onboarding Errors]] page. Otherwise contact the [https://support.internetcomputer.org/hc/en-us Internet Computer Association]

Many thanks for your efforts in building the Internet Computer.

== Requirements ==
* A USB (3.0 speed that can hold at least 4GB) to put the image file on (Faster USBs will allow the process to go much faster.)
* The Nitrokey HSM for your data center.
* [Optional] A USB hub
** This is helpful at some data centers for simultaneously connecting keyboard, mouse, Nitrokey, etc..

== 1. Download installation image ==

# Download the IC-OS USB installer image. '''This link will always provide the current blessed version. Do NOT use an old USB image''': https://download.dfinity.systems/ic/d6d395a480cd6986b4788f4aafffc5c03a07e46e/setup-os/disk-img/disk-img.tar.gz
# Download the corresponding checksum: https://download.dfinity.systems/ic/d6d395a480cd6986b4788f4aafffc5c03a07e46e/setup-os/disk-img/SHA256SUMS

== 2. Verify checksum and unarchive file ==
=== Mac OS X ===
* Open the Terminal and type: <syntaxhighlight lang="shell">shasum -a 256 ~/Downloads/disk-img.tar.gz</syntaxhighlight>
* Compare the calculated checksum with the file downloaded in the previous step. '''Warning:''' Only continue if they are identical, otherwise please contact the [https://support.internetcomputer.org/hc/en-us Internet Computer Association].
* Open the Terminal and type: <syntaxhighlight lang="shell">tar xzvf ~/Downloads/disk-img.tar.gz</syntaxhighlight>

=== Linux / Ubuntu ===
* Open the Terminal and type: <syntaxhighlight lang="shell">sha256sum ~/Downloads/disk-img.tar.gz</syntaxhighlight>
* Compare the calculated checksum with the file downloaded in the previous step. '''Warning:''' Only continue if they are identical, otherwise please contact the [https://support.internetcomputer.org/hc/en-us Internet Computer Association].
* Open the Terminal and type: <syntaxhighlight lang="shell">tar xzvf ~/Downloads/disk-img.tar.gz</syntaxhighlight>

=== Windows ===
* Open PowerShell and type: <syntaxhighlight lang="shell">Get-FileHash -Algorithm SHA256 .\Downloads\disk-img.tar.gz</syntaxhighlight>
* Compare the calculated checksum with the file downloaded in the previous step. '''Warning:''' Only continue if they are identical, otherwise please contact the [https://support.internetcomputer.org/hc/en-us Internet Computer Association].
* Open PowerShell and type: <syntaxhighlight lang="shell">tar xzvf .\Downloads\disk-img.tar.gz</syntaxhighlight>

== 3. Create Bootable USB Stick ==
=== Mac OS X ===
# Open the Terminal and type:
#: <syntaxhighlight lang="shell">diskutil list</syntaxhighlight>
# All available drives should be shown. Identify which device corresponds to your USB stick. You may need to unmount the USB drive:
#:<syntaxhighlight lang="shell">sudo diskutil unmount /dev/YOUR_USB_DEVICE_MOUNTED_PARTITION # E.g. /dev/disk4s1</syntaxhighlight>
# The file path is an example. Use the absolute path to the downloaded image. '''Warning:''' You risk losing your own data if you specify a wrong device.
#:<syntaxhighlight lang="shell">sudo dd if=/Users/YOUR_USER_NAME/Downloads/disk.img of=/dev/YOUR_USB_DEVICE bs=1M</syntaxhighlight>

=== Linux / Ubuntu ===
# Open the Terminal and type <syntaxhighlight lang="shell">blkid</syntaxhighlight>
# All available drives should be shown. Identify which device corresponds to your USB stick. You may need to unmount the USB drive:
#:<syntaxhighlight lang="shell">sudo diskutil unmount /dev/YOUR_USB_DEVICE_MOUNTED_PARTITION # E.g. /dev/sdb1</syntaxhighlight>
# Replace ''/dev/YOUR_USB_DEVICE'' with the device that corresponds to your USB stick. '''Warning:''' You risk losing your own data if you specify a wrong drive.
#: <syntaxhighlight lang="shell">sudo dd if=~/Downloads/disk.img of=/dev/YOUR_USB_DEVICE bs=1M</syntaxhighlight>

=== Windows ===
# Download and install [https://rufus.ie/en/ Rufus Portable]
# Start Rufus
# Select the USB stick under device and select the previously downloaded IC-OS disk image and press start
#: [[File:05.png|480px|screenshot]]
# You may see some warnings. Make sure you don't have any other USBs in your computer and chose OK
#: [[File:06.png|480px|screenshot]]
#: [[File:07.png|480px|screenshot]]
# The "Ready" bar will go from left to right as it completes.

== 4. Add configuration ==
=== Mac OS X ===
# Open Finder. You should now be able to see the CONFIG partition. If it's not visible, remove the USB and insert it again.
#: [[File:mac_01.png|580px|screenshot]]
# Double-click to open it in TextEdit.
# Insert your IPv6 prefix, subnet and gateway.
#: [[File:mac_02.png|580px|screenshot]]
# Once done, don’t forget to save the changes. If you need help, please do not hesitate to contact the Internet Computer Association.
#: [[File:mac_03.png|580px|screenshot]]

=== Windows ===
# Open the Disk Management utility with a right click on the Start menu
#: [[File:09-b.png|300px|screenshot]]#:
# Right click the CONFIG partition
# Select Change drive letter or paths...
#: [[File:10-b.png|780px|screenshot]]
# Select any letter from the drop-down list
#: [[File:11-b.png|480px|screenshot]]
# Click OK.
# You should now be able to see the CONFIG partition in your Windows Explorer. Select the config.ini configuration file
#: [[File:12-b.png|780px|screenshot]]
# Click on Edit to open it.
# Insert your IPv6 prefix, subnet and gateway.
#: [[File:13-b.png|580px|screenshot]]
# Once done, don’t forget to save the changes. If you need help, please do not hesitate to contact the Internet Computer Association.
#: [[File:14-b.png|580px|screenshot]]

=== Linux ===
# Open the File Manager. You should now be able to see the CONFIG partition. If it's not visible, remove the USB and insert it again.
#: [[File:linux_01.png|580px|screenshot]]
# Double-click to open it in KWrite.
# Insert your IPv6 prefix, subnet and gateway.
#: [[File:linux_02.png|580px|screenshot]]
# Once done, don’t forget to save the changes. If you need help, please do not hesitate to contact the Internet Computer Association.
#: [[File:linux_03.png|580px|screenshot]]

== 5. Connect Crash Cart ==
# In order to configure the UEFI and initiate the installation of the IC-OS, please connect a crash cart to the physical machine.
# Plug-in the VGA/Video, keyboard and IC-OS USB stick
#: [[File:08.png|580px|screenshot]]

== 6. UEFI Setup and Boot Menu ==

Use the related page below to set up the BIOS/UEFI according to your hardware vendor.

* [[IC OS Installation - UEFI Configuration - Dell]]
* [[IC OS Installation - UEFI Configuration - Supermicro]]


Resume from this point when you are finished.

== 7. IC-OS Installation ==
# Please wait while the USB Installer is booting up. This process can take up to 3 minutes.
#: [[File:35-sm.png|580px|screenshot]]
# The IC-OS installation starts. Please keep an eye on the progress. This part can take up to 8 minutes. Please remember to check the [[Possible Node Onboarding Errors]] page if you encounter any errors.
#: [[File:36-sm.png|580px|screenshot]]
# Once you get asked to insert the HSM, please remove the keyboard and instead insert the HSM USB device.
#: [[File:37-sm.png|580px|screenshot]]
# If the installation finished successfully, it will initiate a reboot. '''Please do not unplug the USB stick or HSM USB''' device at this point.
#: [[File:38-sm.png|580px|screenshot]]

== 8. First Boot ==
Please remember to check the [[Possible Node Onboarding Errors]] page if you encounter any errors onboarding.
'''Do NOT re-try the onboarding after proceeding to this section, as this can cause duplication within the registry.'''
# The first boot of the IC-OS still requires the HSM USB device. Please wait until further instructions. This step can take up to 2 minutes.
#: [[File:39-sm.png|580px|screenshot]]
# Once you see this message, you may unplug the HSM USB device, USB stick and VGA/Video. Your machine successfully joined the Internet Computer.
#: [[File:40-sm.png|580px|screenshot]]

[[Node Provider Onboarding|Return to the Onboarding Document]]

Node Provider Onboarding

2023-05-11T22:11:48Z

Gary.mcelroy: /* Requirements */ Added educational links

Learn how to participate in the Internet Computer network as a Node Provider and to receive rewards for supporting the network.

== Requirements ==

* [https://wiki.internetcomputer.org/wiki/Node_provider_hardware Node Hardware]
* Rack space with a 10Gb connectivity, RJ45 terminated on the nodes
* [[Node Provider Network Setup Guide#The Bare Minimum Network Requirements|Public IP addresses]]:
** One /64 IPv6 range
** One IPv4 address for every 4 node machines
* [https://www.ledger.com/ Hardware wallet]
* [https://shop.nitrokey.com/shop/product/nkhs2-nitrokey-hsm-2-7/ NitroKey HSM]
* 11 ICP (10 of which are to be staked for the NNS proposal deposit)
* Basic understanding of [[Neurons 101|neurons]], [https://internetcomputer.org/docs/current/tokenomics/nns/nns-staking-voting-rewards staking], and [[Governance of the Internet Computer|governance]] proposals. For instance, understanding what it means to stake a neuron for 8 years.
* The technical knowledge to understand some minor steps that are not explicitly mentioned in these instructions. For instance, when to insert an HSM.

'''Note:''' Please allocate at least 0.5 day for going through the first part, i.e., the registration of a new NP. It may even take a couple of days, depending on how quickly the community votes for the proposals. There is a also fair amount of complexity and the technical knowledge that needs to be absorbed in order to complete the steps. But this only needs to be done once. 
The next step, going to the DC and bringing up and onboarding the machines, is much quicker. Estimate to spend 10-15 minutes per machine. This time should go down to ~5 minutes as you gain experience. Also, multiple machines can be brought up in parallel.

== I. Install the required tools ==
===''' A. Install ic-admin '''===

<code>ic-admin</code> is the tool used to create and submit NNS proposals.

==== MacOS ====
# Retrieve the file <syntaxhighlight lang="shell">
curl "https://download.dfinity.systems/ic/7445081734e6d896d090295967d50710975c4f25/openssl-static-binaries/x86_64-darwin/ic-admin.gz" -o - | gunzip > ./ic-admin
chmod +x ./ic-admin
</syntaxhighlight>
# Verify the binary <syntaxhighlight lang="shell">
diff <(shasum -a 256 ./ic-admin | cut -d' ' -f1) <(echo 3f75026d2f28f171068e332a42c82a2795c93fbf5ab351baef30b30eb901cdba) && echo "ic-admin checksum matches" || echo "***ERROR***: ic-admin checksum does not match"
</syntaxhighlight>

==== Linux ====
NOTE: The instructions below have been tested with the Ubuntu 20.04 release.
# Retrieve the file <syntaxhighlight lang="shell">
curl "https://download.dfinity.systems/ic/7445081734e6d896d090295967d50710975c4f25/openssl-static-binaries/x86_64-linux/ic-admin.gz" -o - | gunzip > ./ic-admin
chmod +x ./ic-admin
</syntaxhighlight>
# Verify the binary <syntaxhighlight lang="shell">
diff <(shasum -a 256 ./ic-admin | cut -d' ' -f1) <(echo e29bb9cc462e800b8b960ad49c412e5f5fdbb5ae2ae9fde0c13058422ba32802) && echo "ic-admin checksum matches" || echo "***ERROR***: ic-admin checksum does not match"
</syntaxhighlight>

===''' B. Install dfx '''===

# <code>dfx</code> allows generating a neuron hotkey, among other things <syntaxhighlight lang="shell">
$ DFX_VERSION=0.9.3 sh -ci "$(curl -fsSL https://sdk.dfinity.org/install.sh)"
</syntaxhighlight>
# Verify that the version is 0.9.3 <syntaxhighlight lang="shell">
$ export PATH=$HOME/bin:$PATH
$ dfx --version dfx 0.9.3
</syntaxhighlight>
# Create an identity for the Node Provider '''Hotkey''' <syntaxhighlight lang="shell">
$ dfx identity new node-provider-hotkey
Creating identity: "node-provider-hotkey".
Created identity: "node-provider-hotkey".
$ dfx --identity node-provider-hotkey identity get-principal
wuyst-x5tpn-g5wri-mp3ps-vjtba-de3xs-w5xgb-crvek-tucbe-o5rqi-mae
</syntaxhighlight>
# '''Note:''' The node provider hotkey is NOT the node provider principal. This is the hotkey that is used for the NNS proposal submissions only.

== II. Create and Manage Neuron via NNS Frontend Dapp and Internet Identity ==

# Setup your hardware wallet: https://medium.com/dfinity/integrating-ledger-nano-with-the-nns-front-end-dapp-user-manual-9c5600925e16
# Send at least 11 ICPs to the hardware wallet address.
# Navigate to Neurons tab and create a Neuron by staking at least 10 ICP from your hardware wallet. Staking more ICP works as well, but 10 is the minimum.
# IMPORTANT! Confirm the transaction on your hardware wallet.
#: [[File:-docs-stake_neuron_1.png|1024px|stake neuron]]
#:
# After the Neuron has been created successfully, confirm to add NNS Dapp as hotkey in the dialogue and on your hardware wallet, and close the dialog after the action completes.
#: [[File:-docs-stake_neuron_2.png|1024px|neuron id]]
# Set the dissolve delay to at least 6 months, and confirm the choice in the dialogue and on your hardware wallet. After the action completes, you can close the "Follow Neurons".
#:
#: [[File:dissolve_delay.png|480px|neuron id]]
# You will now see a Neuron listed with its ID. Copy the Neuron ID, since you will need it in the next steps to place the necessary proposals.
#: [[File:Neuron id.png|1024px]]

== III. Add hotkeys ==

# Select the Neuron you just created to open Neuron management view and press “Add hotkey” button.
#: [[File:Hotkey 1.png|800px]]
# A dialog will pop up where you can enter the principal you generated in step 2 (output from command <code>dfx --identity node-provider-hotkey identity get-principal</code>). This will allow you to submit NNS proposals using <code>ic-admin</code> and will not be used for anything else. 
#:Press the confirm button and confirm the transactions on your hardware wallet. 
#: [[File:Hotkey 2.png|800px]]
#:
# Get the Ledger Hardware Wallet Principal Id: Navigate back to ICP page and select your Ledger hardware wallet account. You will need to use this Ledger Hardware Wallet principal as the Node Provider principal in order to get the rewards directly into the secure hardware wallet.
[[File:Node provider principal 1.png|1024px]]
[[File:Node provider principal 2.png|800px]]
# Copy and save this Node Provider principal by clicking on the copy icon after the principal id. You'll need it in the next steps.

<syntaxhighlight lang="shell">
NODE_PROVIDER_PRINCIPAL=fharn-5vyi2-4xb4a-64yyi-3jpmj-pga23-mxy25-d5uim-fqcro-eoefh-tae # Ledger Hardware Wallet principal, from the NNS FrontEnd dapp https://nns.ic0.app/
</syntaxhighlight>

== IV. Configure your HSM ==
It's first necessary to install the necessary tools.

=== MacOS ===
# Download this OpenSC binary: https://github.com/OpenSC/OpenSC/releases/download/0.22.0/OpenSC-0.22.0.dmg
# Double click the DMG image that you downloaded and then double click the OpenSC PKG file.
# If your system doesn't allow the installation software from an unidentified developer please follow these steps or contact your system administrator:
#* Choose the Apple menu > System Preferences > click Security and Privacy.
#* Click the lock Icon to unlock it, then enter an administrator name and password.
#* Ensure that you're on the tab named “General”.
#* You should see the OpenSC app and you should be able to enable its installation by choosing “Open anyway”.
# Click continue and install until the installation is complete.

=== Linux ===

NOTE: The instructions below have been tested with the Ubuntu 20.04 release.

# Install pcscd and opensc
#: <syntaxhighlight lang="shell">
sudo add-apt-repository universe
sudo apt update
sudo apt install pcscd opensc
</syntaxhighlight>

== V. Setup the HSM ==

# Initialize the HSM. <syntaxhighlight lang="shell">
sc-hsm-tool --initialize --so-pin 3537363231383830 --pin 358138
</syntaxhighlight>
# Change the HSM so-pin.
#* '''WARNING:''' The new HSM so pin must have 16 hexadecimal digits. This is not very well known, and some HSM users have lost access to a Nitrokey HSM because they tried using regular characters and the command below accepted it.
#* '''Do NOT change the user pin. It must remain as the default for the onboarding scripts to work'''<syntaxhighlight lang="shell">
pkcs11-tool --login --login-type so --so-pin 3537363231383830 --change-pin
</syntaxhighlight>
# Create a keypair on the HSM. Enter the default pin 358138 when prompted.
#* '''Note:''' Before initializing the HSM key please refer to the [https://docs.nitrokey.com/pro/openpgp.html Nitrokey HSM documentation] if you wish to create a backup. Creating a backup of the HSM device is NOT possible after the key has already been created. <syntaxhighlight lang="shell">
pkcs11-tool -k --key-type EC:prime256v1 --login -d 01
</syntaxhighlight>

== VI. Get the node operator principal from the HSM ==
# Configure dfx identity (skip this step if you already configured it for an other HSM).
#* '''Note:''' Depending on your installation, the path to the <code>--hsm-pkcs11-lib-path</code> might be different on your platform. You can locate the correct path with the following command: <syntaxhighlight lang="shell">
find / -name opensc-pkcs11.so 2> /dev/null
</syntaxhighlight>
#* MacOS <syntaxhighlight lang="shell">
dfx identity new node-operator-hsm --hsm-key-id 01 --hsm-pkcs11-lib-path /Library/OpenSC/lib/opensc-pkcs11.so
</syntaxhighlight>
#* Linux <syntaxhighlight lang="shell">
dfx identity new node-operator-hsm --hsm-key-id 01 --hsm-pkcs11-lib-path /usr/lib/x86_64-linux-gnu/opensc-pkcs11.so
</syntaxhighlight>
# Get the principal. <syntaxhighlight lang="shell">
$ NODE_OPERATOR_PRINCIPAL=$(DFX_HSM_PIN=358138 dfx --identity node-operator-hsm identity get-principal)
$ echo $NODE_OPERATOR_PRINCIPAL

uqquy-76uhn-2mys5-xxxxx-xxxxx-xxxxx-xxxxx-xxxxx-xxxxx-xxxxx-xxx
</syntaxhighlight>

== VII. Register your NP principal to the network ==

In the next codeblock:
* Replace the <code>NEURON_ID</code> value with your neuron ID from the NNS Frontend Dapp
* Replace the <code>NODE_PROVIDER_PRINCIPAL</code> value with the Ledger Hardware Wallet principal that you got from the NNS Frontend DAPP.
* Replace the <code>NODE_PROVIDER_NAME</code> value with the name of the entity that will provide the nodes. 
* '''''IMPORTANT:''''' Please make sure that you also update the <code>--summary</code> and include a link to the forum discussion, your company's web page, and/or to another place that can convince the voting community that you are making a legitimate request. This way you will avoid the community voting NO to your proposal and you losing the staked ICPs.

# Create the Proposal <syntaxhighlight lang="shell">
NODE_PROVIDER_NAME="My Company"
NODE_PROVIDER_PRINCIPAL=fharn-5vyi2-4xb4a-64yyi-3jpmj-pga23-mxy25-d5uim-fqcro-eoefh-tae # Ledger Hardware Wallet principal, from the NNS FE dapp https://ic0.app/
NEURON_ID=13419667327548602649 # Coming from the NNS FrontEnd dapp https://nns.ic0.app/
./ic-admin \
--nns-url https://ic0.app \
-s ~/.config/dfx/identity/node-provider-hotkey/identity.pem \
propose-to-add-or-remove-node-provider add \
--proposer $NEURON_ID \
--proposal-title "Register a node provider '${NODE_PROVIDER_NAME}'" \
--summary "Register a node provider '${NODE_PROVIDER_NAME}', in line with the announcement and discussion at https://forum.dfinity.org/t/..." \
--node-provider-pid "$NODE_PROVIDER_PRINCIPAL"
</syntaxhighlight>
# Find the proposal on https://dashboard.internetcomputer.org/governance and '''wait until it is executed before proceeding to next step.'''

== VIII. Ensure that your datacenter is registered in the network ==
# Search for your data center on https://dashboard.internetcomputer.org/centers.
#* If you found the datacenter that is hosting your nodes, remember its ID, and skip the following section. Otherwise, proceed with the registration of a new DC. [[File:dc_id.png|1024px]]

=== Create a data center record for a new DC ===
In the next block of code:
* Replace the <code>--proposer</code> argument value with your Neuron ID from the NNS Frontend Dapp.
* Replace the JSON fields from <code>–data-centers-to-add</code> argument and their corresponding values in <code>--summary</code> with: <code>"id"</code>
* The ID should be combination of two letters representing a city that your datacenter is in, and an incrementing number. Search data center IDs on https://dashboard.internetcomputer.org, and find a combination of two letters and a number that’s not yet registered. Examples:
** dl1 (Dallas, no IDs with “dl” prefix)
** zh10 (Zurich, numbers 0-9 are already registered)
[[File:dc_id.png|1024px]]

* <code>"region"</code> represents the local region of a datacenter and is formulated as a three-part string divided by commas. The three parts making the string are continent, country code, and region, in the given order. Examples:
** North America,US,Florida
** Europe,DE,Bavaria
** Asia,SG,Singapore
[[File:datacenter_region.png|1024px]]

* <code>"owner"</code> The entity that provides your datacenter facilities.
** Search https://dashboard.internetcomputer.org for existing data center providers.
** If there’s match, make sure you use the same exact some name for your datacenter.
** Otherwise, name the data center owner to your best knowledge. [[File:datacenter_owner.png|1024px]]

* <code>"gps"</code> GPS coordinates.
** Find your datacenter on https://www.google.com/maps/.
** Right click on location, and select the GPS coordinates (first item in the menu) in order to copy them.
[[File:maps.png|480px|alt=Getting GPS coordinates|Getting GPS coordinates]]

# Create the proposal: <syntaxhighlight lang="shell">
NEURON_ID=13419667327548602649 # Coming from the NNS FE dapp https://nns.ic0.app/
$ ./ic-admin \
--nns-url https://ic0.app \
-s ~/.config/dfx/identity/node-provider-hotkey/identity.pem \
propose-to-add-or-remove-data-centers \
--summary "Register a Flexential datacenter as dl1 in North America,US,Texas" \
--skip-confirmation \
--proposer $NEURON_ID \
--data-centers-to-add '{
"id": "dl1",
"region": "North America,US,Texas",
"owner": "Flexential",
"gps": [
33.00803, -96.66614
]
}'
</syntaxhighlight>
# Find the proposal on https://dashboard.internetcomputer.org/governance and wait until it's executed before proceeding to next step.

== IX. Create a node operator record ==
In the next codeblock:
* Replace the <code>NEURON_ID</code> variable value with your neuron ID obtained from the NNS frontend dapp.
* Replace the <code>NODE_PROVIDER_PRINCIPAL</code> variable value with the Ledger Hardware Wallet principal obtained from the NNS frontend dapp.
* Replace the <code>DC_ID</code> variable value with id of your datacenter.
* Replace the <code>NODE_ALLOWANCE</code> variable value with number of nodes you are providing.

# Create the proposal: <syntaxhighlight lang="shell">
NEURON_ID=13419667327548602649 # Coming from the NNS FE dapp https://nns.ic0.app/
NODE_PROVIDER_PRINCIPAL=fharn-5vyi2-4xb4a-64yyi-3jpmj-pga23-mxy25-d5uim-fqcro-eoefh-tae # Ledger Hardware Wallet principal, from the NNS FE dapp https://nns.ic0.app/
NODE_OPERATOR_PRINCIPAL=$(DFX_HSM_PIN=358138 dfx --identity node-operator-hsm identity get-principal)
NODE_PROVIDER_NAME="My Company"
NODE_ALLOWANCE=28
DC_ID=dl1

./ic-admin \
--nns-url https://ic0.app \
-s ~/.config/dfx/identity/node-provider-hotkey/identity.pem \
propose-to-add-node-operator \
$NODE_PROVIDER_PRINCIPAL \
--summary "Node provider '$NODE_PROVIDER_NAME' is adding $NODE_ALLOWANCE nodes in the $DC_ID data center" \
--proposer $NEURON_ID \
--node-operator-principal-id $NODE_OPERATOR_PRINCIPAL \
--node-allowance $NODE_ALLOWANCE \
--dc-id $DC_ID
</syntaxhighlight>

# Find the proposal on https://dashboard.internetcomputer.org/governance and wait until it's executed before proceeding to next step.

== X. Onboard nodes ==

# Follow the instructions to onboard new nodes:
#* Gen2 - For NP's onboarding in 2023 and later
#** [[IC OS Installation Runbook]]
#* Gen1 - For NP's participating in the IC before 2023
#**[[IC OS Installation Runbook - PowerEdge R6525]]
#**[[IC OS Installation Runbook - Supermicro]]
# Verify that all the nodes were successfully onboarded by checking their status on the dashboard is set to either “Up” or “Unassigned”, or by checking the output from <code>ic-admin get-topology</code> command.
#* The internal dashboard can be searched by your provider principal.
[[File:onboarded_nodes.png|1024px|onboarded nodes]]

== See Also ==
* '''The Internet Computer project website (hosted on the IC): [https://internetcomputer.org/ internetcomputer.org]'''

Node Provider Onboarding

2023-05-11T22:06:42Z

Gary.mcelroy: /* Requirements */ Detailed IP address req's

Learn how to participate in the Internet Computer network as a Node Provider and to receive rewards for supporting the network.

== Requirements ==

* [https://wiki.internetcomputer.org/wiki/Node_provider_hardware Node Hardware]
* Rack space with a 10Gb connectivity, RJ45 terminated on the nodes
* [[Node Provider Network Setup Guide#The Bare Minimum Network Requirements|Public IP addresses]]:
** One /64 IPv6 range
** One IPv4 address for every 4 node machines
* [https://www.ledger.com/ Hardware wallet]
* [https://shop.nitrokey.com/shop/product/nkhs2-nitrokey-hsm-2-7/ NitroKey HSM]
* 11 ICP (10 of which are to be staked for the NNS proposal deposit)
* Basic understanding of neurons, staking, and governance proposals. For instance, understanding what it means to stake a neuron for 8 years.
* The technical knowledge to understand some minor steps that are not explicitly mentioned in these instructions. For instance, when to insert an HSM.

'''Note:''' Please allocate at least 0.5 day for going through the first part, i.e., the registration of a new NP. It may even take a couple of days, depending on how quickly the community votes for the proposals. There is a also fair amount of complexity and the technical knowledge that needs to be absorbed in order to complete the steps. But this only needs to be done once. 
The next step, going to the DC and bringing up and onboarding the machines, is much quicker. Estimate to spend 10-15 minutes per machine. This time should go down to ~5 minutes as you gain experience. Also, multiple machines can be brought up in parallel.

== I. Install the required tools ==
===''' A. Install ic-admin '''===

<code>ic-admin</code> is the tool used to create and submit NNS proposals.

==== MacOS ====
# Retrieve the file <syntaxhighlight lang="shell">
curl "https://download.dfinity.systems/ic/7445081734e6d896d090295967d50710975c4f25/openssl-static-binaries/x86_64-darwin/ic-admin.gz" -o - | gunzip > ./ic-admin
chmod +x ./ic-admin
</syntaxhighlight>
# Verify the binary <syntaxhighlight lang="shell">
diff <(shasum -a 256 ./ic-admin | cut -d' ' -f1) <(echo 3f75026d2f28f171068e332a42c82a2795c93fbf5ab351baef30b30eb901cdba) && echo "ic-admin checksum matches" || echo "***ERROR***: ic-admin checksum does not match"
</syntaxhighlight>

==== Linux ====
NOTE: The instructions below have been tested with the Ubuntu 20.04 release.
# Retrieve the file <syntaxhighlight lang="shell">
curl "https://download.dfinity.systems/ic/7445081734e6d896d090295967d50710975c4f25/openssl-static-binaries/x86_64-linux/ic-admin.gz" -o - | gunzip > ./ic-admin
chmod +x ./ic-admin
</syntaxhighlight>
# Verify the binary <syntaxhighlight lang="shell">
diff <(shasum -a 256 ./ic-admin | cut -d' ' -f1) <(echo e29bb9cc462e800b8b960ad49c412e5f5fdbb5ae2ae9fde0c13058422ba32802) && echo "ic-admin checksum matches" || echo "***ERROR***: ic-admin checksum does not match"
</syntaxhighlight>

===''' B. Install dfx '''===

# <code>dfx</code> allows generating a neuron hotkey, among other things <syntaxhighlight lang="shell">
$ DFX_VERSION=0.9.3 sh -ci "$(curl -fsSL https://sdk.dfinity.org/install.sh)"
</syntaxhighlight>
# Verify that the version is 0.9.3 <syntaxhighlight lang="shell">
$ export PATH=$HOME/bin:$PATH
$ dfx --version dfx 0.9.3
</syntaxhighlight>
# Create an identity for the Node Provider '''Hotkey''' <syntaxhighlight lang="shell">
$ dfx identity new node-provider-hotkey
Creating identity: "node-provider-hotkey".
Created identity: "node-provider-hotkey".
$ dfx --identity node-provider-hotkey identity get-principal
wuyst-x5tpn-g5wri-mp3ps-vjtba-de3xs-w5xgb-crvek-tucbe-o5rqi-mae
</syntaxhighlight>
# '''Note:''' The node provider hotkey is NOT the node provider principal. This is the hotkey that is used for the NNS proposal submissions only.

== II. Create and Manage Neuron via NNS Frontend Dapp and Internet Identity ==

# Setup your hardware wallet: https://medium.com/dfinity/integrating-ledger-nano-with-the-nns-front-end-dapp-user-manual-9c5600925e16
# Send at least 11 ICPs to the hardware wallet address.
# Navigate to Neurons tab and create a Neuron by staking at least 10 ICP from your hardware wallet. Staking more ICP works as well, but 10 is the minimum.
# IMPORTANT! Confirm the transaction on your hardware wallet.
#: [[File:-docs-stake_neuron_1.png|1024px|stake neuron]]
#:
# After the Neuron has been created successfully, confirm to add NNS Dapp as hotkey in the dialogue and on your hardware wallet, and close the dialog after the action completes.
#: [[File:-docs-stake_neuron_2.png|1024px|neuron id]]
# Set the dissolve delay to at least 6 months, and confirm the choice in the dialogue and on your hardware wallet. After the action completes, you can close the "Follow Neurons".
#:
#: [[File:dissolve_delay.png|480px|neuron id]]
# You will now see a Neuron listed with its ID. Copy the Neuron ID, since you will need it in the next steps to place the necessary proposals.
#: [[File:Neuron id.png|1024px]]

== III. Add hotkeys ==

# Select the Neuron you just created to open Neuron management view and press “Add hotkey” button.
#: [[File:Hotkey 1.png|800px]]
# A dialog will pop up where you can enter the principal you generated in step 2 (output from command <code>dfx --identity node-provider-hotkey identity get-principal</code>). This will allow you to submit NNS proposals using <code>ic-admin</code> and will not be used for anything else. 
#:Press the confirm button and confirm the transactions on your hardware wallet. 
#: [[File:Hotkey 2.png|800px]]
#:
# Get the Ledger Hardware Wallet Principal Id: Navigate back to ICP page and select your Ledger hardware wallet account. You will need to use this Ledger Hardware Wallet principal as the Node Provider principal in order to get the rewards directly into the secure hardware wallet.
[[File:Node provider principal 1.png|1024px]]
[[File:Node provider principal 2.png|800px]]
# Copy and save this Node Provider principal by clicking on the copy icon after the principal id. You'll need it in the next steps.

<syntaxhighlight lang="shell">
NODE_PROVIDER_PRINCIPAL=fharn-5vyi2-4xb4a-64yyi-3jpmj-pga23-mxy25-d5uim-fqcro-eoefh-tae # Ledger Hardware Wallet principal, from the NNS FrontEnd dapp https://nns.ic0.app/
</syntaxhighlight>

== IV. Configure your HSM ==
It's first necessary to install the necessary tools.

=== MacOS ===
# Download this OpenSC binary: https://github.com/OpenSC/OpenSC/releases/download/0.22.0/OpenSC-0.22.0.dmg
# Double click the DMG image that you downloaded and then double click the OpenSC PKG file.
# If your system doesn't allow the installation software from an unidentified developer please follow these steps or contact your system administrator:
#* Choose the Apple menu > System Preferences > click Security and Privacy.
#* Click the lock Icon to unlock it, then enter an administrator name and password.
#* Ensure that you're on the tab named “General”.
#* You should see the OpenSC app and you should be able to enable its installation by choosing “Open anyway”.
# Click continue and install until the installation is complete.

=== Linux ===

NOTE: The instructions below have been tested with the Ubuntu 20.04 release.

# Install pcscd and opensc
#: <syntaxhighlight lang="shell">
sudo add-apt-repository universe
sudo apt update
sudo apt install pcscd opensc
</syntaxhighlight>

== V. Setup the HSM ==

# Initialize the HSM. <syntaxhighlight lang="shell">
sc-hsm-tool --initialize --so-pin 3537363231383830 --pin 358138
</syntaxhighlight>
# Change the HSM so-pin.
#* '''WARNING:''' The new HSM so pin must have 16 hexadecimal digits. This is not very well known, and some HSM users have lost access to a Nitrokey HSM because they tried using regular characters and the command below accepted it.
#* '''Do NOT change the user pin. It must remain as the default for the onboarding scripts to work'''<syntaxhighlight lang="shell">
pkcs11-tool --login --login-type so --so-pin 3537363231383830 --change-pin
</syntaxhighlight>
# Create a keypair on the HSM. Enter the default pin 358138 when prompted.
#* '''Note:''' Before initializing the HSM key please refer to the [https://docs.nitrokey.com/pro/openpgp.html Nitrokey HSM documentation] if you wish to create a backup. Creating a backup of the HSM device is NOT possible after the key has already been created. <syntaxhighlight lang="shell">
pkcs11-tool -k --key-type EC:prime256v1 --login -d 01
</syntaxhighlight>

== VI. Get the node operator principal from the HSM ==
# Configure dfx identity (skip this step if you already configured it for an other HSM).
#* '''Note:''' Depending on your installation, the path to the <code>--hsm-pkcs11-lib-path</code> might be different on your platform. You can locate the correct path with the following command: <syntaxhighlight lang="shell">
find / -name opensc-pkcs11.so 2> /dev/null
</syntaxhighlight>
#* MacOS <syntaxhighlight lang="shell">
dfx identity new node-operator-hsm --hsm-key-id 01 --hsm-pkcs11-lib-path /Library/OpenSC/lib/opensc-pkcs11.so
</syntaxhighlight>
#* Linux <syntaxhighlight lang="shell">
dfx identity new node-operator-hsm --hsm-key-id 01 --hsm-pkcs11-lib-path /usr/lib/x86_64-linux-gnu/opensc-pkcs11.so
</syntaxhighlight>
# Get the principal. <syntaxhighlight lang="shell">
$ NODE_OPERATOR_PRINCIPAL=$(DFX_HSM_PIN=358138 dfx --identity node-operator-hsm identity get-principal)
$ echo $NODE_OPERATOR_PRINCIPAL

uqquy-76uhn-2mys5-xxxxx-xxxxx-xxxxx-xxxxx-xxxxx-xxxxx-xxxxx-xxx
</syntaxhighlight>

== VII. Register your NP principal to the network ==

In the next codeblock:
* Replace the <code>NEURON_ID</code> value with your neuron ID from the NNS Frontend Dapp
* Replace the <code>NODE_PROVIDER_PRINCIPAL</code> value with the Ledger Hardware Wallet principal that you got from the NNS Frontend DAPP.
* Replace the <code>NODE_PROVIDER_NAME</code> value with the name of the entity that will provide the nodes. 
* '''''IMPORTANT:''''' Please make sure that you also update the <code>--summary</code> and include a link to the forum discussion, your company's web page, and/or to another place that can convince the voting community that you are making a legitimate request. This way you will avoid the community voting NO to your proposal and you losing the staked ICPs.

# Create the Proposal <syntaxhighlight lang="shell">
NODE_PROVIDER_NAME="My Company"
NODE_PROVIDER_PRINCIPAL=fharn-5vyi2-4xb4a-64yyi-3jpmj-pga23-mxy25-d5uim-fqcro-eoefh-tae # Ledger Hardware Wallet principal, from the NNS FE dapp https://ic0.app/
NEURON_ID=13419667327548602649 # Coming from the NNS FrontEnd dapp https://nns.ic0.app/
./ic-admin \
--nns-url https://ic0.app \
-s ~/.config/dfx/identity/node-provider-hotkey/identity.pem \
propose-to-add-or-remove-node-provider add \
--proposer $NEURON_ID \
--proposal-title "Register a node provider '${NODE_PROVIDER_NAME}'" \
--summary "Register a node provider '${NODE_PROVIDER_NAME}', in line with the announcement and discussion at https://forum.dfinity.org/t/..." \
--node-provider-pid "$NODE_PROVIDER_PRINCIPAL"
</syntaxhighlight>
# Find the proposal on https://dashboard.internetcomputer.org/governance and '''wait until it is executed before proceeding to next step.'''

== VIII. Ensure that your datacenter is registered in the network ==
# Search for your data center on https://dashboard.internetcomputer.org/centers.
#* If you found the datacenter that is hosting your nodes, remember its ID, and skip the following section. Otherwise, proceed with the registration of a new DC. [[File:dc_id.png|1024px]]

=== Create a data center record for a new DC ===
In the next block of code:
* Replace the <code>--proposer</code> argument value with your Neuron ID from the NNS Frontend Dapp.
* Replace the JSON fields from <code>–data-centers-to-add</code> argument and their corresponding values in <code>--summary</code> with: <code>"id"</code>
* The ID should be combination of two letters representing a city that your datacenter is in, and an incrementing number. Search data center IDs on https://dashboard.internetcomputer.org, and find a combination of two letters and a number that’s not yet registered. Examples:
** dl1 (Dallas, no IDs with “dl” prefix)
** zh10 (Zurich, numbers 0-9 are already registered)
[[File:dc_id.png|1024px]]

* <code>"region"</code> represents the local region of a datacenter and is formulated as a three-part string divided by commas. The three parts making the string are continent, country code, and region, in the given order. Examples:
** North America,US,Florida
** Europe,DE,Bavaria
** Asia,SG,Singapore
[[File:datacenter_region.png|1024px]]

* <code>"owner"</code> The entity that provides your datacenter facilities.
** Search https://dashboard.internetcomputer.org for existing data center providers.
** If there’s match, make sure you use the same exact some name for your datacenter.
** Otherwise, name the data center owner to your best knowledge. [[File:datacenter_owner.png|1024px]]

* <code>"gps"</code> GPS coordinates.
** Find your datacenter on https://www.google.com/maps/.
** Right click on location, and select the GPS coordinates (first item in the menu) in order to copy them.
[[File:maps.png|480px|alt=Getting GPS coordinates|Getting GPS coordinates]]

# Create the proposal: <syntaxhighlight lang="shell">
NEURON_ID=13419667327548602649 # Coming from the NNS FE dapp https://nns.ic0.app/
$ ./ic-admin \
--nns-url https://ic0.app \
-s ~/.config/dfx/identity/node-provider-hotkey/identity.pem \
propose-to-add-or-remove-data-centers \
--summary "Register a Flexential datacenter as dl1 in North America,US,Texas" \
--skip-confirmation \
--proposer $NEURON_ID \
--data-centers-to-add '{
"id": "dl1",
"region": "North America,US,Texas",
"owner": "Flexential",
"gps": [
33.00803, -96.66614
]
}'
</syntaxhighlight>
# Find the proposal on https://dashboard.internetcomputer.org/governance and wait until it's executed before proceeding to next step.

== IX. Create a node operator record ==
In the next codeblock:
* Replace the <code>NEURON_ID</code> variable value with your neuron ID obtained from the NNS frontend dapp.
* Replace the <code>NODE_PROVIDER_PRINCIPAL</code> variable value with the Ledger Hardware Wallet principal obtained from the NNS frontend dapp.
* Replace the <code>DC_ID</code> variable value with id of your datacenter.
* Replace the <code>NODE_ALLOWANCE</code> variable value with number of nodes you are providing.

# Create the proposal: <syntaxhighlight lang="shell">
NEURON_ID=13419667327548602649 # Coming from the NNS FE dapp https://nns.ic0.app/
NODE_PROVIDER_PRINCIPAL=fharn-5vyi2-4xb4a-64yyi-3jpmj-pga23-mxy25-d5uim-fqcro-eoefh-tae # Ledger Hardware Wallet principal, from the NNS FE dapp https://nns.ic0.app/
NODE_OPERATOR_PRINCIPAL=$(DFX_HSM_PIN=358138 dfx --identity node-operator-hsm identity get-principal)
NODE_PROVIDER_NAME="My Company"
NODE_ALLOWANCE=28
DC_ID=dl1

./ic-admin \
--nns-url https://ic0.app \
-s ~/.config/dfx/identity/node-provider-hotkey/identity.pem \
propose-to-add-node-operator \
$NODE_PROVIDER_PRINCIPAL \
--summary "Node provider '$NODE_PROVIDER_NAME' is adding $NODE_ALLOWANCE nodes in the $DC_ID data center" \
--proposer $NEURON_ID \
--node-operator-principal-id $NODE_OPERATOR_PRINCIPAL \
--node-allowance $NODE_ALLOWANCE \
--dc-id $DC_ID
</syntaxhighlight>

# Find the proposal on https://dashboard.internetcomputer.org/governance and wait until it's executed before proceeding to next step.

== X. Onboard nodes ==

# Follow the instructions to onboard new nodes:
#* Gen2 - For NP's onboarding in 2023 and later
#** [[IC OS Installation Runbook]]
#* Gen1 - For NP's participating in the IC before 2023
#**[[IC OS Installation Runbook - PowerEdge R6525]]
#**[[IC OS Installation Runbook - Supermicro]]
# Verify that all the nodes were successfully onboarded by checking their status on the dashboard is set to either “Up” or “Unassigned”, or by checking the output from <code>ic-admin get-topology</code> command.
#* The internal dashboard can be searched by your provider principal.
[[File:onboarded_nodes.png|1024px|onboarded nodes]]

== See Also ==
* '''The Internet Computer project website (hosted on the IC): [https://internetcomputer.org/ internetcomputer.org]'''

Node Provider Networking Guide

2023-05-11T22:04:20Z

Gary.mcelroy: /* The Bare Minimum Network Requirements */ Changed wording

= NP Network Requirements - EZ Guide =
'''Who is this for?''' Node Providers (NP’s) who need to set up their servers into a rack and set up a functioning network.

'''What skills are necessary?''' You should be familiar with IP networking, network equipment and network cabling.

== The Bare Minimum Network Requirements ==
To join your servers to the Internet Computer (IC) you will need:

* 10G Network equipment
** [[wikipedia:Small_Form-factor_Pluggable|SFP+]] or [[wikipedia:10_Gigabit_Ethernet|Ethernet]]
** Switch(es)
** Cabling
** Quantity determined by number of nodes deployed
* [[Node provider hardware#Gen%202|“Gen-2” node hardware]]
* Rackspace in a data center
* Internet connection
** Bandwidth
*** ~300Mbps per node
*** Ingress/egress ratio is currently 1:1. We expect egress (serving responses to client queries) to increase faster than ingress in the future.
*** This should guide how many servers to deploy and the appropriate ISP connection speed
*** E.g. a 1Gbps connection will support up to 3 IC nodes.
** One IPv6 /64 subnet - each node gets multiple IPv6 addresses
** One IPv4 address for every 4 nodes. See [[Node Provider Network Setup Guide#Appendix 1: Number of IPv4 Addresses Required|Appendix 1]] for table.
** '''All IP addresses are assigned statically''' and automatically by IC-OS
*** This is configured in the [[IC OS Installation Runbook#4.%20Add%20configuration|IC-OS Installation Runbook]]

== Network Cabling ==
When racking and stacking your servers, ensure the '''first two 10G network ports''' on each server are connected to the 10G switch.

[[File:Supermicro 1124US-TNRP 1U server rear photo diagram.png]]

For example, on a Supermicro 1U server, the bottom two ports are considered ports 1 and 2 and will be enumerated by Linux in this order. Connect the bottom two ports to the switch.

Servers from other vendors will differ! See the server documentation for guidance.

This is subject to change - the IC-OS network configuration logic is undergoing improvements to make it more flexible.

Connect the 10G switch to the ISP endpoint - this could be the Top Of Rack (TOR) switch or other box.

== BMC Setup Recommendations ==

=== What’s a BMC? ===
The [[wikipedia:Intelligent_Platform_Management_Interface#Baseboard_management_controller|Baseboard Management Controller (BMC)]] grants control of the underlying server hardware.

BMC’s have notoriously poor security. Vendors may name their implementation differently (Dell -> iDRAC, HPE -> iLO, etc.).

=== Recommendations ===

==== Change the password ====
BMC’s usually come with a common password. Log in via crash cart, KVM or the web interface and change it to something [https://krebsonsecurity.com/password-dos-and-donts/ strong].

==== No broad internet access ====
It is highly recommended: '''do not expose your BMC''' to the broad internet. This is a safety precaution against attackers.

Options:

* Don’t connect the BMC to the internet.
** Any BMC activities occur via SSH on the host (unreliable on many mainboard vendors) or via crash cart (requires physical interaction with the machine).
* Connect the BMC to a separate dumb switch, dumb switch connects to a Rack Mounted Unit (RMU).
* Connect the BMC to a managed switch, separate VLAN

This can get complicated. It’s outside the scope of this document to explain how to do this.

Resources:

* [https://security.stackexchange.com/questions/46351/best-practice-for-accessing-management-port-of-firewall StackExchange - Best practice for accessing management port of firewall]
* [https://www.supermicro.com/products/nfo/files/IPMI/Best_Practices_BMC_Security.pdf Supermicro Guidance]
* [https://www.unicomengineering.com/blog/ipmi-best-practices/ Unicom Guidance]

== What NOT to do ==

=== Don’t use external firewalls, packet filters, rate limiters ===
Don’t block or interfere with any traffic to the node machines. This can disrupt node machine functionality. Occasionally ports are opened for incoming (and outgoing) connections when new versions of node software are deployed.

==== What about network security? ====
IC-OS manages its own firewall(s) and rate limiters very strictly and is designed with security as a primary principle.

== How DFINITY manages its servers ==
See reference DFINITY [[Gen-2 Data Center runbook|data center runbook]].

== Final Checklist ==

* Did you deploy a 10G switch?
* Do the '''first and second 10G ports''' on each server plug into the 10G switch?
* Do you have '''one IPv6 /64 prefix''' allocated from your ISP?
* Do you have at least '''one IPv4 address for every four nodes''' allocated?
* Does each node have ~300Mbps bandwidth?
* Is your '''BMC inaccessible''' from the broad internet?

== References ==

* [[Gen-2 Network Requirements|Gen2 Network Requirements]] - more detailed, possibly out of date.

== Appendix 1: Number of IPv4 Addresses Required ==
{| class="wikitable"
|# Nodes
|# IPv4 Addresses
|-
|1
|1
|-
|2
|1
|-
|3
|1
|-
|4
|1
|-
|5
|2
|-
|6
|2
|-
|7
|2
|-
|8
|2
|-
|9
|3
|-
|10
|3
|-
|11
|3
|-
|12
|3
|-
|13
|4
|-
|14
|4
|-
|15
|4
|-
|16
|4
|-
|17
|5
|-
|18
|5
|-
|19
|5
|-
|20
|5
|-
|21
|6
|-
|22
|6
|-
|23
|6
|-
|24
|6
|-
|25
|7
|-
|26
|7
|-
|27
|7
|-
|28
|7
|}

Node Provider Networking Guide

2023-05-11T22:03:26Z

Gary.mcelroy: /* The Bare Minimum Network Requirements */

= NP Network Requirements - EZ Guide =
'''Who is this for?''' Node Providers (NP’s) who need to set up their servers into a rack and set up a functioning network.

'''What skills are necessary?''' You should be familiar with IP networking, network equipment and network cabling.

== The Bare Minimum Network Requirements ==
To join your servers to the Internet Computer (IC) you will need:

* 10G Network equipment
** [[wikipedia:Small_Form-factor_Pluggable|SFP+]] or [[wikipedia:10_Gigabit_Ethernet|Ethernet]]
** Switch(es)
** Cabling
** Quantity determined by number of nodes deployed
* [[Node provider hardware#Gen%202|“Gen-2” node hardware]]
* Rackspace in a data center
* Internet connection
** Bandwidth
*** ~300Mbps minimum per node
*** Ingress/egress ratio is currently 1:1. We expect egress (client queries) to increase faster than ingress in the future.
*** This should guide how many servers to deploy and the appropriate ISP connection speed
*** E.g. a 1Gbps connection will support up to 3 IC nodes.
** One IPv6 /64 subnet - each node gets multiple IPv6 addresses
** One IPv4 address for every 4 nodes. See [[Node Provider Network Setup Guide#Appendix 1: Number of IPv4 Addresses Required|Appendix 1]] for table.
** '''All IP addresses are assigned statically''' and automatically by IC-OS
*** This is configured in the [[IC OS Installation Runbook#4.%20Add%20configuration|IC-OS Installation Runbook]]

== Network Cabling ==
When racking and stacking your servers, ensure the '''first two 10G network ports''' on each server are connected to the 10G switch.

[[File:Supermicro 1124US-TNRP 1U server rear photo diagram.png]]

For example, on a Supermicro 1U server, the bottom two ports are considered ports 1 and 2 and will be enumerated by Linux in this order. Connect the bottom two ports to the switch.

Servers from other vendors will differ! See the server documentation for guidance.

This is subject to change - the IC-OS network configuration logic is undergoing improvements to make it more flexible.

Connect the 10G switch to the ISP endpoint - this could be the Top Of Rack (TOR) switch or other box.

== BMC Setup Recommendations ==

=== What’s a BMC? ===
The [[wikipedia:Intelligent_Platform_Management_Interface#Baseboard_management_controller|Baseboard Management Controller (BMC)]] grants control of the underlying server hardware.

BMC’s have notoriously poor security. Vendors may name their implementation differently (Dell -> iDRAC, HPE -> iLO, etc.).

=== Recommendations ===

==== Change the password ====
BMC’s usually come with a common password. Log in via crash cart, KVM or the web interface and change it to something [https://krebsonsecurity.com/password-dos-and-donts/ strong].

==== No broad internet access ====
It is highly recommended: '''do not expose your BMC''' to the broad internet. This is a safety precaution against attackers.

Options:

* Don’t connect the BMC to the internet.
** Any BMC activities occur via SSH on the host (unreliable on many mainboard vendors) or via crash cart (requires physical interaction with the machine).
* Connect the BMC to a separate dumb switch, dumb switch connects to a Rack Mounted Unit (RMU).
* Connect the BMC to a managed switch, separate VLAN

This can get complicated. It’s outside the scope of this document to explain how to do this.

Resources:

* [https://security.stackexchange.com/questions/46351/best-practice-for-accessing-management-port-of-firewall StackExchange - Best practice for accessing management port of firewall]
* [https://www.supermicro.com/products/nfo/files/IPMI/Best_Practices_BMC_Security.pdf Supermicro Guidance]
* [https://www.unicomengineering.com/blog/ipmi-best-practices/ Unicom Guidance]

== What NOT to do ==

=== Don’t use external firewalls, packet filters, rate limiters ===
Don’t block or interfere with any traffic to the node machines. This can disrupt node machine functionality. Occasionally ports are opened for incoming (and outgoing) connections when new versions of node software are deployed.

==== What about network security? ====
IC-OS manages its own firewall(s) and rate limiters very strictly and is designed with security as a primary principle.

== How DFINITY manages its servers ==
See reference DFINITY [[Gen-2 Data Center runbook|data center runbook]].

== Final Checklist ==

* Did you deploy a 10G switch?
* Do the '''first and second 10G ports''' on each server plug into the 10G switch?
* Do you have '''one IPv6 /64 prefix''' allocated from your ISP?
* Do you have at least '''one IPv4 address for every four nodes''' allocated?
* Does each node have ~300Mbps bandwidth?
* Is your '''BMC inaccessible''' from the broad internet?

== References ==

* [[Gen-2 Network Requirements|Gen2 Network Requirements]] - more detailed, possibly out of date.

== Appendix 1: Number of IPv4 Addresses Required ==
{| class="wikitable"
|# Nodes
|# IPv4 Addresses
|-
|1
|1
|-
|2
|1
|-
|3
|1
|-
|4
|1
|-
|5
|2
|-
|6
|2
|-
|7
|2
|-
|8
|2
|-
|9
|3
|-
|10
|3
|-
|11
|3
|-
|12
|3
|-
|13
|4
|-
|14
|4
|-
|15
|4
|-
|16
|4
|-
|17
|5
|-
|18
|5
|-
|19
|5
|-
|20
|5
|-
|21
|6
|-
|22
|6
|-
|23
|6
|-
|24
|6
|-
|25
|7
|-
|26
|7
|-
|27
|7
|-
|28
|7
|}

Node Provider Networking Guide

2023-05-10T03:05:13Z

Gary.mcelroy: /* References */ Added small note

= NP Network Requirements - EZ Guide =
'''Who is this for?''' Node Providers (NP’s) who need to set up their servers into a rack and set up a functioning network.

'''What skills are necessary?''' You should be familiar with IP networking, network equipment and network cabling.

== The Bare Minimum Network Requirements ==
To join your servers to the Internet Computer (IC) you will need:

* 10G Network equipment
** [[wikipedia:Small_Form-factor_Pluggable|SFP+]] or [[wikipedia:10_Gigabit_Ethernet|Ethernet]]
** Switch(es)
** Cabling
** Quantity determined by number of nodes deployed
* [[Node provider hardware#Gen%202|“Gen-2” node hardware]]
* Rackspace in a data center
* Internet connection
** Bandwidth
*** ~300Mbps minimum per node
*** Ingress/egress ratio is currently 1:1. We expect egress (client queries) to increase faster than ingress in the future.
*** This should guide how many servers to deploy and the appropriate ISP connection speed
*** E.g. a 1Gbps connection will support up to 3 IC nodes.
** One IPv6 /64 subnet - each node gets multiple IPv6 addresses
** One IPv4 address for every 4 nodes. See [[Node Provider Network Setup Guide#Appendix 1: Number of IPv4 Addresses Required|Appendix 1]] for more details.
** '''All IP addresses are assigned statically''' and automatically by IC-OS
*** This is configured in the [[IC OS Installation Runbook#4.%20Add%20configuration|IC-OS Installation Runbook]]

== Network Cabling ==
When racking and stacking your servers, ensure the '''first two 10G network ports''' on each server are connected to the 10G switch.

[[File:Supermicro 1124US-TNRP 1U server rear photo diagram.png]]

For example, on a Supermicro 1U server, the bottom two ports are considered ports 1 and 2 and will be enumerated by Linux in this order. Connect the bottom two ports to the switch.

Servers from other vendors will differ! See the server documentation for guidance.

This is subject to change - the IC-OS network configuration logic is undergoing improvements to make it more flexible.

Connect the 10G switch to the ISP endpoint - this could be the Top Of Rack (TOR) switch or other box.

== BMC Setup Recommendations ==

=== What’s a BMC? ===
The [[wikipedia:Intelligent_Platform_Management_Interface#Baseboard_management_controller|Baseboard Management Controller (BMC)]] grants control of the underlying server hardware.

BMC’s have notoriously poor security. Vendors may name their implementation differently (Dell -> iDRAC, HPE -> iLO, etc.).

=== Recommendations ===

==== Change the password ====
BMC’s usually come with a common password. Log in via crash cart, KVM or the web interface and change it to something [https://krebsonsecurity.com/password-dos-and-donts/ strong].

==== No broad internet access ====
It is highly recommended: '''do not expose your BMC''' to the broad internet. This is a safety precaution against attackers.

Options:

* Don’t connect the BMC to the internet.
** Any BMC activities occur via SSH on the host (unreliable on many mainboard vendors) or via crash cart (requires physical interaction with the machine).
* Connect the BMC to a separate dumb switch, dumb switch connects to a Rack Mounted Unit (RMU).
* Connect the BMC to a managed switch, separate VLAN

This can get complicated. It’s outside the scope of this document to explain how to do this.

Resources:

* [https://security.stackexchange.com/questions/46351/best-practice-for-accessing-management-port-of-firewall StackExchange - Best practice for accessing management port of firewall]
* [https://www.supermicro.com/products/nfo/files/IPMI/Best_Practices_BMC_Security.pdf Supermicro Guidance]
* [https://www.unicomengineering.com/blog/ipmi-best-practices/ Unicom Guidance]

== What NOT to do ==

=== Don’t use external firewalls, packet filters, rate limiters ===
Don’t block or interfere with any traffic to the node machines. This can disrupt node machine functionality. Occasionally ports are opened for incoming (and outgoing) connections when new versions of node software are deployed.

==== What about network security? ====
IC-OS manages its own firewall(s) and rate limiters very strictly and is designed with security as a primary principle.

== How DFINITY manages its servers ==
See reference DFINITY [[Gen-2 Data Center runbook|data center runbook]].

== Final Checklist ==

* Did you deploy a 10G switch?
* Do the '''first and second 10G ports''' on each server plug into the 10G switch?
* Do you have '''one IPv6 /64 prefix''' allocated from your ISP?
* Do you have at least '''one IPv4 address for every four nodes''' allocated?
* Does each node have ~300Mbps bandwidth?
* Is your '''BMC inaccessible''' from the broad internet?

== References ==

* [[Gen-2 Network Requirements|Gen2 Network Requirements]] - more detailed, possibly out of date.

== Appendix 1: Number of IPv4 Addresses Required ==
{| class="wikitable"
|# Nodes
|# IPv4 Addresses
|-
|1
|1
|-
|2
|1
|-
|3
|1
|-
|4
|1
|-
|5
|2
|-
|6
|2
|-
|7
|2
|-
|8
|2
|-
|9
|3
|-
|10
|3
|-
|11
|3
|-
|12
|3
|-
|13
|4
|-
|14
|4
|-
|15
|4
|-
|16
|4
|-
|17
|5
|-
|18
|5
|-
|19
|5
|-
|20
|5
|-
|21
|6
|-
|22
|6
|-
|23
|6
|-
|24
|6
|-
|25
|7
|-
|26
|7
|-
|27
|7
|-
|28
|7
|}

Node Provider Networking Guide

2023-05-10T03:04:52Z

Gary.mcelroy: Added link to Gen2 Network Requirements document

= NP Network Requirements - EZ Guide =
'''Who is this for?''' Node Providers (NP’s) who need to set up their servers into a rack and set up a functioning network.

'''What skills are necessary?''' You should be familiar with IP networking, network equipment and network cabling.

== The Bare Minimum Network Requirements ==
To join your servers to the Internet Computer (IC) you will need:

* 10G Network equipment
** [[wikipedia:Small_Form-factor_Pluggable|SFP+]] or [[wikipedia:10_Gigabit_Ethernet|Ethernet]]
** Switch(es)
** Cabling
** Quantity determined by number of nodes deployed
* [[Node provider hardware#Gen%202|“Gen-2” node hardware]]
* Rackspace in a data center
* Internet connection
** Bandwidth
*** ~300Mbps minimum per node
*** Ingress/egress ratio is currently 1:1. We expect egress (client queries) to increase faster than ingress in the future.
*** This should guide how many servers to deploy and the appropriate ISP connection speed
*** E.g. a 1Gbps connection will support up to 3 IC nodes.
** One IPv6 /64 subnet - each node gets multiple IPv6 addresses
** One IPv4 address for every 4 nodes. See [[Node Provider Network Setup Guide#Appendix 1: Number of IPv4 Addresses Required|Appendix 1]] for more details.
** '''All IP addresses are assigned statically''' and automatically by IC-OS
*** This is configured in the [[IC OS Installation Runbook#4.%20Add%20configuration|IC-OS Installation Runbook]]

== Network Cabling ==
When racking and stacking your servers, ensure the '''first two 10G network ports''' on each server are connected to the 10G switch.

[[File:Supermicro 1124US-TNRP 1U server rear photo diagram.png]]

For example, on a Supermicro 1U server, the bottom two ports are considered ports 1 and 2 and will be enumerated by Linux in this order. Connect the bottom two ports to the switch.

Servers from other vendors will differ! See the server documentation for guidance.

This is subject to change - the IC-OS network configuration logic is undergoing improvements to make it more flexible.

Connect the 10G switch to the ISP endpoint - this could be the Top Of Rack (TOR) switch or other box.

== BMC Setup Recommendations ==

=== What’s a BMC? ===
The [[wikipedia:Intelligent_Platform_Management_Interface#Baseboard_management_controller|Baseboard Management Controller (BMC)]] grants control of the underlying server hardware.

BMC’s have notoriously poor security. Vendors may name their implementation differently (Dell -> iDRAC, HPE -> iLO, etc.).

=== Recommendations ===

==== Change the password ====
BMC’s usually come with a common password. Log in via crash cart, KVM or the web interface and change it to something [https://krebsonsecurity.com/password-dos-and-donts/ strong].

==== No broad internet access ====
It is highly recommended: '''do not expose your BMC''' to the broad internet. This is a safety precaution against attackers.

Options:

* Don’t connect the BMC to the internet.
** Any BMC activities occur via SSH on the host (unreliable on many mainboard vendors) or via crash cart (requires physical interaction with the machine).
* Connect the BMC to a separate dumb switch, dumb switch connects to a Rack Mounted Unit (RMU).
* Connect the BMC to a managed switch, separate VLAN

This can get complicated. It’s outside the scope of this document to explain how to do this.

Resources:

* [https://security.stackexchange.com/questions/46351/best-practice-for-accessing-management-port-of-firewall StackExchange - Best practice for accessing management port of firewall]
* [https://www.supermicro.com/products/nfo/files/IPMI/Best_Practices_BMC_Security.pdf Supermicro Guidance]
* [https://www.unicomengineering.com/blog/ipmi-best-practices/ Unicom Guidance]

== What NOT to do ==

=== Don’t use external firewalls, packet filters, rate limiters ===
Don’t block or interfere with any traffic to the node machines. This can disrupt node machine functionality. Occasionally ports are opened for incoming (and outgoing) connections when new versions of node software are deployed.

==== What about network security? ====
IC-OS manages its own firewall(s) and rate limiters very strictly and is designed with security as a primary principle.

== How DFINITY manages its servers ==
See reference DFINITY [[Gen-2 Data Center runbook|data center runbook]].

== Final Checklist ==

* Did you deploy a 10G switch?
* Do the '''first and second 10G ports''' on each server plug into the 10G switch?
* Do you have '''one IPv6 /64 prefix''' allocated from your ISP?
* Do you have at least '''one IPv4 address for every four nodes''' allocated?
* Does each node have ~300Mbps bandwidth?
* Is your '''BMC inaccessible''' from the broad internet?

== References ==

* [[Gen-2 Network Requirements|Gen2 Network Requirements]] - possibly out of date.

== Appendix 1: Number of IPv4 Addresses Required ==
{| class="wikitable"
|# Nodes
|# IPv4 Addresses
|-
|1
|1
|-
|2
|1
|-
|3
|1
|-
|4
|1
|-
|5
|2
|-
|6
|2
|-
|7
|2
|-
|8
|2
|-
|9
|3
|-
|10
|3
|-
|11
|3
|-
|12
|3
|-
|13
|4
|-
|14
|4
|-
|15
|4
|-
|16
|4
|-
|17
|5
|-
|18
|5
|-
|19
|5
|-
|20
|5
|-
|21
|6
|-
|22
|6
|-
|23
|6
|-
|24
|6
|-
|25
|7
|-
|26
|7
|-
|27
|7
|-
|28
|7
|}

Node Provider Documentation

2023-05-10T03:02:38Z

Gary.mcelroy: /* Gen-2 Documentation */ Removed network req's link, add network guide link, removed DFINITY network runbook link (available on network guide)

==Introduction==

ICP runs on a [[Sovereign Network]] that is a governed by a DAO using [[Deterministic Decentralization | deterministic decentralization]] to maximize its [[Decentralization in ICP: Infrastructure Governance | decentralization]]. To be part of the ICP infrastructure, any potential node providers can submit NNS proposals to the DAO controlling the ICP blockchain. ICP community then votes on whether to include the node provider. Node providers invest in and operate the node hardware which powers the Internet Computer. Running these nodes in data centers provides the high performance and the cost-effectiveness of the Internet Computer. Every node provider is allowed a limited amount of nodes.

This article is the hub for ICP node provider documentation.

==Node Provider Tokenomics & Remuneration==
Node providers receive rewards (remuneration) for operating node machines that run the IC network. The single source of truth for node provider rewards is the NNS, where changes can only be made through NNS proposals adopted by the IC community.

This page summarizes the current node provider rewards and serves to discuss proposals for future reward models: [[Node Provider Remuneration]]

==Node Machine Hardware Requirements==

Node machines on ICP network need to keep up with the requirements of the network, please see: [[Node Machine Hardware#Gen_2 | Gen-2 Node Machine Hardware]].

==Submitting Proposal to Join the Network==

As part of the process to become a node provider, a candidate node provider has to declare their intent and self-identify so the ICP DAO can make an informed decision. Please see [[Node Provider Self-declaration]] for more info.

==Onboarding for accepted Node Providers==

===Gen-2 Documentation===

These articles are for candidate node providers considering becoming node providers or node providers recently accepted by the ICP DAO. The term "Gen-2" refers to "Generation 2", which is the current set of protocols for new node providers.

* [[Node Machine Hardware#Gen_2 | Gen-2 Node Machine Hardware]]
* [[Node Provider Network Setup Guide]]
* [[Node Provider Remuneration]]
* [[Node Provider Self-declaration]]
* [[Node Provider Onboarding]]
* [[IC OS Installation Runbook]] for Gen-2

Users accepted by the ICP DAO to be a node provider, can follow these instructions to add their node: [[Node Provider Onboarding]].

===Node Provider Troubleshooting===
* [[Node Provider Troubleshooting]]
** [[Possible Node Onboarding Errors]]
** [[Unhealthy Nodes]]
** [[Updating Firmware]]
** [[iDRAC access and TSR logs]]

===Gen-1 Documentation===
The first batch of ICP node providers joined under Gen-1 (Generation 1). These documents are for those legacy node providers.

* [[IC OS Installation Runbook - Dell Poweredge]] for Gen-1
* [[IC OS Installation Runbook - Supermicro]] for Gen-1
* [[Storage Runbook]] for Gen-1

Node Provider Networking Guide

2023-05-10T02:58:23Z

Gary.mcelroy: /* The Bare Minimum Network Requirements */ Added local link

= NP Network Requirements - EZ Guide =
'''Who is this for?''' Node Providers (NP’s) who need to set up their servers into a rack and set up a functioning network.

'''What skills are necessary?''' You should be familiar with IP networking, network equipment and network cabling.

== The Bare Minimum Network Requirements ==
To join your servers to the Internet Computer (IC) you will need:

* 10G Network equipment
** [[wikipedia:Small_Form-factor_Pluggable|SFP+]] or [[wikipedia:10_Gigabit_Ethernet|Ethernet]]
** Switch(es)
** Cabling
** Quantity determined by number of nodes deployed
* [[Node provider hardware#Gen%202|“Gen-2” node hardware]]
* Rackspace in a data center
* Internet connection
** Bandwidth
*** ~300Mbps minimum per node
*** Ingress/egress ratio is currently 1:1. We expect egress (client queries) to increase faster than ingress in the future.
*** This should guide how many servers to deploy and the appropriate ISP connection speed
*** E.g. a 1Gbps connection will support up to 3 IC nodes.
** One IPv6 /64 subnet - each node gets multiple IPv6 addresses
** One IPv4 address for every 4 nodes. See [[Node Provider Network Setup Guide#Appendix 1: Number of IPv4 Addresses Required|Appendix 1]] for more details.
** '''All IP addresses are assigned statically''' and automatically by IC-OS
*** This is configured in the [[IC OS Installation Runbook#4.%20Add%20configuration|IC-OS Installation Runbook]]

== Network Cabling ==
When racking and stacking your servers, ensure the '''first two 10G network ports''' on each server are connected to the 10G switch.

[[File:Supermicro 1124US-TNRP 1U server rear photo diagram.png]]

For example, on a Supermicro 1U server, the bottom two ports are considered ports 1 and 2 and will be enumerated by Linux in this order. Connect the bottom two ports to the switch.

Servers from other vendors will differ! See the server documentation for guidance.

This is subject to change - the IC-OS network configuration logic is undergoing improvements to make it more flexible.

Connect the 10G switch to the ISP endpoint - this could be the Top Of Rack (TOR) switch or other box.

== BMC Setup Recommendations ==

=== What’s a BMC? ===
The [[wikipedia:Intelligent_Platform_Management_Interface#Baseboard_management_controller|Baseboard Management Controller (BMC)]] grants control of the underlying server hardware.

BMC’s have notoriously poor security. Vendors may name their implementation differently (Dell -> iDRAC, HPE -> iLO, etc.).

=== Recommendations ===

==== Change the password ====
BMC’s usually come with a common password. Log in via crash cart, KVM or the web interface and change it to something [https://krebsonsecurity.com/password-dos-and-donts/ strong].

==== No broad internet access ====
It is highly recommended: '''do not expose your BMC''' to the broad internet. This is a safety precaution against attackers.

Options:

* Don’t connect the BMC to the internet.
** Any BMC activities occur via SSH on the host (unreliable on many mainboard vendors) or via crash cart (requires physical interaction with the machine).
* Connect the BMC to a separate dumb switch, dumb switch connects to a Rack Mounted Unit (RMU).
* Connect the BMC to a managed switch, separate VLAN

This can get complicated. It’s outside the scope of this document to explain how to do this.

Resources:

* [https://security.stackexchange.com/questions/46351/best-practice-for-accessing-management-port-of-firewall StackExchange - Best practice for accessing management port of firewall]
* [https://www.supermicro.com/products/nfo/files/IPMI/Best_Practices_BMC_Security.pdf Supermicro Guidance]
* [https://www.unicomengineering.com/blog/ipmi-best-practices/ Unicom Guidance]

== What NOT to do ==

=== Don’t use external firewalls, packet filters, rate limiters ===
Don’t block or interfere with any traffic to the node machines. This can disrupt node machine functionality. Occasionally ports are opened for incoming (and outgoing) connections when new versions of node software are deployed.

==== What about network security? ====
IC-OS manages its own firewall(s) and rate limiters very strictly and is designed with security as a primary principle.

== How DFINITY manages its servers ==
See reference DFINITY [[Gen-2 Data Center runbook|data center runbook]].

== Final Checklist ==

* Did you deploy a 10G switch?
* Do the '''first and second 10G ports''' on each server plug into the 10G switch?
* Do you have '''one IPv6 /64 prefix''' allocated from your ISP?
* Do you have at least '''one IPv4 address for every four nodes''' allocated?
* Does each node have ~300Mbps bandwidth?
* Is your '''BMC inaccessible''' from the broad internet?

== Appendix 1: Number of IPv4 Addresses Required ==
{| class="wikitable"
|# Nodes
|# IPv4 Addresses
|-
|1
|1
|-
|2
|1
|-
|3
|1
|-
|4
|1
|-
|5
|2
|-
|6
|2
|-
|7
|2
|-
|8
|2
|-
|9
|3
|-
|10
|3
|-
|11
|3
|-
|12
|3
|-
|13
|4
|-
|14
|4
|-
|15
|4
|-
|16
|4
|-
|17
|5
|-
|18
|5
|-
|19
|5
|-
|20
|5
|-
|21
|6
|-
|22
|6
|-
|23
|6
|-
|24
|6
|-
|25
|7
|-
|26
|7
|-
|27
|7
|-
|28
|7
|}

Node Provider Networking Guide

2023-05-10T02:57:44Z

Gary.mcelroy: Created NP 'EZ' network guide.

= NP Network Requirements - EZ Guide =
'''Who is this for?''' Node Providers (NP’s) who need to set up their servers into a rack and set up a functioning network.

'''What skills are necessary?''' You should be familiar with IP networking, network equipment and network cabling.

== The Bare Minimum Network Requirements ==
To join your servers to the Internet Computer (IC) you will need:

* 10G Network equipment
** [[wikipedia:Small_Form-factor_Pluggable|SFP+]] or [[wikipedia:10_Gigabit_Ethernet|Ethernet]]
** Switch(es)
** Cabling
** Quantity determined by number of nodes deployed
* [[Node provider hardware#Gen%202|“Gen-2” node hardware]]
* Rackspace in a data center
* Internet connection
** Bandwidth
*** ~300Mbps minimum per node
*** Ingress/egress ratio is currently 1:1. We expect egress (client queries) to increase faster than ingress in the future.
*** This should guide how many servers to deploy and the appropriate ISP connection speed
*** E.g. a 1Gbps connection will support up to 3 IC nodes.
** One IPv6 /64 subnet - each node gets multiple IPv6 addresses
** One IPv4 address for every 4 nodes. See Appendix 1 for more details.
** '''All IP addresses are assigned statically''' and automatically by IC-OS
*** This is configured in the [[IC OS Installation Runbook#4.%20Add%20configuration|IC-OS Installation Runbook]]

== Network Cabling ==
When racking and stacking your servers, ensure the '''first two 10G network ports''' on each server are connected to the 10G switch.

[[File:Supermicro 1124US-TNRP 1U server rear photo diagram.png]]

For example, on a Supermicro 1U server, the bottom two ports are considered ports 1 and 2 and will be enumerated by Linux in this order. Connect the bottom two ports to the switch.

Servers from other vendors will differ! See the server documentation for guidance.

This is subject to change - the IC-OS network configuration logic is undergoing improvements to make it more flexible.

Connect the 10G switch to the ISP endpoint - this could be the Top Of Rack (TOR) switch or other box.

== BMC Setup Recommendations ==

=== What’s a BMC? ===
The [[wikipedia:Intelligent_Platform_Management_Interface#Baseboard_management_controller|Baseboard Management Controller (BMC)]] grants control of the underlying server hardware.

BMC’s have notoriously poor security. Vendors may name their implementation differently (Dell -> iDRAC, HPE -> iLO, etc.).

=== Recommendations ===

==== Change the password ====
BMC’s usually come with a common password. Log in via crash cart, KVM or the web interface and change it to something [https://krebsonsecurity.com/password-dos-and-donts/ strong].

==== No broad internet access ====
It is highly recommended: '''do not expose your BMC''' to the broad internet. This is a safety precaution against attackers.

Options:

* Don’t connect the BMC to the internet.
** Any BMC activities occur via SSH on the host (unreliable on many mainboard vendors) or via crash cart (requires physical interaction with the machine).
* Connect the BMC to a separate dumb switch, dumb switch connects to a Rack Mounted Unit (RMU).
* Connect the BMC to a managed switch, separate VLAN

This can get complicated. It’s outside the scope of this document to explain how to do this.

Resources:

* [https://security.stackexchange.com/questions/46351/best-practice-for-accessing-management-port-of-firewall StackExchange - Best practice for accessing management port of firewall]
* [https://www.supermicro.com/products/nfo/files/IPMI/Best_Practices_BMC_Security.pdf Supermicro Guidance]
* [https://www.unicomengineering.com/blog/ipmi-best-practices/ Unicom Guidance]

== What NOT to do ==

=== Don’t use external firewalls, packet filters, rate limiters ===
Don’t block or interfere with any traffic to the node machines. This can disrupt node machine functionality. Occasionally ports are opened for incoming (and outgoing) connections when new versions of node software are deployed.

==== What about network security? ====
IC-OS manages its own firewall(s) and rate limiters very strictly and is designed with security as a primary principle.

== How DFINITY manages its servers ==
See reference DFINITY [[Gen-2 Data Center runbook|data center runbook]].

== Final Checklist ==

* Did you deploy a 10G switch?
* Do the '''first and second 10G ports''' on each server plug into the 10G switch?
* Do you have '''one IPv6 /64 prefix''' allocated from your ISP?
* Do you have at least '''one IPv4 address for every four nodes''' allocated?
* Does each node have ~300Mbps bandwidth?
* Is your '''BMC inaccessible''' from the broad internet?

== Appendix 1: Number of IPv4 Addresses Required ==
{| class="wikitable"
|# Nodes
|# IPv4 Addresses
|-
|1
|1
|-
|2
|1
|-
|3
|1
|-
|4
|1
|-
|5
|2
|-
|6
|2
|-
|7
|2
|-
|8
|2
|-
|9
|3
|-
|10
|3
|-
|11
|3
|-
|12
|3
|-
|13
|4
|-
|14
|4
|-
|15
|4
|-
|16
|4
|-
|17
|5
|-
|18
|5
|-
|19
|5
|-
|20
|5
|-
|21
|6
|-
|22
|6
|-
|23
|6
|-
|24
|6
|-
|25
|7
|-
|26
|7
|-
|27
|7
|-
|28
|7
|}

File:Supermicro 1124US-TNRP 1U server rear photo diagram.png

2023-05-10T02:52:01Z

Gary.mcelroy:

Supermicro 1124US-TNRP 1U server rear photo diagram

Node Provider Onboarding

2023-05-10T02:24:45Z

Gary.mcelroy: /* X. Onboard nodes */ Promoted Gen2 in the order of instructions

Learn how to participate in the Internet Computer network as a Node Provider and to receive rewards for supporting the network.

== Requirements ==

* [https://wiki.internetcomputer.org/wiki/Node_provider_hardware Node Hardware]
* Rack space with a 10Gb connectivity, RJ45 terminated on the nodes
* Public /28 IPv4 range and /64 IPv6 range
* [https://www.ledger.com/ Hardware wallet]
* [https://shop.nitrokey.com/shop/product/nkhs2-nitrokey-hsm-2-7/ NitroKey HSM]
* 11 ICP (10 of which are to be staked for the NNS proposal deposit)
* Basic understanding of neurons, staking, and governance proposals. For instance, understanding what it means to stake a neuron for 8 years.
* The technical knowledge to understand some minor steps that are not explicitly mentioned in these instructions. For instance, when to insert an HSM.

'''Note:''' Please allocate at least 0.5 day for going through the first part, i.e., the registration of a new NP. It may even take a couple of days, depending on how quickly the community votes for the proposals. There is a also fair amount of complexity and the technical knowledge that needs to be absorbed in order to complete the steps. But this only needs to be done once. 
The next step, going to the DC and bringing up and onboarding the machines, is much quicker. Estimate to spend 10-15 minutes per machine. This time should go down to ~5 minutes as you gain experience. Also, multiple machines can be brought up in parallel.

== I. Install the required tools ==
===''' A. Install ic-admin '''===

<code>ic-admin</code> is the tool used to create and submit NNS proposals.

==== MacOS ====
# Retrieve the file <syntaxhighlight lang="shell">
curl "https://download.dfinity.systems/ic/7445081734e6d896d090295967d50710975c4f25/openssl-static-binaries/x86_64-darwin/ic-admin.gz" -o - | gunzip > ./ic-admin
chmod +x ./ic-admin
</syntaxhighlight>
# Verify the binary <syntaxhighlight lang="shell">
diff <(shasum -a 256 ./ic-admin | cut -d' ' -f1) <(echo 3f75026d2f28f171068e332a42c82a2795c93fbf5ab351baef30b30eb901cdba) && echo "ic-admin checksum matches" || echo "***ERROR***: ic-admin checksum does not match"
</syntaxhighlight>

==== Linux ====
NOTE: The instructions below have been tested with the Ubuntu 20.04 release.
# Retrieve the file <syntaxhighlight lang="shell">
curl "https://download.dfinity.systems/ic/7445081734e6d896d090295967d50710975c4f25/openssl-static-binaries/x86_64-linux/ic-admin.gz" -o - | gunzip > ./ic-admin
chmod +x ./ic-admin
</syntaxhighlight>
# Verify the binary <syntaxhighlight lang="shell">
diff <(shasum -a 256 ./ic-admin | cut -d' ' -f1) <(echo e29bb9cc462e800b8b960ad49c412e5f5fdbb5ae2ae9fde0c13058422ba32802) && echo "ic-admin checksum matches" || echo "***ERROR***: ic-admin checksum does not match"
</syntaxhighlight>

===''' B. Install dfx '''===

# <code>dfx</code> allows generating a neuron hotkey, among other things <syntaxhighlight lang="shell">
$ DFX_VERSION=0.9.3 sh -ci "$(curl -fsSL https://sdk.dfinity.org/install.sh)"
</syntaxhighlight>
# Verify that the version is 0.9.3 <syntaxhighlight lang="shell">
$ export PATH=$HOME/bin:$PATH
$ dfx --version dfx 0.9.3
</syntaxhighlight>
# Create an identity for the Node Provider '''Hotkey''' <syntaxhighlight lang="shell">
$ dfx identity new node-provider-hotkey
Creating identity: "node-provider-hotkey".
Created identity: "node-provider-hotkey".
$ dfx --identity node-provider-hotkey identity get-principal
wuyst-x5tpn-g5wri-mp3ps-vjtba-de3xs-w5xgb-crvek-tucbe-o5rqi-mae
</syntaxhighlight>
# '''Note:''' The node provider hotkey is NOT the node provider principal. This is the hotkey that is used for the NNS proposal submissions only.

== II. Create and Manage Neuron via NNS Frontend Dapp and Internet Identity ==

# Setup your hardware wallet: https://medium.com/dfinity/integrating-ledger-nano-with-the-nns-front-end-dapp-user-manual-9c5600925e16
# Send at least 11 ICPs to the hardware wallet address.
# Navigate to Neurons tab and create a Neuron by staking at least 10 ICP from your hardware wallet. Staking more ICP works as well, but 10 is the minimum.
# IMPORTANT! Confirm the transaction on your hardware wallet.
#: [[File:-docs-stake_neuron_1.png|1024px|stake neuron]]
#:
# After the Neuron has been created successfully, confirm to add NNS Dapp as hotkey in the dialogue and on your hardware wallet, and close the dialog after the action completes.
#: [[File:-docs-stake_neuron_2.png|1024px|neuron id]]
# Set the dissolve delay to at least 6 months, and confirm the choice in the dialogue and on your hardware wallet. After the action completes, you can close the "Follow Neurons".
#:
#: [[File:dissolve_delay.png|480px|neuron id]]
# You will now see a Neuron listed with its ID. Copy the Neuron ID, since you will need it in the next steps to place the necessary proposals.
#: [[File:Neuron id.png|1024px]]

== III. Add hotkeys ==

# Select the Neuron you just created to open Neuron management view and press “Add hotkey” button.
#: [[File:Hotkey 1.png|800px]]
# A dialog will pop up where you can enter the principal you generated in step 2 (output from command <code>dfx --identity node-provider-hotkey identity get-principal</code>). This will allow you to submit NNS proposals using <code>ic-admin</code> and will not be used for anything else. 
#:Press the confirm button and confirm the transactions on your hardware wallet. 
#: [[File:Hotkey 2.png|800px]]
#:
# Get the Ledger Hardware Wallet Principal Id: Navigate back to ICP page and select your Ledger hardware wallet account. You will need to use this Ledger Hardware Wallet principal as the Node Provider principal in order to get the rewards directly into the secure hardware wallet.
[[File:Node provider principal 1.png|1024px]]
[[File:Node provider principal 2.png|800px]]
# Copy and save this Node Provider principal by clicking on the copy icon after the principal id. You'll need it in the next steps.

<syntaxhighlight lang="shell">
NODE_PROVIDER_PRINCIPAL=fharn-5vyi2-4xb4a-64yyi-3jpmj-pga23-mxy25-d5uim-fqcro-eoefh-tae # Ledger Hardware Wallet principal, from the NNS FrontEnd dapp https://nns.ic0.app/
</syntaxhighlight>

== IV. Configure your HSM ==
It's first necessary to install the necessary tools.

=== MacOS ===
# Download this OpenSC binary: https://github.com/OpenSC/OpenSC/releases/download/0.22.0/OpenSC-0.22.0.dmg
# Double click the DMG image that you downloaded and then double click the OpenSC PKG file.
# If your system doesn't allow the installation software from an unidentified developer please follow these steps or contact your system administrator:
#* Choose the Apple menu > System Preferences > click Security and Privacy.
#* Click the lock Icon to unlock it, then enter an administrator name and password.
#* Ensure that you're on the tab named “General”.
#* You should see the OpenSC app and you should be able to enable its installation by choosing “Open anyway”.
# Click continue and install until the installation is complete.

=== Linux ===

NOTE: The instructions below have been tested with the Ubuntu 20.04 release.

# Install pcscd and opensc
#: <syntaxhighlight lang="shell">
sudo add-apt-repository universe
sudo apt update
sudo apt install pcscd opensc
</syntaxhighlight>

== V. Setup the HSM ==

# Initialize the HSM. <syntaxhighlight lang="shell">
sc-hsm-tool --initialize --so-pin 3537363231383830 --pin 358138
</syntaxhighlight>
# Change the HSM so-pin.
#* '''WARNING:''' The new HSM so pin must have 16 hexadecimal digits. This is not very well known, and some HSM users have lost access to a Nitrokey HSM because they tried using regular characters and the command below accepted it.
#* '''Do NOT change the user pin. It must remain as the default for the onboarding scripts to work'''<syntaxhighlight lang="shell">
pkcs11-tool --login --login-type so --so-pin 3537363231383830 --change-pin
</syntaxhighlight>
# Create a keypair on the HSM. Enter the default pin 358138 when prompted.
#* '''Note:''' Before initializing the HSM key please refer to the [https://docs.nitrokey.com/pro/openpgp.html Nitrokey HSM documentation] if you wish to create a backup. Creating a backup of the HSM device is NOT possible after the key has already been created. <syntaxhighlight lang="shell">
pkcs11-tool -k --key-type EC:prime256v1 --login -d 01
</syntaxhighlight>

== VI. Get the node operator principal from the HSM ==
# Configure dfx identity (skip this step if you already configured it for an other HSM).
#* '''Note:''' Depending on your installation, the path to the <code>--hsm-pkcs11-lib-path</code> might be different on your platform. You can locate the correct path with the following command: <syntaxhighlight lang="shell">
find / -name opensc-pkcs11.so 2> /dev/null
</syntaxhighlight>
#* MacOS <syntaxhighlight lang="shell">
dfx identity new node-operator-hsm --hsm-key-id 01 --hsm-pkcs11-lib-path /Library/OpenSC/lib/opensc-pkcs11.so
</syntaxhighlight>
#* Linux <syntaxhighlight lang="shell">
dfx identity new node-operator-hsm --hsm-key-id 01 --hsm-pkcs11-lib-path /usr/lib/x86_64-linux-gnu/opensc-pkcs11.so
</syntaxhighlight>
# Get the principal. <syntaxhighlight lang="shell">
$ NODE_OPERATOR_PRINCIPAL=$(DFX_HSM_PIN=358138 dfx --identity node-operator-hsm identity get-principal)
$ echo $NODE_OPERATOR_PRINCIPAL

uqquy-76uhn-2mys5-xxxxx-xxxxx-xxxxx-xxxxx-xxxxx-xxxxx-xxxxx-xxx
</syntaxhighlight>

== VII. Register your NP principal to the network ==

In the next codeblock:
* Replace the <code>NEURON_ID</code> value with your neuron ID from the NNS Frontend Dapp
* Replace the <code>NODE_PROVIDER_PRINCIPAL</code> value with the Ledger Hardware Wallet principal that you got from the NNS Frontend DAPP.
* Replace the <code>NODE_PROVIDER_NAME</code> value with the name of the entity that will provide the nodes. 
* '''''IMPORTANT:''''' Please make sure that you also update the <code>--summary</code> and include a link to the forum discussion, your company's web page, and/or to another place that can convince the voting community that you are making a legitimate request. This way you will avoid the community voting NO to your proposal and you losing the staked ICPs.

# Create the Proposal <syntaxhighlight lang="shell">
NODE_PROVIDER_NAME="My Company"
NODE_PROVIDER_PRINCIPAL=fharn-5vyi2-4xb4a-64yyi-3jpmj-pga23-mxy25-d5uim-fqcro-eoefh-tae # Ledger Hardware Wallet principal, from the NNS FE dapp https://ic0.app/
NEURON_ID=13419667327548602649 # Coming from the NNS FrontEnd dapp https://nns.ic0.app/
./ic-admin \
--nns-url https://ic0.app \
-s ~/.config/dfx/identity/node-provider-hotkey/identity.pem \
propose-to-add-or-remove-node-provider add \
--proposer $NEURON_ID \
--proposal-title "Register a node provider '${NODE_PROVIDER_NAME}'" \
--summary "Register a node provider '${NODE_PROVIDER_NAME}', in line with the announcement and discussion at https://forum.dfinity.org/t/..." \
--node-provider-pid "$NODE_PROVIDER_PRINCIPAL"
</syntaxhighlight>
# Find the proposal on https://dashboard.internetcomputer.org/governance and '''wait until it is executed before proceeding to next step.'''

== VIII. Ensure that your datacenter is registered in the network ==
# Search for your data center on https://dashboard.internetcomputer.org/centers.
#* If you found the datacenter that is hosting your nodes, remember its ID, and skip the following section. Otherwise, proceed with the registration of a new DC. [[File:dc_id.png|1024px]]

=== Create a data center record for a new DC ===
In the next block of code:
* Replace the <code>--proposer</code> argument value with your Neuron ID from the NNS Frontend Dapp.
* Replace the JSON fields from <code>–data-centers-to-add</code> argument and their corresponding values in <code>--summary</code> with: <code>"id"</code>
* The ID should be combination of two letters representing a city that your datacenter is in, and an incrementing number. Search data center IDs on https://dashboard.internetcomputer.org, and find a combination of two letters and a number that’s not yet registered. Examples:
** dl1 (Dallas, no IDs with “dl” prefix)
** zh10 (Zurich, numbers 0-9 are already registered)
[[File:dc_id.png|1024px]]

* <code>"region"</code> represents the local region of a datacenter and is formulated as a three-part string divided by commas. The three parts making the string are continent, country code, and region, in the given order. Examples:
** North America,US,Florida
** Europe,DE,Bavaria
** Asia,SG,Singapore
[[File:datacenter_region.png|1024px]]

* <code>"owner"</code> The entity that provides your datacenter facilities.
** Search https://dashboard.internetcomputer.org for existing data center providers.
** If there’s match, make sure you use the same exact some name for your datacenter.
** Otherwise, name the data center owner to your best knowledge. [[File:datacenter_owner.png|1024px]]

* <code>"gps"</code> GPS coordinates.
** Find your datacenter on https://www.google.com/maps/.
** Right click on location, and select the GPS coordinates (first item in the menu) in order to copy them.
[[File:maps.png|480px|alt=Getting GPS coordinates|Getting GPS coordinates]]

# Create the proposal: <syntaxhighlight lang="shell">
NEURON_ID=13419667327548602649 # Coming from the NNS FE dapp https://nns.ic0.app/
$ ./ic-admin \
--nns-url https://ic0.app \
-s ~/.config/dfx/identity/node-provider-hotkey/identity.pem \
propose-to-add-or-remove-data-centers \
--summary "Register a Flexential datacenter as dl1 in North America,US,Texas" \
--skip-confirmation \
--proposer $NEURON_ID \
--data-centers-to-add '{
"id": "dl1",
"region": "North America,US,Texas",
"owner": "Flexential",
"gps": [
33.00803, -96.66614
]
}'
</syntaxhighlight>
# Find the proposal on https://dashboard.internetcomputer.org/governance and wait until it's executed before proceeding to next step.

== IX. Create a node operator record ==
In the next codeblock:
* Replace the <code>NEURON_ID</code> variable value with your neuron ID obtained from the NNS frontend dapp.
* Replace the <code>NODE_PROVIDER_PRINCIPAL</code> variable value with the Ledger Hardware Wallet principal obtained from the NNS frontend dapp.
* Replace the <code>DC_ID</code> variable value with id of your datacenter.
* Replace the <code>NODE_ALLOWANCE</code> variable value with number of nodes you are providing.

# Create the proposal: <syntaxhighlight lang="shell">
NEURON_ID=13419667327548602649 # Coming from the NNS FE dapp https://nns.ic0.app/
NODE_PROVIDER_PRINCIPAL=fharn-5vyi2-4xb4a-64yyi-3jpmj-pga23-mxy25-d5uim-fqcro-eoefh-tae # Ledger Hardware Wallet principal, from the NNS FE dapp https://nns.ic0.app/
NODE_OPERATOR_PRINCIPAL=$(DFX_HSM_PIN=358138 dfx --identity node-operator-hsm identity get-principal)
NODE_PROVIDER_NAME="My Company"
NODE_ALLOWANCE=28
DC_ID=dl1

./ic-admin \
--nns-url https://ic0.app \
-s ~/.config/dfx/identity/node-provider-hotkey/identity.pem \
propose-to-add-node-operator \
$NODE_PROVIDER_PRINCIPAL \
--summary "Node provider '$NODE_PROVIDER_NAME' is adding $NODE_ALLOWANCE nodes in the $DC_ID data center" \
--proposer $NEURON_ID \
--node-operator-principal-id $NODE_OPERATOR_PRINCIPAL \
--node-allowance $NODE_ALLOWANCE \
--dc-id $DC_ID
</syntaxhighlight>

# Find the proposal on https://dashboard.internetcomputer.org/governance and wait until it's executed before proceeding to next step.

== X. Onboard nodes ==

# Follow the instructions to onboard new nodes:
#* Gen2 - For NP's onboarding in 2023 and later
#** [[IC OS Installation Runbook]]
#* Gen1 - For NP's participating in the IC before 2023
#**[[IC OS Installation Runbook - PowerEdge R6525]]
#**[[IC OS Installation Runbook - Supermicro]]
# Verify that all the nodes were successfully onboarded by checking their status on the dashboard is set to either “Up” or “Unassigned”, or by checking the output from <code>ic-admin get-topology</code> command.
#* The internal dashboard can be searched by your provider principal.
[[File:onboarded_nodes.png|1024px|onboarded nodes]]

== XI. Set the reward configuration for your nodes ==
In the next codeblock:
* Replace the <code>NEURON_ID</code> variable value with your neuron ID obtained from the NNS frontend dapp. 
* Replace the <code><NODE_X_PRINCIPAL></code> placeholders with your node principals. 
* Replace the <code><number-of-nodes></code> placeholder with the number of nodes you listed.
* Note: The current maximum number of nodes per node operator are 28.

<syntaxhighlight lang="shell">
NEURON_ID=13419667327548602649 # Coming from the NNS FE dapp https://nns.ic0.app/
NODE_OPERATOR_PRINCIPAL=$(DFX_HSM_PIN=358138 dfx --identity node-operator-hsm identity get-principal)

./ic-admin \
--nns-url https://ic0.app \
-s ~/.config/dfx/identity/node-provider-hotkey/identity.pem \
propose-to-update-node-operator-config \
--proposer $NEURON_ID \
--summary "Set rewards for the following nodes:

* <NODE_1_PRINCIPAL>
* <NODE_2_PRINCIPAL>
* ...
" \
--node-operator-id $NODE_OPERATOR_PRINCIPAL \
--rewardable-nodes '{"type0": <number-of-nodes>}'
</syntaxhighlight>

==See Also==
* '''The Internet Computer project website (hosted on the IC): [https://internetcomputer.org/ internetcomputer.org]'''

Node Provider Machine Hardware Guide

2023-05-08T20:39:21Z

Gary.mcelroy: /* Gen 2 Node Machine */ Clarified CPU requirements

== What are the Hardware Requirements for Node Machines? ==
Node providers operate one or more node machines than run in the IC network. Gen1 Hardware requirements have been used by Node Providers to set up node machines during Genesis launch.

The Gen2 Hardware requirements have been defined for the further growth of the IC network. The specifications for the Gen2 node machines are generic (instead of vendor specific) and supports VM memory encryption and attestation which will be needed in future features on the IC.

Below are the up-to-date specifications for both the Gen2 node machines and Gen1 node machines.

== Gen 2 Node Machine ==
=== Generic specification Gen2 Node Machine ===
{| class="wikitable"
|-
| Dual Socket AMD EPYC Milan CPU - Recommended: [https://en.wikichip.org/wiki/amd/epyc/7313 7313] (16C/32T 3 Ghz)
[https://en.wikichip.org/wiki/amd/epyc#7003_Series_.28Zen_3.29 optionally] 7343, 7373, 73F3
|-
| 16x 32GB RDIMM, 3200MT/s, Dual Rank
|-
| 5x 6.4TB NVMe Mixed Mode (DWPD >= 3)
|-
| Dual Port 10G SFP or BASE-T
|-
| TPM 2.0
|}
 
Note the NVMe drives should be recognized by Linux as NVMe (i.e., show up as `/dev/nvme*` devices). SATA backplanes or any other hardware which prevents this should not be used.
 
=== Validated Configurations ===
DFINITY has [https://forum.dfinity.org/t/draft-motion-proposal-new-hardware-specification-and-remuneration-for-ic-nodes/14202/14?u=garym validated] the following Gen2 hardware configurations.
 
==== Validated configuration: Dell ====
{| class="wikitable"
|-
| 2 || AMD EPYC 7343 3.2GHz, 16C/32T, 128M Cache (190W)
|-
| 16 || 32GB RDIMM, 3200MT/s, Dual Rank 16Gb BASE x8
|-
| 5 || 6.4TB Enterprise NVMe Mixed Use AG Drive U.2 Gen4 with carrier
|-
| 1 || PowerEdge R6525 Motherboard, with 2 x 1Gb Onboard LOM (BCM5720)MLK V2
|-
| 2 || Dual, Hot-plug, Redundant Power Supply (1+1) 1100W, Mixed Mode Titanium
|-
| 1 || Intel X710 Dual Port 10GbE SFP+, OCP NIC 3.0
|-
| 1 || Trusted Platform Module 2.0 V3
|}
 

==== Validated configuration: ASUS ====
{| class="wikitable"
|-
| 2 || AMD EPYC 7313 (3,00 GHz, 16-Core, 128 MB)
|-
| 16 || 32GB ECC Reg ATP DDR4 3200 RAM
|-
| 5 || 6.4 TB NVMe Kioxia SSD 3D-NAND TLC U.3 (Kioxia CM6-V)
|-
| 1 || Asus Mainboard KMPP-D32 Series (without OCP 3.0, without Pike)
|-
| 2 || 1600 Watt redundant PSU
|-
| 1 || Broadcom 25 Gigabit P225P SFP28 Dual Port Network Card
|}
 
==== Validated configurations: Supermicro & Gigabyte ====
Validation is being re-run on Supermicro and Gigabyte machines which match the spec. This section will be updated when those results are ready.

== Gen 1 Node Machine ==
=== Node Machine Type 1 - Dell ===
{| class="wikitable"
|-
| 1 || R6525
|-
| 1 || Chassis - Supports Up to 10 NVMe drives, 12 drives total
|-
| 1 || Dual 1 GB on Motherboard
|-
| 3 || Low Profile PCIe Slots
|-
| - || 3 Year Basic NBD Support
|-
| - || iDrac Enterprise
|-
| 1 || Dual port 10GbE Base - T Adapter Broadcom, PCIe Low Profile
|-
| 10 || 3.2TB NVMe, Mixed Use, 2.5" with Carrier
|-
| 16 || 32GB RDIMM (3200MT/s)
|-
| 2 || AMD 7302 3GHz, 16C/32T, 128M, 155W, 3200
|-
| 1 || Single Power Supply (800W)
|-
| 1 || C13-C14, 3M, 125V 15A Power Cored
|-
|}
 
=== Node Machine Type 1 - SuperMicro ===
{| class="wikitable"
|-
| 1 || AS - 1023US - TR4
|-
| 2 || Rome 7302 DP/UP 16C/32T 3.0
|-
| 16 || 32GB DDR4-3200 2Rx4 ECC REG DIMM
|-
| 5 || Samsung PM983 3.2TB NVMe PCIE/SATA Hybrid M.2 & 1 PCIE
|-
| 2 || 800W Power Supply
|-
| 1 || Std LP 2-port 10G RJ45, Intel x540
|-
| 5 || Micron 5300 PRO 7.4TB, SATA, 2.5', 3D TLC, .6DWPD (with Caddie)
|-
| 1 || C13/C14 13A Power Cord
|}
 
=== Node Machine Type 2 - Dell ===
{| class="wikitable"
|-
| 1 || R6515
|-
| 1 || 3.5" Chassis with up to 4 Hot-Plug Hard Drives and OS RAID
|-
| 1 || Dual 1 Gb on Motherboard
|-
| 3 || Low Profile PCIe Slots
|-
| 1 || Standard Fan
|-
| - || 3 Year Basic NBD Support
|-
| - || iDrac Enterprise
|-
| 2 || Dual Port 10GbE Base - T Adapter Broadcom, PCIe LOw Profile
|-
| 2 || 480GB SSD SATA Mix Use 6Gbps 512 2.5in Hot-Plug AG Drive, 3.5in
|-
| 4 || 8GB RDIMM, 3200 MT/s, Single Rank
|-
| 1 || AMD EPYC 7232P 3.10GHz, 8C/16T, 64M Cache (120W) DDR4-3200
|-
| 1 || Dual Hot-Plug Redundant Power Supply (1+1), 550W
|-
| 2 || Jumper Cord - C13/C14, .6M, 250V, 13A
|}

Gen-2 Data Center runbook

2023-05-03T17:33:49Z

Gary.mcelroy: Changed from thumbnail to basic.

== ICR Gen2 build - DFINITY-managed reference design ==
''This is a work-in-progress''

This runbook illustrates how DFINITY manages a typical rack of IC nodes (an 'ICR'). It is published for the benefit of node providers to show one possible implementation of the ICR Gen2 networking requirements.

This runbook is NOT mandatory for ICR builds that are not managed by DFINITY.

=== Prerequisites ===
Collect the following information for the variables that will need to be adjusted for your installation.
* site-ID (i.e. “zh2” or “mr1”)
* PDU outlet type in racks (IEC 60320 C13 or national power outlet)
* IPv4 subnet assignment for management network - a new /25 subnet has to be assigned by the node provider based on NP’s addressing plan (DFINITY-owned ICRs use 10.10.X.128/25 addressing scheme where X is assigned to each ICR)
* RMU or jump-box HW (server, power cords and 5x 2m (7ft) Cat6 cables and the required installation artifacts are available

=== Uplink configuration ===
Collect information about the uplinks and verify the minimum requirements:
# Management Port
#* Assigned public IPv4 range (min /31): [FILL IN]
#* Default GW address: [FILL IN]
#* 1G/10G, media type fiber/copper: [FILL IN]
#** if fiber: multi-mode/single-mode: [FILL IN]
#** if fiber: patch panel connector type (SC/PC or LC/PC or E2000/APC …): [FILL IN]
#** if 1G fiber: required transceiver type (LX/SX/other): [FILL IN]
#** if 10G fiber: required transceiver type (LR/SR/other): [FILL IN]
#Production Port
#* Assigned public IPv6 range (/64): [FILL IN]
#* IPv6 Default GW address: [FILL IN]
#* (optional) assigned public IPv4 range (min /29): [FILL IN]
#* (optional) IPv4 Default GW address: [FILL IN]
#* 1G/10G, media type fiber/copper: [FILL IN]
#** if fiber: multi-mode/single-mode: [FILL IN]
#** if fiber: patch panel connector type (SC/PC or LC/PC or E2000/APC …): [FILL IN]
#** if 1G fiber: required transceiver type (LX/SX/other): [FILL IN]
#** if 10G fiber: required transceiver type (LR/SR/other): [FILL IN]
#** if other fiber: required transceiver type: [FILL IN]
# Location and the circuit IDs or patch-panel positions of the management and production ports: [FILL IN]
# Number and location of the racks (1 or 2): [FILL IN]

=== HW requirements ===
===== Verify BOM for each rack: =====
* 1x Dell EMC S3048-ON with 2x AC PSU and PSU->ports airflow
* 1x Dell EMC S4148T-ON with 2x AC PSU and PSU->ports airflow (or alternative - see below)
* 4x power cords for the switches (PDU outlet type to C13)

Alternatives for S4148T-ON (can be used when servers have SFP+ or SFP28 cages):
* 1x Dell EMC S4148F-ON with 2x AC PSU and PSU->ports airflow
* 1x Dell EMC S5248F-ON with 2x AC PSU and PSU->ports airflow

===== Verify BOM for ICR: =====
* If management port is single-mode fiber:
* 1x management port fiber patch cord 9/125 LC/PC to patch-panel type connector
* 1x transceiver (SFP or SFP+) matching the management port required transceiver type (probably -LX or -LR)

If production port is single-mode fiber:
* 1x production port fiber patch cord 9/125 LC/PC to patch-panel type connector
* 1x transceiver (SFP or SFP+ or QSFP+ or QSFP28) matching the production port required transceiver type
* 40G QSFP+ to 10G SFP+ Adapter Converter Module if switch is S4148T-ON and the transceiver is SFP or SFP+

If management port is multi-mode fiber:
* 1x management port fiber patch cord 50/125 LC/PC to patch-panel type connector
* 1x transceiver (SFP or SFP+) matching the management port required transceiver type (probably -SX or -SR)

If production port is multi-mode fiber:
* 1x production port fiber patch cord 50/125 LC/PC to patch-panel type connector
* 1x transceiver (SFP or SFP+ or QSFP+ or QSFP28) matching the production port required transceiver type if production port is fiber (probably -SX or -SR)
* 40G QSFP+ to 10G SFP+ Adapter Converter Module if switch is S4148T-ON and the transceiver is SFP or SFP+

If management port is copper:
* 2m (7ft) Cat6 copper cable

If production port is copper:
* 2m (7ft) Cat6 copper cable
* 40G QSFP+ to 10G SFP+ Adapter Converter Module if switch is S4148T-ON and the transceiver is SFP or SFP+
* 10GBASE-T SFP+ if production switch is Dell EMC S4148F-ON or Dell EMC S5248F-ON

===== Verify BOM for 2-rack ICR: =====
If the DC rules allows running DAC or AOC cables between the racks and the position of the racks and the maximum length of the available AOC or DAC allows connecting the switches:
* 1x 100G QSFP28 AOC or DAC cable (matching the required length from one production switch to another - with neighboring racks it is usually 5m (16ft) )
* 2x Cat6 interconnects between the racks (matching the required length from one management switch to another and from RMU to the switch in the second rack, usually 5m (16ft))

Otherwise:
* 2x single-mode (9/125) optical paths between the racks (four fibers or two pairs in total)
* 2x 100G QSFP28 -LR4, -LR or -CWDM4 transceivers
* 2x 9/125 2m (7ft) patch cords LC/PC to connector of the optical path termination
* 2x Cat6 interconnects between the racks
* 4x Cat6 2m (7ft) patch cables

=== Marking ===
* (virtually) mark the rack with both management and production uplink Rack 1
* (virtually) mark the rack with no uplinks Rack 2 (if the rack exists)
* Mark the production switch in Rack 1: {site-ID}-sw02
* Mark the production switch in Rack 2: {site-ID}-sw04 (if the rack exists)
* Mark the management switch in the Rack 1: {site-ID}-msw01
* Mark the management switch in the Rack 2: {site-ID}-msw02 (if the rack exists)

=== Rack and stack devices ===
Racking and stacking of devices is beyond the scope of this runbook.

The site should include the following components:
* Rack 1
** {site-ID}-sw02
** {site-ID}-msw01
** RMU / jump-box
** 0-14x servers (IC nodes)
* Rack 2 (if present)
** {site-ID}-sw04
** {site-ID}-msw02
** 0-14x servers (IC nodes)

=== Cabling ===
===== Minimum required steps =====
* In each rack connect PSUs of each switch the PDUs using the power cords, select different power rails for PSU1 and PSU2
* In Rack 1 connect the production uplink ([carrier] Internet) to {site-ID}-sw02 port 25 using the selected transceiver and 40G QSFP+ to 10G SFP+ Adapter Converter Module if switch is S4148T-ON and the transceiver is SFP or SFP+
* In Rack 1 connect the management uplink to RMU/jump-box port wan (for jump-box TBD runbook)
* In Rack 1 connect Cat6 cable from RMU port management1 to {site-ID}-msw01 port 46
* In Rack 1 connect Cat6 cable from RMU port management2 to {site-ID}-msw01 management port
* In Rack 1 connect Cat6 cable from RMU port lan1 to {site-ID}-sw02 port 54
* In Rack 1 connect Cat6 cable from {site-ID}-sw02 management port to {site-ID}-msw01 port 47

If Rack 2 is present:
* Connect AOC/DAC or fiber optical path with QSFP28 transceivers from Rack 1 {site-ID}-sw02 port 30 to Rack 2 {site-ID}-sw04 port 30
* Connect Cat6 cable from Rack 1 {site-ID}-mw01 port 48 to Rack 2 {site-ID}-msw02 port 48
* Connect Cat6 cable from Rack 1 RMU port management3 to Rack 2 {site-ID}-msw02 management port
* Connect Cat6 cable from port {site-ID}-msw02 port 47 to Rack 2 {site-ID}-sw04 management port

===== Wiring diagram =====
[[File:2023-05-03_updated_dfinity_icr_rack_diagram.png.png|alt=]]

===== Servers =====
The exact cabling instructions for servers are not part of this runbook because the servers differ in the port type, number of ports and location of ports. High-level requirements for each server:
* Connect the BMC / IPMI / iLO port of the server to the management switch in the same rack (allocate ports from left to right, take the first free ports with the matching speed and port type)
* Connect the first 10G/25G port to the production switch in the same rack (allocate ports from left to right, take the first free ports with the matching speed and port type)
* Use Cat6 cables or PatchBox Cat6 modules for connecting the servers if 10GBASE-T is supported by the switch and the server or at least one side; use 10GBASE-T SFP+ transceivers to adapt switch or server side with SFP+ or SFP28 interfaces
* Use AOC or DAC cables in case both the switch and the server have SFP+ or SFP28 interfaces; select the matching speed and type of the cable for the interface and verify the vendor compatibility on switch side and also on server side

===== Dell OS10 NOS install =====
Verify that Dell OS10 (version >=10.4) is installed on all switches.
* If no NOS is installed or older version is installed, use NOS installation guide:
https://www.dell.com/support/manuals/en-us/force10-s4048-on/ee-upgrade-downgrade/installing-smartfabric-os10?guid=guid-9bf59a6c-9be9-4abb-99cf-b2671091f3e0&lang=en-us
* It is suggested to install the OS10 from USB stick as described in the “Manual installation” section:
https://www.dell.com/support/manuals/en-us/smartfabric-os10-emp-partner/ee-upgrade-downgrade/manual-installation?guid=guid-d4a157a0-e1fc-4ad7-bb68-cd98fdcc0025&lang=en-us
* For upgrading OS10 see the instructions in ”Upgrading OS10 software” chapter: https://www.dell.com/support/manuals/en-sg/smartfabric-os10-emp-partner/ee-upgrade-downgrade/upgrading-os10-software?guid=guid-29a7887c-d5ed-4896-9cc6-9dcd614c0aee&lang=en-us

===== Dell OS10 switch minimum configuration =====
* All switches are active and can be connected right after the NOS installation. Further configuration is required to set user accounts, re-set default passwords and switch hardening.

Prerequisites:
* Generate a password for admin user
* Generate a password for linuxadmin user
* (optional) Collect SSH key for admin user
* Prepare serial console connection to the switch (see “Log in to an OS10 switch” chapter: https://www.dell.com/support/manuals/en-sg/dell-emc-smartfabric-os10/ee-upgrade-downgrade/log-in-to-an-os10-switch?guid=guid-977e7f9f-3175-49b4-a0bc-5e8a15d8c424&lang=en-us ) or collect the IP address from DHCP server if the switch has been auto-installed
* Assign management network IP addresses (following example is based on 10.10.X.128/25 assignment - DFINITY-only address plan):
** 10.10.X.254 - RMU or jump-box acting as default GW
** 10.10.X.140 - {site-ID}-msw01
** 10.10.X.141 - {site-ID}-msw02
** 10.10.X.142 - {site-ID}-sw02
** 10.10.X.144 - {site-ID}-sw04

Procedure:
* Connect to the switch using the serial console and screen {switch tty device - i.e. /dev/ttyUSB0} 115200 (Linux) or PuTTY (Windows)
* Enter configuration mode:
{site-ID}-{switch name}# configure terminal
{site-ID}-{switch name}(config)#
Clear interface mgm1/1/1:
interface mgmt1/1/1
shutdown
no ip address dhcp
no ipv6 address autoconfig
Configure management VRF and the management interface
ip vrf management
interface management
!
interface mgmt1/1/1
no shutdown
no ip address dhcp
ip address {switch IP address - 10.10.X.Y/25}
ipv6 address autoconfig
!
management route 0.0.0.0/0 10.10.X.254
Configure users and basic configuration
ip http vrf management
default mtu 9216
hostname {site-ID}-{switch name}
system-user linuxadmin password {linuxadmin password}
ip name-server vrf management 1.1.1.1
username admin password {admin password} role sysadmin priv-lvl 15
username admin sshkey "{admin SSH key}"
snmp-server community public ro
snmp-server contact "Contact Support"
ntp server pool.ntp.org
ntp source mgmt1/1/1
ntp enable vrf management

=== Tests ===
* Ping from RMU / jump-box to the switch management interfaces:
ping -c4 10.10.X.140
ping -c4 10.10.X.141 (if installed)
ping -c4 10.10.X.142
ping -c4 10.10.X.144 (if installed)
* Test server BMC and IPv6 connectivity
** Deploy at least one server in each rack, collect BMC IP address in the management network and the server IP in production network
** Ping the collected server IPs from RMU / jump-box

=== Dell OS10 operation recommendations ===
The details of Dell OS10-based switch operation is beyond the scope of this runbook. The ultimate responsibility for the network connection availability, quality and security lies with the node provider. The qualitative parameters depend among others also on the switching fabric performance and health. Therefore it is strongly recommended to implement the following steps:

** Overall health of all switches should be watched - PSU, fan and memory and CPU load (SNMP-based monitoring or gRPC Streaming Telemetry)
** Port load of all switches should be watched (using gRPC Streaming Telemetry)
** New versions of OS10, new known issues and security advisories should be periodically (once a month) evaluated and upgrades should be scheduled when there is a relevant issue or enhancement in the newly available version
** The HW lifetime upgrade path should be followed according to the vendor’s recommendations

==See Also==
* '''The Internet Computer project website (hosted on the IC): [https://internetcomputer.org/ internetcomputer.org]'''

Gen-2 Data Center runbook

2023-05-03T17:33:01Z

Gary.mcelroy: Use updated diagram. Mgmt switches don't get node traffic.

== ICR Gen2 build - DFINITY-managed reference design ==
''This is a work-in-progress''

This runbook illustrates how DFINITY manages a typical rack of IC nodes (an 'ICR'). It is published for the benefit of node providers to show one possible implementation of the ICR Gen2 networking requirements.

This runbook is NOT mandatory for ICR builds that are not managed by DFINITY.

=== Prerequisites ===
Collect the following information for the variables that will need to be adjusted for your installation.
* site-ID (i.e. “zh2” or “mr1”)
* PDU outlet type in racks (IEC 60320 C13 or national power outlet)
* IPv4 subnet assignment for management network - a new /25 subnet has to be assigned by the node provider based on NP’s addressing plan (DFINITY-owned ICRs use 10.10.X.128/25 addressing scheme where X is assigned to each ICR)
* RMU or jump-box HW (server, power cords and 5x 2m (7ft) Cat6 cables and the required installation artifacts are available

=== Uplink configuration ===
Collect information about the uplinks and verify the minimum requirements:
# Management Port
#* Assigned public IPv4 range (min /31): [FILL IN]
#* Default GW address: [FILL IN]
#* 1G/10G, media type fiber/copper: [FILL IN]
#** if fiber: multi-mode/single-mode: [FILL IN]
#** if fiber: patch panel connector type (SC/PC or LC/PC or E2000/APC …): [FILL IN]
#** if 1G fiber: required transceiver type (LX/SX/other): [FILL IN]
#** if 10G fiber: required transceiver type (LR/SR/other): [FILL IN]
#Production Port
#* Assigned public IPv6 range (/64): [FILL IN]
#* IPv6 Default GW address: [FILL IN]
#* (optional) assigned public IPv4 range (min /29): [FILL IN]
#* (optional) IPv4 Default GW address: [FILL IN]
#* 1G/10G, media type fiber/copper: [FILL IN]
#** if fiber: multi-mode/single-mode: [FILL IN]
#** if fiber: patch panel connector type (SC/PC or LC/PC or E2000/APC …): [FILL IN]
#** if 1G fiber: required transceiver type (LX/SX/other): [FILL IN]
#** if 10G fiber: required transceiver type (LR/SR/other): [FILL IN]
#** if other fiber: required transceiver type: [FILL IN]
# Location and the circuit IDs or patch-panel positions of the management and production ports: [FILL IN]
# Number and location of the racks (1 or 2): [FILL IN]

=== HW requirements ===
===== Verify BOM for each rack: =====
* 1x Dell EMC S3048-ON with 2x AC PSU and PSU->ports airflow
* 1x Dell EMC S4148T-ON with 2x AC PSU and PSU->ports airflow (or alternative - see below)
* 4x power cords for the switches (PDU outlet type to C13)

Alternatives for S4148T-ON (can be used when servers have SFP+ or SFP28 cages):
* 1x Dell EMC S4148F-ON with 2x AC PSU and PSU->ports airflow
* 1x Dell EMC S5248F-ON with 2x AC PSU and PSU->ports airflow

===== Verify BOM for ICR: =====
* If management port is single-mode fiber:
* 1x management port fiber patch cord 9/125 LC/PC to patch-panel type connector
* 1x transceiver (SFP or SFP+) matching the management port required transceiver type (probably -LX or -LR)

If production port is single-mode fiber:
* 1x production port fiber patch cord 9/125 LC/PC to patch-panel type connector
* 1x transceiver (SFP or SFP+ or QSFP+ or QSFP28) matching the production port required transceiver type
* 40G QSFP+ to 10G SFP+ Adapter Converter Module if switch is S4148T-ON and the transceiver is SFP or SFP+

If management port is multi-mode fiber:
* 1x management port fiber patch cord 50/125 LC/PC to patch-panel type connector
* 1x transceiver (SFP or SFP+) matching the management port required transceiver type (probably -SX or -SR)

If production port is multi-mode fiber:
* 1x production port fiber patch cord 50/125 LC/PC to patch-panel type connector
* 1x transceiver (SFP or SFP+ or QSFP+ or QSFP28) matching the production port required transceiver type if production port is fiber (probably -SX or -SR)
* 40G QSFP+ to 10G SFP+ Adapter Converter Module if switch is S4148T-ON and the transceiver is SFP or SFP+

If management port is copper:
* 2m (7ft) Cat6 copper cable

If production port is copper:
* 2m (7ft) Cat6 copper cable
* 40G QSFP+ to 10G SFP+ Adapter Converter Module if switch is S4148T-ON and the transceiver is SFP or SFP+
* 10GBASE-T SFP+ if production switch is Dell EMC S4148F-ON or Dell EMC S5248F-ON

===== Verify BOM for 2-rack ICR: =====
If the DC rules allows running DAC or AOC cables between the racks and the position of the racks and the maximum length of the available AOC or DAC allows connecting the switches:
* 1x 100G QSFP28 AOC or DAC cable (matching the required length from one production switch to another - with neighboring racks it is usually 5m (16ft) )
* 2x Cat6 interconnects between the racks (matching the required length from one management switch to another and from RMU to the switch in the second rack, usually 5m (16ft))

Otherwise:
* 2x single-mode (9/125) optical paths between the racks (four fibers or two pairs in total)
* 2x 100G QSFP28 -LR4, -LR or -CWDM4 transceivers
* 2x 9/125 2m (7ft) patch cords LC/PC to connector of the optical path termination
* 2x Cat6 interconnects between the racks
* 4x Cat6 2m (7ft) patch cables

=== Marking ===
* (virtually) mark the rack with both management and production uplink Rack 1
* (virtually) mark the rack with no uplinks Rack 2 (if the rack exists)
* Mark the production switch in Rack 1: {site-ID}-sw02
* Mark the production switch in Rack 2: {site-ID}-sw04 (if the rack exists)
* Mark the management switch in the Rack 1: {site-ID}-msw01
* Mark the management switch in the Rack 2: {site-ID}-msw02 (if the rack exists)

=== Rack and stack devices ===
Racking and stacking of devices is beyond the scope of this runbook.

The site should include the following components:
* Rack 1
** {site-ID}-sw02
** {site-ID}-msw01
** RMU / jump-box
** 0-14x servers (IC nodes)
* Rack 2 (if present)
** {site-ID}-sw04
** {site-ID}-msw02
** 0-14x servers (IC nodes)

=== Cabling ===
===== Minimum required steps =====
* In each rack connect PSUs of each switch the PDUs using the power cords, select different power rails for PSU1 and PSU2
* In Rack 1 connect the production uplink ([carrier] Internet) to {site-ID}-sw02 port 25 using the selected transceiver and 40G QSFP+ to 10G SFP+ Adapter Converter Module if switch is S4148T-ON and the transceiver is SFP or SFP+
* In Rack 1 connect the management uplink to RMU/jump-box port wan (for jump-box TBD runbook)
* In Rack 1 connect Cat6 cable from RMU port management1 to {site-ID}-msw01 port 46
* In Rack 1 connect Cat6 cable from RMU port management2 to {site-ID}-msw01 management port
* In Rack 1 connect Cat6 cable from RMU port lan1 to {site-ID}-sw02 port 54
* In Rack 1 connect Cat6 cable from {site-ID}-sw02 management port to {site-ID}-msw01 port 47

If Rack 2 is present:
* Connect AOC/DAC or fiber optical path with QSFP28 transceivers from Rack 1 {site-ID}-sw02 port 30 to Rack 2 {site-ID}-sw04 port 30
* Connect Cat6 cable from Rack 1 {site-ID}-mw01 port 48 to Rack 2 {site-ID}-msw02 port 48
* Connect Cat6 cable from Rack 1 RMU port management3 to Rack 2 {site-ID}-msw02 management port
* Connect Cat6 cable from port {site-ID}-msw02 port 47 to Rack 2 {site-ID}-sw04 management port

===== Wiring diagram =====
[[File:2023-05-03 updated dfinity icr rack diagram.png.png|none|thumb]]

===== Servers =====
The exact cabling instructions for servers are not part of this runbook because the servers differ in the port type, number of ports and location of ports. High-level requirements for each server:
* Connect the BMC / IPMI / iLO port of the server to the management switch in the same rack (allocate ports from left to right, take the first free ports with the matching speed and port type)
* Connect the first 10G/25G port to the production switch in the same rack (allocate ports from left to right, take the first free ports with the matching speed and port type)
* Use Cat6 cables or PatchBox Cat6 modules for connecting the servers if 10GBASE-T is supported by the switch and the server or at least one side; use 10GBASE-T SFP+ transceivers to adapt switch or server side with SFP+ or SFP28 interfaces
* Use AOC or DAC cables in case both the switch and the server have SFP+ or SFP28 interfaces; select the matching speed and type of the cable for the interface and verify the vendor compatibility on switch side and also on server side

===== Dell OS10 NOS install =====
Verify that Dell OS10 (version >=10.4) is installed on all switches.
* If no NOS is installed or older version is installed, use NOS installation guide:
https://www.dell.com/support/manuals/en-us/force10-s4048-on/ee-upgrade-downgrade/installing-smartfabric-os10?guid=guid-9bf59a6c-9be9-4abb-99cf-b2671091f3e0&lang=en-us
* It is suggested to install the OS10 from USB stick as described in the “Manual installation” section:
https://www.dell.com/support/manuals/en-us/smartfabric-os10-emp-partner/ee-upgrade-downgrade/manual-installation?guid=guid-d4a157a0-e1fc-4ad7-bb68-cd98fdcc0025&lang=en-us
* For upgrading OS10 see the instructions in ”Upgrading OS10 software” chapter: https://www.dell.com/support/manuals/en-sg/smartfabric-os10-emp-partner/ee-upgrade-downgrade/upgrading-os10-software?guid=guid-29a7887c-d5ed-4896-9cc6-9dcd614c0aee&lang=en-us

===== Dell OS10 switch minimum configuration =====
* All switches are active and can be connected right after the NOS installation. Further configuration is required to set user accounts, re-set default passwords and switch hardening.

Prerequisites:
* Generate a password for admin user
* Generate a password for linuxadmin user
* (optional) Collect SSH key for admin user
* Prepare serial console connection to the switch (see “Log in to an OS10 switch” chapter: https://www.dell.com/support/manuals/en-sg/dell-emc-smartfabric-os10/ee-upgrade-downgrade/log-in-to-an-os10-switch?guid=guid-977e7f9f-3175-49b4-a0bc-5e8a15d8c424&lang=en-us ) or collect the IP address from DHCP server if the switch has been auto-installed
* Assign management network IP addresses (following example is based on 10.10.X.128/25 assignment - DFINITY-only address plan):
** 10.10.X.254 - RMU or jump-box acting as default GW
** 10.10.X.140 - {site-ID}-msw01
** 10.10.X.141 - {site-ID}-msw02
** 10.10.X.142 - {site-ID}-sw02
** 10.10.X.144 - {site-ID}-sw04

Procedure:
* Connect to the switch using the serial console and screen {switch tty device - i.e. /dev/ttyUSB0} 115200 (Linux) or PuTTY (Windows)
* Enter configuration mode:
{site-ID}-{switch name}# configure terminal
{site-ID}-{switch name}(config)#
Clear interface mgm1/1/1:
interface mgmt1/1/1
shutdown
no ip address dhcp
no ipv6 address autoconfig
Configure management VRF and the management interface
ip vrf management
interface management
!
interface mgmt1/1/1
no shutdown
no ip address dhcp
ip address {switch IP address - 10.10.X.Y/25}
ipv6 address autoconfig
!
management route 0.0.0.0/0 10.10.X.254
Configure users and basic configuration
ip http vrf management
default mtu 9216
hostname {site-ID}-{switch name}
system-user linuxadmin password {linuxadmin password}
ip name-server vrf management 1.1.1.1
username admin password {admin password} role sysadmin priv-lvl 15
username admin sshkey "{admin SSH key}"
snmp-server community public ro
snmp-server contact "Contact Support"
ntp server pool.ntp.org
ntp source mgmt1/1/1
ntp enable vrf management

=== Tests ===
* Ping from RMU / jump-box to the switch management interfaces:
ping -c4 10.10.X.140
ping -c4 10.10.X.141 (if installed)
ping -c4 10.10.X.142
ping -c4 10.10.X.144 (if installed)
* Test server BMC and IPv6 connectivity
** Deploy at least one server in each rack, collect BMC IP address in the management network and the server IP in production network
** Ping the collected server IPs from RMU / jump-box

=== Dell OS10 operation recommendations ===
The details of Dell OS10-based switch operation is beyond the scope of this runbook. The ultimate responsibility for the network connection availability, quality and security lies with the node provider. The qualitative parameters depend among others also on the switching fabric performance and health. Therefore it is strongly recommended to implement the following steps:

** Overall health of all switches should be watched - PSU, fan and memory and CPU load (SNMP-based monitoring or gRPC Streaming Telemetry)
** Port load of all switches should be watched (using gRPC Streaming Telemetry)
** New versions of OS10, new known issues and security advisories should be periodically (once a month) evaluated and upgrades should be scheduled when there is a relevant issue or enhancement in the newly available version
** The HW lifetime upgrade path should be followed according to the vendor’s recommendations

==See Also==
* '''The Internet Computer project website (hosted on the IC): [https://internetcomputer.org/ internetcomputer.org]'''

File:2023-05-03 updated dfinity icr rack diagram.png.png

2023-05-03T17:31:17Z

Gary.mcelroy:

2023-05-03_updated_dfinity_icr_rack_diagram.png

Node Deployment Guide

2023-03-17T18:08:28Z

Gary.mcelroy: /* 3. Create Bootable USB Stick */

This runbook covers all steps necessary to install the Internet Computer Operating System (IC-OS) on [[Node_provider_hardware#Gen_2|Gen-2 hardware]]. Gen-1 hardware Node Providers should use the [[Internet Computer wiki#For Node Providers|Gen-1 runbooks]].

The physical machine is expected to be racked and stacked according to the respective manual.

In case you encounter any issues during the installation process, check the [[Possible Node Onboarding Errors]] page. Otherwise contact the [https://support.internetcomputer.org/hc/en-us Internet Computer Association]

Many thanks for your efforts in building the Internet Computer.

== Requirements ==
* A USB (3.0 speed that can hold at least 4GB) to put the image file on (Faster USBs will allow the process to go much faster.)
* The Nitrokey HSM for your data center.
* [Optional] A USB hub
** This is helpful at some data centers for simultaneously connecting keyboard, mouse, Nitrokey, etc..

== 1. Download installation image ==

# Download the IC-OS USB installer image here. '''This link will always provide the current blessed version. Do NOT use an old USB image.''': https://download.dfinity.systems/ic/efeb38b3bfa1133383e293a65ccce29263318ef0/setup-os/disk-img/disk-img.tar.gz 
# Download the corresponding checksum here: https://download.dfinity.systems/ic/efeb38b3bfa1133383e293a65ccce29263318ef0/setup-os/disk-img/SHA256SUMS

== 2. Verify checksum and unarchive file ==
=== Mac OS X ===
* Open the Terminal and type: <syntaxhighlight lang="shell">shasum -a 256 ~/Downloads/disk-img.tar.gz</syntaxhighlight>
* Compare the calculated checksum with the file downloaded in the previous step. '''Warning:''' Only continue if they are identical, otherwise please contact the [https://support.internetcomputer.org/hc/en-us Internet Computer Association].
* Open the Terminal and type: <syntaxhighlight lang="shell">tar xzvf ~/Downloads/disk-img.tar.gz</syntaxhighlight>

=== Linux / Ubuntu ===
* Open the Terminal and type: <syntaxhighlight lang="shell">sha256sum ~/Downloads/disk-img.tar.gz</syntaxhighlight>
* Compare the calculated checksum with the file downloaded in the previous step. '''Warning:''' Only continue if they are identical, otherwise please contact the [https://support.internetcomputer.org/hc/en-us Internet Computer Association].
* Open the Terminal and type: <syntaxhighlight lang="shell">tar xzvf ~/Downloads/disk-img.tar.gz</syntaxhighlight>

=== Windows ===
* Open PowerShell and type: <syntaxhighlight lang="shell">Get-FileHash -Algorithm SHA256 .\Downloads\disk-img.tar.gz</syntaxhighlight>
* Compare the calculated checksum with the file downloaded in the previous step. '''Warning:''' Only continue if they are identical, otherwise please contact the [https://support.internetcomputer.org/hc/en-us Internet Computer Association].
* Open PowerShell and type: <syntaxhighlight lang="shell">tar xzvf .\Downloads\disk-img.tar.gz</syntaxhighlight>

== 3. Create Bootable USB Stick ==
=== Mac OS X ===
# Open the Terminal and type:
#: <syntaxhighlight lang="shell">diskutil list</syntaxhighlight>
# All available drives should be shown. Identify which device corresponds to your USB stick. You may need to unmount the USB drive:
#:<syntaxhighlight lang="shell">sudo diskutil unmount /dev/YOUR_USB_DEVICE_MOUNTED_PARTITION # E.g. /dev/disk4s1</syntaxhighlight>
# The file path is an example. Use the absolute path to the downloaded image. '''Warning:''' You risk losing your own data if you specify a wrong device.
#:<syntaxhighlight lang="shell">sudo dd if=/Users/YOUR_USER_NAME/Downloads/disk.img of=/dev/YOUR_USB_DEVICE bs=1M</syntaxhighlight>

=== Linux / Ubuntu ===
# Open the Terminal and type <syntaxhighlight lang="shell">blkid</syntaxhighlight>
# All available drives should be shown. Identify which device corresponds to your USB stick. You may need to unmount the USB drive:
#:<syntaxhighlight lang="shell">sudo diskutil unmount /dev/YOUR_USB_DEVICE_MOUNTED_PARTITION # E.g. /dev/sdb1</syntaxhighlight>
# Replace ''/dev/YOUR_USB_DEVICE'' with the device that corresponds to your USB stick. '''Warning:''' You risk losing your own data if you specify a wrong drive.
#: <syntaxhighlight lang="shell">sudo dd if=~/Downloads/disk.img of=/dev/YOUR_USB_DEVICE bs=1M</syntaxhighlight>

=== Windows ===
# Download and install [https://rufus.ie/en/ Rufus Portable]
# Start Rufus
# Select the USB stick under device and select the previously downloaded IC-OS disk image and press start
#: [[File:05.png|480px|screenshot]]
# You may see some warnings. Make sure you don't have any other USBs in your computer and chose OK
#: [[File:06.png|480px|screenshot]]
#: [[File:07.png|480px|screenshot]]
# The "Ready" bar will go from left to right as it completes.

== 4. Add configuration ==
=== Mac OS X ===
# Open Finder. You should now be able to see the CONFIG partition. If it's not visible, remove the USB and insert it again.
#: [[File:mac_01.png|580px|screenshot]]
# Double-click to open it in TextEdit.
# Insert your IPv6 prefix, subnet and gateway.
#: [[File:mac_02.png|580px|screenshot]]
# Once done, don’t forget to save the changes. If you need help, please do not hesitate to contact the Internet Computer Association.
#: [[File:mac_03.png|580px|screenshot]]

=== Windows ===
# Open the Disk Management utility with a right click on the Start menu
#: [[File:09-b.png|300px|screenshot]]#:
# Right click the CONFIG partition
# Select Change drive letter or paths...
#: [[File:10-b.png|780px|screenshot]]
# Select any letter from the drop-down list
#: [[File:11-b.png|480px|screenshot]]
# Click OK.
# You should now be able to see the CONFIG partition in your Windows Explorer. Select the config.ini configuration file
#: [[File:12-b.png|780px|screenshot]]
# Click on Edit to open it.
# Insert your IPv6 prefix, subnet and gateway.
#: [[File:13-b.png|580px|screenshot]]
# Once done, don’t forget to save the changes. If you need help, please do not hesitate to contact the Internet Computer Association.
#: [[File:14-b.png|580px|screenshot]]

=== Linux ===
# Open the File Manager. You should now be able to see the CONFIG partition. If it's not visible, remove the USB and insert it again.
#: [[File:linux_01.png|580px|screenshot]]
# Double-click to open it in KWrite.
# Insert your IPv6 prefix, subnet and gateway.
#: [[File:linux_02.png|580px|screenshot]]
# Once done, don’t forget to save the changes. If you need help, please do not hesitate to contact the Internet Computer Association.
#: [[File:linux_03.png|580px|screenshot]]

== 5. Connect Crash Cart ==
# In order to configure the UEFI and initiate the installation of the IC-OS, please connect a crash cart to the physical machine.
# Plug-in the VGA/Video, keyboard and IC-OS USB stick
#: [[File:08.png|580px|screenshot]]

== 6. UEFI Setup and Boot Menu ==

Use the related page below to set up the BIOS/UEFI according to your hardware vendor.

* [[IC OS Installation - UEFI Configuration - Dell]]
* [[IC OS Installation - UEFI Configuration - Supermicro]]


Resume from this point when you are finished.

== 7. IC-OS Installation ==
# Please wait while the USB Installer is booting up. This process can take up to 3 minutes.
#: [[File:35-sm.png|580px|screenshot]]
# The IC-OS installation starts. Please keep an eye on the progress. This part can take up to 8 minutes. Please remember to check the [[Possible Node Onboarding Errors]] page if you encounter any errors.
#: [[File:36-sm.png|580px|screenshot]]
# Once you get asked to insert the HSM, please remove the keyboard and instead insert the HSM USB device.
#: [[File:37-sm.png|580px|screenshot]]
# If the installation finished successfully, it will initiate a reboot. '''Please do not unplug the USB stick or HSM USB''' device at this point.
#: [[File:38-sm.png|580px|screenshot]]

== 8. First Boot ==
Please remember to check the [[Possible Node Onboarding Errors]] page if you encounter any errors onboarding.
'''Do NOT re-try the onboarding after proceeding to this section, as this can cause duplication within the registry.'''
# The first boot of the IC-OS still requires the HSM USB device. Please wait until further instructions. This step can take up to 2 minutes.
#: [[File:39-sm.png|580px|screenshot]]
# Once you see this message, you may unplug the HSM USB device, USB stick and VGA/Video. Your machine successfully joined the Internet Computer.
#: [[File:40-sm.png|580px|screenshot]]

[[Node Provider Onboarding|Return to the Onboarding Document]]

Node Deployment Guide

2023-03-17T16:04:50Z

Gary.mcelroy: /* 3. Create Bootable USB Stick */

This runbook covers all steps necessary to install the Internet Computer Operating System (IC-OS) on [[Node_provider_hardware#Gen_2|Gen-2 hardware]]. Gen-1 hardware Node Providers should use the [[Internet Computer wiki#For Node Providers|Gen-1 runbooks]].

The physical machine is expected to be racked and stacked according to the respective manual.

In case you encounter any issues during the installation process, check the [[Possible Node Onboarding Errors]] page. Otherwise contact the [https://support.internetcomputer.org/hc/en-us Internet Computer Association]

Many thanks for your efforts in building the Internet Computer.

== Requirements ==
* A USB (3.0 speed that can hold at least 4GB) to put the image file on (Faster USBs will allow the process to go much faster.)
* The Nitrokey HSM for your data center.
* [Optional] A USB hub
** This is helpful at some data centers for simultaneously connecting keyboard, mouse, Nitrokey, etc..

== 1. Download installation image ==

# Download the IC-OS USB installer image here. '''This link will always provide the current blessed version. Do NOT use an old USB image.''': https://download.dfinity.systems/ic/efeb38b3bfa1133383e293a65ccce29263318ef0/setup-os/disk-img/disk-img.tar.gz 
# Download the corresponding checksum here: https://download.dfinity.systems/ic/efeb38b3bfa1133383e293a65ccce29263318ef0/setup-os/disk-img/SHA256SUMS

== 2. Verify checksum and unarchive file ==
=== Mac OS X ===
* Open the Terminal and type: <syntaxhighlight lang="shell">shasum -a 256 ~/Downloads/disk-img.tar.gz</syntaxhighlight>
* Compare the calculated checksum with the file downloaded in the previous step. '''Warning:''' Only continue if they are identical, otherwise please contact the [https://support.internetcomputer.org/hc/en-us Internet Computer Association].
* Open the Terminal and type: <syntaxhighlight lang="shell">tar xzvf ~/Downloads/disk-img.tar.gz</syntaxhighlight>

=== Linux / Ubuntu ===
* Open the Terminal and type: <syntaxhighlight lang="shell">sha256sum ~/Downloads/disk-img.tar.gz</syntaxhighlight>
* Compare the calculated checksum with the file downloaded in the previous step. '''Warning:''' Only continue if they are identical, otherwise please contact the [https://support.internetcomputer.org/hc/en-us Internet Computer Association].
* Open the Terminal and type: <syntaxhighlight lang="shell">tar xzvf ~/Downloads/disk-img.tar.gz</syntaxhighlight>

=== Windows ===
* Open PowerShell and type: <syntaxhighlight lang="shell">Get-FileHash -Algorithm SHA256 .\Downloads\disk-img.tar.gz</syntaxhighlight>
* Compare the calculated checksum with the file downloaded in the previous step. '''Warning:''' Only continue if they are identical, otherwise please contact the [https://support.internetcomputer.org/hc/en-us Internet Computer Association].
* Open PowerShell and type: <syntaxhighlight lang="shell">tar xzvf .\Downloads\disk-img.tar.gz</syntaxhighlight>

== 3. Create Bootable USB Stick ==
=== Mac OS X ===
# Open the Terminal and type:
#: <syntaxhighlight lang="shell">diskutil list</syntaxhighlight>
# All available drives should be shown. Identify which device corresponds to your USB stick.
# The following command is an example. Replace ''/dev/YOUR_USB_DEVICE'' with the device that corresponds to your USB stick. Replace ''YOUR_USER_NAME'' accordingly or with the whole path to the downloaded image. '''Warning:''' You risk losing your own data if you specify a wrong device.
#:<syntaxhighlight lang="shell">sudo dd if=/Users/YOUR_USER_NAME/Downloads/disk.img of=/dev/YOUR_USB_DEVICE bs=1M</syntaxhighlight>

=== Linux / Ubuntu ===
# Open the Terminal and type <syntaxhighlight lang="shell">blkid</syntaxhighlight>
# All available drives should be shown. Identify which device corresponds to your USB stick.
# For this next command, replace ''/dev/YOUR_USB_DEVICE'' with the device that corresponds to your USB stick. '''Warning:''' You risk losing your own data if you specify a wrong drive.
#: <syntaxhighlight lang="shell">sudo dd if=~/Downloads/disk.img of=/dev/YOUR_USB_DEVICE bs=1M</syntaxhighlight>

=== Windows ===
# Download and install [https://rufus.ie/en/ Rufus Portable]
# Start Rufus
# Select the USB stick under device and select the previously downloaded IC-OS disk image and press start
#: [[File:05.png|480px|screenshot]]
# You may see some warnings. Make sure you don't have any other USBs in your computer and chose OK
#: [[File:06.png|480px|screenshot]]
#: [[File:07.png|480px|screenshot]]
# The "Ready" bar will go from left to right as it completes.

== 4. Add configuration ==
=== Mac OS X ===
# Open Finder. You should now be able to see the CONFIG partition. If it's not visible, remove the USB and insert it again.
#: [[File:mac_01.png|580px|screenshot]]
# Double-click to open it in TextEdit.
# Insert your IPv6 prefix, subnet and gateway.
#: [[File:mac_02.png|580px|screenshot]]
# Once done, don’t forget to save the changes. If you need help, please do not hesitate to contact the Internet Computer Association.
#: [[File:mac_03.png|580px|screenshot]]

=== Windows ===
# Open the Disk Management utility with a right click on the Start menu
#: [[File:09-b.png|300px|screenshot]]#:
# Right click the CONFIG partition
# Select Change drive letter or paths...
#: [[File:10-b.png|780px|screenshot]]
# Select any letter from the drop-down list
#: [[File:11-b.png|480px|screenshot]]
# Click OK.
# You should now be able to see the CONFIG partition in your Windows Explorer. Select the config.ini configuration file
#: [[File:12-b.png|780px|screenshot]]
# Click on Edit to open it.
# Insert your IPv6 prefix, subnet and gateway.
#: [[File:13-b.png|580px|screenshot]]
# Once done, don’t forget to save the changes. If you need help, please do not hesitate to contact the Internet Computer Association.
#: [[File:14-b.png|580px|screenshot]]

=== Linux ===
# Open the File Manager. You should now be able to see the CONFIG partition. If it's not visible, remove the USB and insert it again.
#: [[File:linux_01.png|580px|screenshot]]
# Double-click to open it in KWrite.
# Insert your IPv6 prefix, subnet and gateway.
#: [[File:linux_02.png|580px|screenshot]]
# Once done, don’t forget to save the changes. If you need help, please do not hesitate to contact the Internet Computer Association.
#: [[File:linux_03.png|580px|screenshot]]

== 5. Connect Crash Cart ==
# In order to configure the UEFI and initiate the installation of the IC-OS, please connect a crash cart to the physical machine.
# Plug-in the VGA/Video, keyboard and IC-OS USB stick
#: [[File:08.png|580px|screenshot]]

== 6. UEFI Setup and Boot Menu ==

Use the related page below to set up the BIOS/UEFI according to your hardware vendor.

* [[IC OS Installation - UEFI Configuration - Dell]]
* [[IC OS Installation - UEFI Configuration - Supermicro]]


Resume from this point when you are finished.

== 7. IC-OS Installation ==
# Please wait while the USB Installer is booting up. This process can take up to 3 minutes.
#: [[File:35-sm.png|580px|screenshot]]
# The IC-OS installation starts. Please keep an eye on the progress. This part can take up to 8 minutes. Please remember to check the [[Possible Node Onboarding Errors]] page if you encounter any errors.
#: [[File:36-sm.png|580px|screenshot]]
# Once you get asked to insert the HSM, please remove the keyboard and instead insert the HSM USB device.
#: [[File:37-sm.png|580px|screenshot]]
# If the installation finished successfully, it will initiate a reboot. '''Please do not unplug the USB stick or HSM USB''' device at this point.
#: [[File:38-sm.png|580px|screenshot]]

== 8. First Boot ==
Please remember to check the [[Possible Node Onboarding Errors]] page if you encounter any errors onboarding.
'''Do NOT re-try the onboarding after proceeding to this section, as this can cause duplication within the registry.'''
# The first boot of the IC-OS still requires the HSM USB device. Please wait until further instructions. This step can take up to 2 minutes.
#: [[File:39-sm.png|580px|screenshot]]
# Once you see this message, you may unplug the HSM USB device, USB stick and VGA/Video. Your machine successfully joined the Internet Computer.
#: [[File:40-sm.png|580px|screenshot]]

[[Node Provider Onboarding|Return to the Onboarding Document]]

Node Deployment Guide

2023-03-17T16:04:37Z

Gary.mcelroy: /* 3. Create Bootable USB Stick */

This runbook covers all steps necessary to install the Internet Computer Operating System (IC-OS) on [[Node_provider_hardware#Gen_2|Gen-2 hardware]]. Gen-1 hardware Node Providers should use the [[Internet Computer wiki#For Node Providers|Gen-1 runbooks]].

The physical machine is expected to be racked and stacked according to the respective manual.

In case you encounter any issues during the installation process, check the [[Possible Node Onboarding Errors]] page. Otherwise contact the [https://support.internetcomputer.org/hc/en-us Internet Computer Association]

Many thanks for your efforts in building the Internet Computer.

== Requirements ==
* A USB (3.0 speed that can hold at least 4GB) to put the image file on (Faster USBs will allow the process to go much faster.)
* The Nitrokey HSM for your data center.
* [Optional] A USB hub
** This is helpful at some data centers for simultaneously connecting keyboard, mouse, Nitrokey, etc..

== 1. Download installation image ==

# Download the IC-OS USB installer image here. '''This link will always provide the current blessed version. Do NOT use an old USB image.''': https://download.dfinity.systems/ic/efeb38b3bfa1133383e293a65ccce29263318ef0/setup-os/disk-img/disk-img.tar.gz 
# Download the corresponding checksum here: https://download.dfinity.systems/ic/efeb38b3bfa1133383e293a65ccce29263318ef0/setup-os/disk-img/SHA256SUMS

== 2. Verify checksum and unarchive file ==
=== Mac OS X ===
* Open the Terminal and type: <syntaxhighlight lang="shell">shasum -a 256 ~/Downloads/disk-img.tar.gz</syntaxhighlight>
* Compare the calculated checksum with the file downloaded in the previous step. '''Warning:''' Only continue if they are identical, otherwise please contact the [https://support.internetcomputer.org/hc/en-us Internet Computer Association].
* Open the Terminal and type: <syntaxhighlight lang="shell">tar xzvf ~/Downloads/disk-img.tar.gz</syntaxhighlight>

=== Linux / Ubuntu ===
* Open the Terminal and type: <syntaxhighlight lang="shell">sha256sum ~/Downloads/disk-img.tar.gz</syntaxhighlight>
* Compare the calculated checksum with the file downloaded in the previous step. '''Warning:''' Only continue if they are identical, otherwise please contact the [https://support.internetcomputer.org/hc/en-us Internet Computer Association].
* Open the Terminal and type: <syntaxhighlight lang="shell">tar xzvf ~/Downloads/disk-img.tar.gz</syntaxhighlight>

=== Windows ===
* Open PowerShell and type: <syntaxhighlight lang="shell">Get-FileHash -Algorithm SHA256 .\Downloads\disk-img.tar.gz</syntaxhighlight>
* Compare the calculated checksum with the file downloaded in the previous step. '''Warning:''' Only continue if they are identical, otherwise please contact the [https://support.internetcomputer.org/hc/en-us Internet Computer Association].
* Open PowerShell and type: <syntaxhighlight lang="shell">tar xzvf .\Downloads\disk-img.tar.gz</syntaxhighlight>

== 3. Create Bootable USB Stick ==
=== Mac OS X ===
# Open the Terminal and type:
#: <syntaxhighlight lang="shell">diskutil list</syntaxhighlight>
# All available drives should be shown. Identify which device corresponds to your USB stick.
# The following command is an example. Replace ''/dev/YOUR_USB_DEVICE'' with the device that corresponds to your USB stick. Replace ''YOUR_USER_NAME'' accordingly or with the path to the downloaded image. '''Warning:''' You risk losing your own data if you specify a wrong device.
#:<syntaxhighlight lang="shell">sudo dd if=/Users/YOUR_USER_NAME/Downloads/disk.img of=/dev/YOUR_USB_DEVICE bs=1M</syntaxhighlight>

=== Linux / Ubuntu ===
# Open the Terminal and type <syntaxhighlight lang="shell">blkid</syntaxhighlight>
# All available drives should be shown. Identify which device corresponds to your USB stick.
# For this next command, replace ''/dev/YOUR_USB_DEVICE'' with the device that corresponds to your USB stick. '''Warning:''' You risk losing your own data if you specify a wrong drive.
#: <syntaxhighlight lang="shell">sudo dd if=~/Downloads/disk.img of=/dev/YOUR_USB_DEVICE bs=1M</syntaxhighlight>

=== Windows ===
# Download and install [https://rufus.ie/en/ Rufus Portable]
# Start Rufus
# Select the USB stick under device and select the previously downloaded IC-OS disk image and press start
#: [[File:05.png|480px|screenshot]]
# You may see some warnings. Make sure you don't have any other USBs in your computer and chose OK
#: [[File:06.png|480px|screenshot]]
#: [[File:07.png|480px|screenshot]]
# The "Ready" bar will go from left to right as it completes.

== 4. Add configuration ==
=== Mac OS X ===
# Open Finder. You should now be able to see the CONFIG partition. If it's not visible, remove the USB and insert it again.
#: [[File:mac_01.png|580px|screenshot]]
# Double-click to open it in TextEdit.
# Insert your IPv6 prefix, subnet and gateway.
#: [[File:mac_02.png|580px|screenshot]]
# Once done, don’t forget to save the changes. If you need help, please do not hesitate to contact the Internet Computer Association.
#: [[File:mac_03.png|580px|screenshot]]

=== Windows ===
# Open the Disk Management utility with a right click on the Start menu
#: [[File:09-b.png|300px|screenshot]]#:
# Right click the CONFIG partition
# Select Change drive letter or paths...
#: [[File:10-b.png|780px|screenshot]]
# Select any letter from the drop-down list
#: [[File:11-b.png|480px|screenshot]]
# Click OK.
# You should now be able to see the CONFIG partition in your Windows Explorer. Select the config.ini configuration file
#: [[File:12-b.png|780px|screenshot]]
# Click on Edit to open it.
# Insert your IPv6 prefix, subnet and gateway.
#: [[File:13-b.png|580px|screenshot]]
# Once done, don’t forget to save the changes. If you need help, please do not hesitate to contact the Internet Computer Association.
#: [[File:14-b.png|580px|screenshot]]

=== Linux ===
# Open the File Manager. You should now be able to see the CONFIG partition. If it's not visible, remove the USB and insert it again.
#: [[File:linux_01.png|580px|screenshot]]
# Double-click to open it in KWrite.
# Insert your IPv6 prefix, subnet and gateway.
#: [[File:linux_02.png|580px|screenshot]]
# Once done, don’t forget to save the changes. If you need help, please do not hesitate to contact the Internet Computer Association.
#: [[File:linux_03.png|580px|screenshot]]

== 5. Connect Crash Cart ==
# In order to configure the UEFI and initiate the installation of the IC-OS, please connect a crash cart to the physical machine.
# Plug-in the VGA/Video, keyboard and IC-OS USB stick
#: [[File:08.png|580px|screenshot]]

== 6. UEFI Setup and Boot Menu ==

Use the related page below to set up the BIOS/UEFI according to your hardware vendor.

* [[IC OS Installation - UEFI Configuration - Dell]]
* [[IC OS Installation - UEFI Configuration - Supermicro]]


Resume from this point when you are finished.

== 7. IC-OS Installation ==
# Please wait while the USB Installer is booting up. This process can take up to 3 minutes.
#: [[File:35-sm.png|580px|screenshot]]
# The IC-OS installation starts. Please keep an eye on the progress. This part can take up to 8 minutes. Please remember to check the [[Possible Node Onboarding Errors]] page if you encounter any errors.
#: [[File:36-sm.png|580px|screenshot]]
# Once you get asked to insert the HSM, please remove the keyboard and instead insert the HSM USB device.
#: [[File:37-sm.png|580px|screenshot]]
# If the installation finished successfully, it will initiate a reboot. '''Please do not unplug the USB stick or HSM USB''' device at this point.
#: [[File:38-sm.png|580px|screenshot]]

== 8. First Boot ==
Please remember to check the [[Possible Node Onboarding Errors]] page if you encounter any errors onboarding.
'''Do NOT re-try the onboarding after proceeding to this section, as this can cause duplication within the registry.'''
# The first boot of the IC-OS still requires the HSM USB device. Please wait until further instructions. This step can take up to 2 minutes.
#: [[File:39-sm.png|580px|screenshot]]
# Once you see this message, you may unplug the HSM USB device, USB stick and VGA/Video. Your machine successfully joined the Internet Computer.
#: [[File:40-sm.png|580px|screenshot]]

[[Node Provider Onboarding|Return to the Onboarding Document]]

Node Deployment Guide

2023-03-17T16:04:18Z

Gary.mcelroy: /* 3. Create Bootable USB Stick */

This runbook covers all steps necessary to install the Internet Computer Operating System (IC-OS) on [[Node_provider_hardware#Gen_2|Gen-2 hardware]]. Gen-1 hardware Node Providers should use the [[Internet Computer wiki#For Node Providers|Gen-1 runbooks]].

The physical machine is expected to be racked and stacked according to the respective manual.

In case you encounter any issues during the installation process, check the [[Possible Node Onboarding Errors]] page. Otherwise contact the [https://support.internetcomputer.org/hc/en-us Internet Computer Association]

Many thanks for your efforts in building the Internet Computer.

== Requirements ==
* A USB (3.0 speed that can hold at least 4GB) to put the image file on (Faster USBs will allow the process to go much faster.)
* The Nitrokey HSM for your data center.
* [Optional] A USB hub
** This is helpful at some data centers for simultaneously connecting keyboard, mouse, Nitrokey, etc..

== 1. Download installation image ==

# Download the IC-OS USB installer image here. '''This link will always provide the current blessed version. Do NOT use an old USB image.''': https://download.dfinity.systems/ic/efeb38b3bfa1133383e293a65ccce29263318ef0/setup-os/disk-img/disk-img.tar.gz 
# Download the corresponding checksum here: https://download.dfinity.systems/ic/efeb38b3bfa1133383e293a65ccce29263318ef0/setup-os/disk-img/SHA256SUMS

== 2. Verify checksum and unarchive file ==
=== Mac OS X ===
* Open the Terminal and type: <syntaxhighlight lang="shell">shasum -a 256 ~/Downloads/disk-img.tar.gz</syntaxhighlight>
* Compare the calculated checksum with the file downloaded in the previous step. '''Warning:''' Only continue if they are identical, otherwise please contact the [https://support.internetcomputer.org/hc/en-us Internet Computer Association].
* Open the Terminal and type: <syntaxhighlight lang="shell">tar xzvf ~/Downloads/disk-img.tar.gz</syntaxhighlight>

=== Linux / Ubuntu ===
* Open the Terminal and type: <syntaxhighlight lang="shell">sha256sum ~/Downloads/disk-img.tar.gz</syntaxhighlight>
* Compare the calculated checksum with the file downloaded in the previous step. '''Warning:''' Only continue if they are identical, otherwise please contact the [https://support.internetcomputer.org/hc/en-us Internet Computer Association].
* Open the Terminal and type: <syntaxhighlight lang="shell">tar xzvf ~/Downloads/disk-img.tar.gz</syntaxhighlight>

=== Windows ===
* Open PowerShell and type: <syntaxhighlight lang="shell">Get-FileHash -Algorithm SHA256 .\Downloads\disk-img.tar.gz</syntaxhighlight>
* Compare the calculated checksum with the file downloaded in the previous step. '''Warning:''' Only continue if they are identical, otherwise please contact the [https://support.internetcomputer.org/hc/en-us Internet Computer Association].
* Open PowerShell and type: <syntaxhighlight lang="shell">tar xzvf .\Downloads\disk-img.tar.gz</syntaxhighlight>

== 3. Create Bootable USB Stick ==
=== Mac OS X ===
# Open the Terminal and type:
#: <syntaxhighlight lang="shell">diskutil list</syntaxhighlight>
# All available drives should be shown. Identify which device corresponds to your USB stick.
# The following command is an example. Replace ''/dev/YOUR_USB_DEVICE'' with the device that corresponds to your USB stick. Replace ''YOUR_USER_NAME'' accordingly or with the path you downloaded the image to. '''Warning:''' You risk losing your own data if you specify a wrong device.
#:<syntaxhighlight lang="shell">sudo dd if=/Users/YOUR_USER_NAME/Downloads/disk.img of=/dev/YOUR_USB_DEVICE bs=1M</syntaxhighlight>

=== Linux / Ubuntu ===
# Open the Terminal and type <syntaxhighlight lang="shell">blkid</syntaxhighlight>
# All available drives should be shown. Identify which device corresponds to your USB stick.
# For this next command, replace ''/dev/YOUR_USB_DEVICE'' with the device that corresponds to your USB stick. '''Warning:''' You risk losing your own data if you specify a wrong drive.
#: <syntaxhighlight lang="shell">sudo dd if=~/Downloads/disk.img of=/dev/YOUR_USB_DEVICE bs=1M</syntaxhighlight>

=== Windows ===
# Download and install [https://rufus.ie/en/ Rufus Portable]
# Start Rufus
# Select the USB stick under device and select the previously downloaded IC-OS disk image and press start
#: [[File:05.png|480px|screenshot]]
# You may see some warnings. Make sure you don't have any other USBs in your computer and chose OK
#: [[File:06.png|480px|screenshot]]
#: [[File:07.png|480px|screenshot]]
# The "Ready" bar will go from left to right as it completes.

== 4. Add configuration ==
=== Mac OS X ===
# Open Finder. You should now be able to see the CONFIG partition. If it's not visible, remove the USB and insert it again.
#: [[File:mac_01.png|580px|screenshot]]
# Double-click to open it in TextEdit.
# Insert your IPv6 prefix, subnet and gateway.
#: [[File:mac_02.png|580px|screenshot]]
# Once done, don’t forget to save the changes. If you need help, please do not hesitate to contact the Internet Computer Association.
#: [[File:mac_03.png|580px|screenshot]]

=== Windows ===
# Open the Disk Management utility with a right click on the Start menu
#: [[File:09-b.png|300px|screenshot]]#:
# Right click the CONFIG partition
# Select Change drive letter or paths...
#: [[File:10-b.png|780px|screenshot]]
# Select any letter from the drop-down list
#: [[File:11-b.png|480px|screenshot]]
# Click OK.
# You should now be able to see the CONFIG partition in your Windows Explorer. Select the config.ini configuration file
#: [[File:12-b.png|780px|screenshot]]
# Click on Edit to open it.
# Insert your IPv6 prefix, subnet and gateway.
#: [[File:13-b.png|580px|screenshot]]
# Once done, don’t forget to save the changes. If you need help, please do not hesitate to contact the Internet Computer Association.
#: [[File:14-b.png|580px|screenshot]]

=== Linux ===
# Open the File Manager. You should now be able to see the CONFIG partition. If it's not visible, remove the USB and insert it again.
#: [[File:linux_01.png|580px|screenshot]]
# Double-click to open it in KWrite.
# Insert your IPv6 prefix, subnet and gateway.
#: [[File:linux_02.png|580px|screenshot]]
# Once done, don’t forget to save the changes. If you need help, please do not hesitate to contact the Internet Computer Association.
#: [[File:linux_03.png|580px|screenshot]]

== 5. Connect Crash Cart ==
# In order to configure the UEFI and initiate the installation of the IC-OS, please connect a crash cart to the physical machine.
# Plug-in the VGA/Video, keyboard and IC-OS USB stick
#: [[File:08.png|580px|screenshot]]

== 6. UEFI Setup and Boot Menu ==

Use the related page below to set up the BIOS/UEFI according to your hardware vendor.

* [[IC OS Installation - UEFI Configuration - Dell]]
* [[IC OS Installation - UEFI Configuration - Supermicro]]


Resume from this point when you are finished.

== 7. IC-OS Installation ==
# Please wait while the USB Installer is booting up. This process can take up to 3 minutes.
#: [[File:35-sm.png|580px|screenshot]]
# The IC-OS installation starts. Please keep an eye on the progress. This part can take up to 8 minutes. Please remember to check the [[Possible Node Onboarding Errors]] page if you encounter any errors.
#: [[File:36-sm.png|580px|screenshot]]
# Once you get asked to insert the HSM, please remove the keyboard and instead insert the HSM USB device.
#: [[File:37-sm.png|580px|screenshot]]
# If the installation finished successfully, it will initiate a reboot. '''Please do not unplug the USB stick or HSM USB''' device at this point.
#: [[File:38-sm.png|580px|screenshot]]

== 8. First Boot ==
Please remember to check the [[Possible Node Onboarding Errors]] page if you encounter any errors onboarding.
'''Do NOT re-try the onboarding after proceeding to this section, as this can cause duplication within the registry.'''
# The first boot of the IC-OS still requires the HSM USB device. Please wait until further instructions. This step can take up to 2 minutes.
#: [[File:39-sm.png|580px|screenshot]]
# Once you see this message, you may unplug the HSM USB device, USB stick and VGA/Video. Your machine successfully joined the Internet Computer.
#: [[File:40-sm.png|580px|screenshot]]

[[Node Provider Onboarding|Return to the Onboarding Document]]