Node Provider Maintenance Guide

2023-11-19T14:30:57Z

Jueming: NODE_OPERATOR_ID is changed to NODE_OPERATOR_PRINCIPAL

== Submitting NNS proposals ==
Here are some NNS proposals you may have to submit after onboarding nodes.

=== Adjusting the node allowance in a Data Center ===
To adjust the node allowance for an existing node operator record, you would need to use the <code>propose-to-update-node-operator-config</code> subcommand of the <code>ic-admin</code> tool. You should typically not add a new node operator record if you just want to add more nodes to the ''existing'' DC.

Here's a step-by-step guide on how to do this:

1. '''Gather Necessary Information:''' Ensure you have the following details:

#<code>NODE_PROVIDER_ID</code>: The principal ID of the node provider under which the node operator record is registered.
# <code>NODE_OPERATOR_ID</code>: The principal ID of the node operator whose allowance you want to change.
# <code>NEURON_ID</code>: The ID of the neuron that will propose this change.
# <code>CURRENTLY_REMAINING_NODE_ALLOWANCE</code>: The number of nodes that the node operator is allowed to add to the network without submitting a proposal.
#<code>NEW_NODE_ALLOWANCE</code>: The new number of nodes that the node operator is allowed to add.
The parts 1, 2, and 3 should be in your records, and should be the same principals (IDs) used to onboard nodes ''in the given DC''. The part 4 can be obtained from the registry, with <code>ic-admin</code>
$ ic-admin --nns-url <nowiki>https://ic0.app</nowiki> get-node-operator $NODE_OPERATOR_ID
For example:
$ ic-admin --nns-url <nowiki>https://ic0.app</nowiki> get-node-operator yl63e-n74ks-fnefm-einyj-kwqot-7nkim-g5rq4-ctn3h-3ee6h-24fe4-uqe
Fetching the most recent value for key: node_operator_record_yl63e-n74ks-fnefm-einyj-kwqot-7nkim-g5rq4-ctn3h-3ee6h-24fe4-uqe
Most recent version is 35791. Value:
NodeOperator { node_operator_principal_id: yl63e-n74ks-fnefm-einyj-kwqot-7nkim-g5rq4-ctn3h-3ee6h-24fe4-uqe, node_allowance: 0, node_provider_principal_id: niw4y-easue-l3qvz-sozsi-tfkvb-cxcx6-pzslg-5dqld-ooudp-hsuui-xae, dc_id: "mu1", rewardable_nodes: {"type0": 0, "type1": 28}, ipv6: None }
In the above example, the <code>CURRENTLY_REMAINING_NODE_ALLOWANCE</code> is 0. So if you want to add 5 more nodes with the same node operator (i.e. in the same DC), you should use <code>NEW_NODE_ALLOWANCE=5</code>. However, if the <code>CURRENTLY_REMAINING_NODE_ALLOWANCE</code> had value 2, you would only need 3 more nodes on top of your currently remaining allowance (2+3=5), so you should use <code>NEW_NODE_ALLOWANCE=3</code> in the proposal

2. '''Prepare the Command''': Construct the <code>ic-admin</code> command using the gathered information. Here's an example template:
$ NEURON_ID=XXXXXXXXXXXXXXXXXXXX
$ NODE_PROVIDER_PRINCIPAL=xxxxx-xxxxx-xxxxx-xxxxx-xxxxx-xxxxx-xxxxx-xxxxx-xxxxx-xxxxx-xxx
$ NODE_OPERATOR_PRINCIPAL=xxxxx-xxxxx-xxxxx-xxxxx-xxxxx-xxxxx-xxxxx-xxxxx-xxxxx-xxxxx-xxx
$ NODE_PROVIDER_NAME="My Company"
$ NEW_NODE_ALLOWANCE=5
$ FORUM_POST_URL=[https://forum.dfinity.org/ https://forum.dfinity.org/...]

$ ./ic-admin \
--nns-url <nowiki>https://ic0.app</nowiki> \
-s ~/.config/dfx/identity/node-provider-hotkey/identity.pem \
propose-to-update-node-operator-config \
--node-provider-id $NODE_PROVIDER_PRINCIPAL \
--node-operator-id $NODE_OPERATOR_PRINCIPAL \
--summary "Node provider '$NODE_PROVIDER_NAME' is adjusting the node allowance $NODE_ALLOWANCE to nodes in the $DC_ID data center. Link to the forum post for: $FORUM_POST_URL" \
--proposer $NEURON_ID \
$NEW_NODE_ALLOWANCE
Replace all placeholder variables above with the actual values before submitting the proposal.

3. '''Dry Run (strongly recommended)''': To preview the proposal without actually submitting it, you can add the <code>--dry-run</code> flag to the above command. This is useful for checking the proposal payload and ensuring everything is correct before the actual submission.

4. '''Execute the Command''': Once you are sure about the command and the details, execute it in your terminal. This will submit a proposal to update the node allowance in the node operator's configuration.

5. '''Monitor and Voting''': After submitting the proposal, it will go through a voting process by the governance system. You should monitor this to see if the proposal gets accepted or rejected.

7. '''Verification (Post-Approval)''': If the proposal is approved, you may want to verify that the node allowance has been updated as expected. This might involve querying the node operator's record with <code>get-node-operator</code> as described above.

Note that the exact command and options will vary based on your specific configuration and requirements. Make sure to replace placeholders with actual values relevant to your setup.

To see all available options, you can run:
$ ic-admin --nns-url <nowiki>https://ic0.app</nowiki> propose-to-update-node-operator-config --help

== Joining the Node Provider Matrix Channel ==

Node Providers are encouraged to join the dedicated [[Node Provider Matrix channel]]. This platform is essential for discussing maintenance-related queries and sharing insights about node operations.

=== Communication Guidelines ===

* '''Active Participation''': Ensure your notifications are enabled to receive new messages promptly. Your input or intervention might be crucial, especially in urgent situations.
* '''Regular Operations''': Regularly monitor the health of your node. Node health status is available on the public dashboard, which. Example: [https://dashboard.internetcomputer.org/node/b5d56-nm7ae-p24jg-t25gp-5bmhb-rjbnt-3dmoq-goqby-5tf6c-ygnnu-aqe node status].

== Tools and Resources ==

Several node providers have generously shared tools to facilitate monitoring node health. These tools can provide notifications in case of node issues.

=== Aviate Labs Node Monitor ===

* '''Turnkey Solution''': Receive email alerts for unhealthy nodes.
* '''Link''': [https://www.aviatelabs.co/node-monitor AviateLabs Node Monitor]

=== DIY Node Monitoring ===

* '''GitHub Repository''': Run your own node monitoring system.
* '''Link''': [https://github.com/aviate-labs/node-monitor Aviate Labs GitHub]

=== Prometheus Exporter for Node Status ===

* '''GitHub Repository''': A tool for exporting node status to a Prometheus-compatible format.
* '''Link''': [https://github.com/virtualhive/ic-node-status-prometheus-exporter IC Node Status Prometheus Exporter]

== Additional Notes ==

* '''Screenshots''': Include screenshots of the node status from the public dashboard for reference and troubleshooting.
In case you observe issues, follow: [[Unhealthy Nodes]] and [[Node Provider Troubleshooting]]

Internet Computer Wiki - User contributions [en]

Node Provider Maintenance Guide