Difference between revisions of "Node Provider Maintenance Guide"

Latest revision as of 19:00, 1 March 2024

Submitting NNS proposals

As a part of being a Node Provider, you will likely have to submit some NNS proposals. The page at the following link describes some of these proposals: Node Provider NNS proposals

Monitoring

You are expected to regularly monitor the health of your nodes. Node health status is available on the public dashboard. Example: node status.

Also, check out the Tools and Resources section below, for some useful tools that can help you with the monitoring and alerting activites.

Permitted tools

For security and confidentiality reasons, other tools are not allowed to run on the same machine in parallel with the replica. In case you need to troubleshoot an issue, it is recommended to either boot the machine from a USB drive that has a live Linux distribution (e.g. Ubuntu) or to debug from an auxiliary machine in the same rack on which you have complete control, as described in Unhealthy Nodes#Setting Up an Auxiliary Machine for Network Diagnostics

Scheduled DC outages

When your DC notifies you of a scheduled outage, you must:

Notify DFINITY on the Node Provider Matrix channel
Make sure your nodes return to one of the healthy statuses when the DC outage is resolved:
- Active in Subnet - The node is healthy and actively functioning within a subnet.
- Awaiting Subnet - The node is operational and prepared to join a subnet when necessary.
If a node is degraded at first, give it a little bit of time in case it needs to catch up, but make sure that it does return to one of the two healthy statuses.

Handling degraded nodes

Please take a look at Node Provider Troubleshooting

Handling dead nodes

Please take a look at Node Provider Troubleshooting

Node rewards based on useful work

The Internet Computer protocol can tolerate up to 1/3 of nodes misbehaving. There is an ongoing activity to automatically issue node rewards based on useful work, and also to automatically reduce node remuneration in case nodes are misbehaving. This will provide a financial incentive for honest behavior. Please follow the forum and the Matrix channel to stay informed about these activities.

In the meantime, the recommendation is to prepare for this by making sure that your nodes are online and healthy at all times, otherwise you risk penalties even before the automatic node rewards based on useful work become active.

Subnet recovery

In case subnet recovery is needed, we may have to reach out to you for assistance. Please make sure you closely follow activities in the Matrix Channel, and enable notifications on new messages -- especially direct mentions.

Peer-support and bug reports / resolution: Node Provider Matrix Channel

Node Providers are encouraged to join the dedicated Node Provider Matrix channel. This platform can be used for discussing maintenance-related queries and sharing insights, report issues, and search for previous resolutions for operations.

Communication Guidelines on the Matrix Channel

As a Node Provider, ensure your notifications are enabled to receive new messages promptly. Your input or intervention might be crucial, especially in urgent situations.

It is recommended to add the node provider name to your alias (handle) on the communication platform, to facilitate communication and enable others to quickly and easily mention you.

Tools and Resources

Several node providers have generously shared tools to facilitate monitoring node health. These tools can provide notifications in case of node issues.

Aviate Labs Node Monitor

Turnkey Solution: Receive email alerts for unhealthy nodes.
Link: AviateLabs Node Monitor

DIY Node Monitoring

GitHub Repository: Run your own node monitoring system.
Link: Aviate Labs GitHub

Prometheus Exporter for Node Status

GitHub Repository: A tool for exporting node status to a Prometheus-compatible format.
Link: IC Node Status Prometheus Exporter

Additional Notes

Screenshots: Include screenshots of the node status from the public dashboard for reference and troubleshooting.

In case you observe issues, follow: Unhealthy Nodes and Node Provider Troubleshooting

@@ Line 1: / Line 1: @@
 == Submitting NNS proposals ==
-Here are some NNS proposals you may have to submit after onboarding nodes.
+As a part of being a Node Provider, you will likely have to submit some NNS proposals. The page at the following link describes some of these proposals: [[Node Provider NNS proposals]]
-=== Adjusting the node allowance in a Data Center ===
+== Monitoring ==
-To adjust the node allowance for an existing node operator record, you would need to use the <code>propose-to-update-node-operator-config</code> subcommand of the <code>ic-admin</code> tool. Here's a step-by-step guide on how to do this:
+You are expected to regularly monitor the health of your nodes. Node health status is available on the public dashboard. Example: [https://dashboard.internetcomputer.org/node/b5d56-nm7ae-p24jg-t25gp-5bmhb-rjbnt-3dmoq-goqby-5tf6c-ygnnu-aqe node status].
-.  '''Gather Necessary Information:''' Ensure you have the following details:
+Also, check out the Tools and Resources section below, for some useful tools that can help you with the monitoring and alerting activites.
-* <code>NODE_OPERATOR_ID</code>: The principal ID of the node operator whose allowance you want to change.
+== Permitted tools ==
-* <code>NODE_ALLOWANCE</code>: The new number of nodes that the node operator is allowed to add.
+For security and confidentiality reasons, other tools are not allowed to run on the same machine in parallel with the replica. In case you need to troubleshoot an issue, it is recommended to either boot the machine from a USB drive that has a live Linux distribution (e.g. [https://ubuntu.com/tutorials/try-ubuntu-before-you-install#3-boot-from-usb-flash-drive Ubuntu]) or to debug from an auxiliary machine in the same rack on which you have complete control, as described in [[Unhealthy Nodes#Setting Up an Auxiliary Machine for Network Diagnostics]]
-* <code>NEURON_ID</code>: The ID of the neuron that will propose this change.
-* <code>DC_ID</code> data center ID where nodes should be added.
-. '''Prepare the Command''': Construct the <code>ic-admin</code> command using the gathered information. Here's an example template:
+== Scheduled DC outages ==
- $ ./ic-admin \
+When your DC notifies you of a scheduled outage, you must:
-           --nns-url <nowiki>https://ic0.app</nowiki> \
-           propose-to-update-node-operator-config \
-           --node-operator-id $NODE_OPERATOR_ID \
-           --proposer $NEURON_ID \
-           --node-allowance $NEW_NODE_ALLOWANCE \
-           [Other Options as Required]
-Replace <code>$NODE_OPERATOR_ID</code>, <code>$NEURON_ID</code>, and <code>$NEW_NODE_ALLOWANCE</code> with the actual values. Include additional options as needed, such as `--dc-id` if you're specifying a particular data center.
-.  '''Summary and Proposal Details''': Add a summary and any other proposal details. This could include the reason for the allowance change, expected impacts, etc. These can be added using `--summary` or `--summary-file` options.
+* Notify DFINITY on the [[Node Provider Matrix channel]]
+* Make sure your nodes return to one of the healthy statuses when the DC outage is resolved:
+** Active in Subnet - The node is healthy and actively functioning within a subnet.
+** Awaiting Subnet - The node is operational and prepared to join a subnet when necessary.
+* If a node is degraded at first, give it a little bit of time in case it needs to catch up, but make sure that it does return to one of the two healthy statuses.
-.  '''Dry Run (recommended)''': If you want to preview the proposal without actually submitting it, you can add the `--dry-run` flag to the command. This is useful for checking the proposal payload and ensuring everything is correct before the actual submission.
+== Handling degraded nodes ==
+Please take a look at [[Node Provider Troubleshooting]]
-.  '''Execute the Command''': Once you are sure about the command and the details, execute it in your terminal. This will submit a proposal to update the node operator's configuration, including the node allowance.
+== Handling dead nodes ==
+Please take a look at [[Node Provider Troubleshooting]]
-.  '''Monitor and Voting''': After submitting the proposal, it will typically go through a voting process by the governance system. You should monitor this to see if the proposal gets accepted or rejected.
+== Node rewards based on useful work ==
+The Internet Computer protocol can tolerate up to 1/3 of nodes misbehaving. There is an ongoing activity to automatically issue node rewards based on useful work, and also to automatically reduce node remuneration in case nodes are misbehaving. This will provide a financial incentive for honest behavior. Please follow the forum and the Matrix channel to stay informed about these activities.
-.  '''Verification (Post-Approval)''': If the proposal is approved, you may want to verify that the node allowance has been updated as expected. This might involve querying the node operator's record or using other `ic-admin` subcommands for confirmation.
+In the meantime, the recommendation is to prepare for this by making sure that your nodes are online and healthy at all times, otherwise you risk penalties even before the automatic node rewards based on useful work become active.
+== Subnet recovery ==
+In case subnet recovery is needed, we may have to reach out to you for assistance. Please make sure you closely follow activities in the Matrix Channel, and enable notifications on new messages -- especially direct mentions.
-Note that the exact command and options will vary based on your specific configuration and requirements. Make sure to replace placeholders with actual values relevant to your setup.
+== Peer-support and bug reports / resolution: Node Provider Matrix Channel ==
-To see all available options, you can run:
+Node Providers are encouraged to join the dedicated [[Node Provider Matrix channel]]. This platform can be used for discussing maintenance-related queries and sharing insights, report issues, and search for previous resolutions for operations.
- $ ic-admin --nns-url <nowiki>https://ic0.app</nowiki> propose-to-update-node-operator-config --help
-== Joining the Node Provider Matrix Channel ==
+'''Communication Guidelines on the Matrix Channel'''
-Node Providers are encouraged to join the dedicated [[Node Provider Matrix channel]]. This platform is essential for discussing maintenance-related queries and sharing insights about node operations.
+As a Node Provider, ensure your notifications are enabled to receive new messages promptly. Your input or intervention might be crucial, especially in urgent situations.
-=== Communication Guidelines ===
+It is recommended to add the node provider name to your alias (handle) on the communication platform, to facilitate communication and enable others to quickly and easily mention you.
-* '''Active Participation''': Ensure your notifications are enabled to receive new messages promptly. Your input or intervention might be crucial, especially in urgent situations.
-* '''Regular Operations''': Regularly monitor the health of your node. Node health status is available on the public dashboard, which. Example: [https://dashboard.internetcomputer.org/node/b5d56-nm7ae-p24jg-t25gp-5bmhb-rjbnt-3dmoq-goqby-5tf6c-ygnnu-aqe node status].
 == Tools and Resources ==