Difference between revisions of "Node Provider Maintenance Guide"
Katie.peters (talk | contribs) (Adding Best Practices links) |
Katie.peters (talk | contribs) |
||
| Line 28: | Line 28: | ||
* [[RMU build Gen-1.5: IV. Best Practices - A. Proxmox Security|Proxmox Security]] | * [[RMU build Gen-1.5: IV. Best Practices - A. Proxmox Security|Proxmox Security]] | ||
| − | * [[Install RACADM Tool | + | * [[RMU build Gen-1.5: IV. Best Practices - B. Install RACADM Tool on RMU|Install RACADM Tool on RMU]] |
| − | * [[Dell OpenManage Enterprise (OME) (Dell Nodes)]] | + | * [[RMU build Gen-1.5: IV. Best Practices - C. Dell OpenManage Enterprise|Dell OpenManage Enterprise (OME) (Dell Nodes)]] |
** This includes instructions for [https://wiki.internetcomputer.org/wiki/RMU_build_Gen-1.5:_IV._Best_Practices_-_C._Dell_OpenManage_Enterprise?veaction=edit§ion=9#Applying_Firmware_Updates updating firmware] | ** This includes instructions for [https://wiki.internetcomputer.org/wiki/RMU_build_Gen-1.5:_IV._Best_Practices_-_C._Dell_OpenManage_Enterprise?veaction=edit§ion=9#Applying_Firmware_Updates updating firmware] | ||
* [[RMU build Gen-1.5: IV. Best Practices - E. Enable Remote Access to Servers Console|Enable Remote Access to Servers Console]] | * [[RMU build Gen-1.5: IV. Best Practices - E. Enable Remote Access to Servers Console|Enable Remote Access to Servers Console]] | ||
| − | |||
== Node rewards based on useful work == | == Node rewards based on useful work == | ||
Revision as of 17:56, 1 March 2024
Submitting NNS proposals
As a part of being a Node Provider, you will likely have to submit some NNS proposals. The page at the following link describes some of these proposals: Node Provider NNS proposals
Monitoring
You are expected to regularly monitor the health of your nodes. Node health status is available on the public dashboard. Example: node status.
Also, check out the Tools and Resources section below, for some useful tools that can help you with the monitoring and alerting activites.
Permitted tools
For security and confidentiality reasons, other tools are not allowed to run on the same machine in parallel with the replica. In case you need to troubleshoot an issue, it is recommended to either boot the machine from a USB drive that has a live Linux distribution (e.g. Ubuntu) or to debug from an auxiliary machine in the same rack on which you have complete control, as described in Unhealthy Nodes#Setting Up an Auxiliary Machine for Network Diagnostics
Scheduled DC outages
When your DC notifies you of a scheduled outage, you must:
- Notify DFINITY on the Node Provider Matrix channel
- Make sure your nodes return to one of the healthy statuses when the DC outage is resolved:
- Active in Subnet - The node is healthy and actively functioning within a subnet.
- Awaiting Subnet - The node is operational and prepared to join a subnet when necessary.
- If a node is degraded at first, give it a little bit of time in case it needs to catch up, but make sure that it does return to one of the two healthy statuses.
Handling degraded nodes
Please take a look at Node Provider Troubleshooting
Handling dead nodes
Please take a look at Node Provider Troubleshooting
Best Practices
- Proxmox Security
- Install RACADM Tool on RMU
- Dell OpenManage Enterprise (OME) (Dell Nodes)
- This includes instructions for updating firmware
- Enable Remote Access to Servers Console
Node rewards based on useful work
The Internet Computer protocol can tolerate up to 1/3 of nodes misbehaving. There is an ongoing activity to automatically issue node rewards based on useful work, and also to automatically reduce node remuneration in case nodes are misbehaving. This will provide a financial incentive for honest behavior. Please follow the forum and the Matrix channel to stay informed about these activities.
In the meantime, the recommendation is to prepare for this by making sure that your nodes are online and healthy at all times, otherwise you risk penalties even before the automatic node rewards based on useful work become active.
Subnet recovery
In case subnet recovery is needed, we may have to reach out to you for assistance. Please make sure you closely follow activities in the Matrix Channel, and enable notifications on new messages -- especially direct mentions.
Peer-support and bug reports / resolution: Node Provider Matrix Channel
Node Providers are encouraged to join the dedicated Node Provider Matrix channel. This platform can be used for discussing maintenance-related queries and sharing insights, report issues, and search for previous resolutions for operations.
Communication Guidelines on the Matrix Channel
As a Node Provider, ensure your notifications are enabled to receive new messages promptly. Your input or intervention might be crucial, especially in urgent situations.
It is recommended to add the node provider name to your alias (handle) on the communication platform, to facilitate communication and enable others to quickly and easily mention you.
Tools and Resources
Several node providers have generously shared tools to facilitate monitoring node health. These tools can provide notifications in case of node issues.
Aviate Labs Node Monitor
- Turnkey Solution: Receive email alerts for unhealthy nodes.
- Link: AviateLabs Node Monitor
DIY Node Monitoring
- GitHub Repository: Run your own node monitoring system.
- Link: Aviate Labs GitHub
Prometheus Exporter for Node Status
- GitHub Repository: A tool for exporting node status to a Prometheus-compatible format.
- Link: IC Node Status Prometheus Exporter
Additional Notes
- Screenshots: Include screenshots of the node status from the public dashboard for reference and troubleshooting.
In case you observe issues, follow: Unhealthy Nodes and Node Provider Troubleshooting