Difference between revisions of "Troubleshooting Unhealthy Nodes"
From Internet Computer Wiki
Katie.peters (talk | contribs) (Added the last "Use the dashboard" section because NPs aren't doing this. :() |
Katie.peters (talk | contribs) m (Added last sentence) |
||
| Line 19: | Line 19: | ||
* Look for the principal ID for the node which you are servicing and make sure that it is "Awaiting Subnet" status | * Look for the principal ID for the node which you are servicing and make sure that it is "Awaiting Subnet" status | ||
* If the node isn't listed at all, then it needs to be [[IC OS Installation Runbook - Dell Poweredge|redeployed the node with a fresh IC-OS image]]. | * If the node isn't listed at all, then it needs to be [[IC OS Installation Runbook - Dell Poweredge|redeployed the node with a fresh IC-OS image]]. | ||
| + | |||
| + | |||
| + | '''If a node is healthy ("Awaiting Subnet" status) for a while and then changes to "Offline" then whatever the issue was originally still exists.''' Troubleshoot hardware, upgrade firmware, etc to resolve the issue. | ||
[[Node Provider Troubleshooting|All Node Provider Troubleshooting links]] | [[Node Provider Troubleshooting|All Node Provider Troubleshooting links]] | ||
Revision as of 17:12, 26 June 2023
Steps to take when a server is unhealthy or has been removed from the network, but the connectivity in the data center is functioning correctly:
- Ensure that the server is powered on.
- Ensure that all link lights for active network interfaces are on.
- If any link lights are off, check for failed cables by swapping them out for known good cables as needed.
- Hook up a crash cart and check for errors on the screen, troubleshoot as needed.
- Contact your hardware vender if hardware issues are found or suspected.
- If your vendor requires a TSR log, see IDRAC access and TSR logs for an example of how to retrieve one from a Dell server.
- Updating the firmware might also resolve the issue.
- Consider updating the firmware if it has been a long time, and/or if you have recently had other nodes that needed firmware upgrades to become healthy again.
- If no known error is found, please redeploy the node with a fresh IC-OS image.
- The deployment process identifies/fixes many software issues.
- Note that if an old IC-OS image is used, the node will "appear" to be healthy at first, but it will not be able to catch up to the blockchain and will therefore fall behind and become unhealthy again. Thus, a current IC-OS image must be used.
- At the end, obtain the new principal ID for the node from the crash cart screen.
- Then search for the node's principal on the IC dashboard to verify that the node is healthy.
Use the dashboard to verify that the node is healthy
- The node count for your data center should match the number of nodes in that data center.
- Look for the principal ID for the node which you are servicing and make sure that it is "Awaiting Subnet" status
- If the node isn't listed at all, then it needs to be redeployed the node with a fresh IC-OS image.
If a node is healthy ("Awaiting Subnet" status) for a while and then changes to "Offline" then whatever the issue was originally still exists. Troubleshoot hardware, upgrade firmware, etc to resolve the issue.