Manual Node Recovery Guide

This runbook describes what steps node providers need to take during an NNS recovery.

Security warning

⚠️⚠️⚠️ Don’t get tricked into compromising your nodes. Only complete a manual node recovery if all of the following conditions are met:

• A subnet recovery is announced on the Internet Computer Status Page
• The DFINITY team reached out on the dedicated Matrix channel #ic-node-providers-incident-response:matrix.org.
  • Only the DFINITY team is able to send messages on this channel. In case of an incident, permissions are adapted so that everyone can send messages.

Prerequisites

• The recovery coordinator should have communicated with you the following:
  • The recovery parameters:
    • The VERSION: the commit ID of the recovery-GuestOS update image
    • The VERSION-HASH: the SHA256 sum of the recovery-GuestOS update image.
    • The RECOVERY-HASH: the SHA256 sum of the recovery.tar.zst
  • The node(s): which specific nodes managed by the NP/NO are part of the target subnet.
• Obtain console access to all nodes you run that are part of the target subnet.
  • Note that the recovery can be completed from a physical console or from the node's remote BMC virtual console view.

Recovery Steps

For each node to recover, you should perform the following process.

Obtain console access

Again, note that the recovery can be completed from a physical console or from the node's remote BMC virtual console view.