Internet Computer Wiki - User contributions [en]

File:PO Receipt.pdf

2024-02-03T12:05:04Z

Lerak:

Confirmation that my reseller placed the their order with the distributer on 30 November 2023

537a604f61a5c17aa5c22ee4850ffa2c7f072a61ace112a14cff053e9d80c327 PO_Receipt.pdf

File:AP-OrderConfirmation.pdf

2024-02-03T12:03:14Z

Lerak:

Confirmation of order placed before 1 December 2023

fb9a6b4febc48e2b3030245d0d3a223b0da98ed824969c9ea5ce10b820966065 AP-OrderConfirmation.pdf

Node Deployment Guide

2023-12-13T07:48:34Z

Lerak: Added step to check UTC date/time

This runbook covers all steps necessary to install the Internet Computer Operating System (IC-OS).

The physical machine is expected to be racked and stacked according to its respective manual.

To complete these steps, you are expected to by physically present with your machine(s). Once you successfully onboarded your first node, you can bring up the other nodes in parallel.

If you encounter issues through any of these steps, check the [[Node Provider Troubleshooting]] page. If that does not solve your problem, you are encouraged to ask for assistance in the [[Node Provider Matrix channel]].

==1. Choose onboarding path (HSM vs. no HSM)==
If you chose the [[Node Provider Onboarding#5. Choose onboarding path .28HSM vs no HSM.29|HSM Node Provider Onboarding Path]], follow the [[NitroKey HSM installation runbook]] to onboard your nodes.

If you chose to onboard '''without''' a Nitrokey HSM, '''continue to the next step.'''

==2. Obtain requirements ==
*A USB (3.0 speed that can hold at least 4GB) to put the image file on.
**Faster USBs will allow the process to go much faster.
*The <code>node_operator_private_key.pem</code> for your data center (Acquired from [[Node Provider Onboarding#6. Setup the Node Operator keys|Node Provider Onboarding step 6]])
* It is recommended that each server have a label with the BMC's MAC address for ease of identification in future dashboard upgrades.

== 3. Download installation image==
Download the latest release of the '''IC-OS USB Installer Image''' and the '''corresponding checksum''' from the [https://dashboard.internetcomputer.org/releases Internet Computer Dashboard Releases].
*Note that you should always use a release that is less than 6 weeks old in order to ensure that your node can keep up with the blockchain.

== 4. Verify checksum and unarchive file==
===Mac OS X ===
#Open the Terminal and type:
#:<syntaxhighlight lang="shell">shasum -a 256 ~/Downloads/disk-img.tar.gz</syntaxhighlight>
#Compare the calculated checksum with the '''IC-OS installation image checksum''' file downloaded in the previous step. '''Warning:''' Only continue if they are identical, otherwise please post your issue in the [[Node Provider Matrix channel]].
#:Open the Terminal and type: <syntaxhighlight lang="shell">tar xzvf ~/Downloads/disk-img.tar.gz</syntaxhighlight>

===Linux / Ubuntu===
#Open the Terminal and type:
#:<syntaxhighlight lang="shell">sha256sum ~/Downloads/disk-img.tar.gz</syntaxhighlight>
#Compare the calculated checksum with the '''IC-OS installation image checksum''' file downloaded in the previous step. '''Warning:''' Only continue if they are identical, otherwise please post your issue in the [[Node Provider Matrix channel]].
#:Open the Terminal and type: <syntaxhighlight lang="shell">tar xzvf ~/Downloads/disk-img.tar.gz</syntaxhighlight>

===Windows===
#Open PowerShell and type:
#:<syntaxhighlight lang="shell">Get-FileHash -Algorithm SHA256 .\Downloads\disk-img.tar.gz</syntaxhighlight>
#Compare the calculated checksum with the '''IC-OS installation image checksum''' file downloaded in the previous step. '''Warning:''' Only continue if they are identical, otherwise please post your issue in the [[Node Provider Matrix channel]].
#:Open PowerShell and type: <syntaxhighlight lang="shell">tar xzvf .\Downloads\disk-img.tar.gz</syntaxhighlight>

==5. Create Bootable USB Stick ==
===Mac OS X===
# Open the Terminal and type:
#:<syntaxhighlight lang="shell">diskutil list</syntaxhighlight>
#All available drives should be shown. Identify which device corresponds to your USB stick. You may need to unmount the USB drive:
#:<syntaxhighlight lang="shell">sudo diskutil unmount /dev/YOUR_USB_DEVICE_MOUNTED_PARTITION # E.g. /dev/disk4s1</syntaxhighlight>
#The file path is an example. Use the absolute path to the downloaded image. '''Warning:''' You risk losing your own data if you specify a wrong device.
#:<syntaxhighlight lang="shell">sudo dd if=/Users/YOUR_USER_NAME/Downloads/disk.img of=/dev/YOUR_USB_DEVICE bs=1M</syntaxhighlight>If you get a “device is busy” error from the dd command, you can try running the following command to unmount all of the partitions on the disk, then re-run the dd command:
#:<syntaxhighlight lang="shell">sudo diskutil unmountDisk /dev/YOUR_USB_DEVICE # E.g. /dev/disk4</syntaxhighlight>

=== Linux / Ubuntu===
#Open the Terminal and type
#:<syntaxhighlight lang="shell">blkid</syntaxhighlight>
# All available drives should be shown. Identify which device corresponds to your USB stick. You may need to unmount the USB drive:
#:<syntaxhighlight lang="shell">sudo diskutil unmount /dev/YOUR_USB_DEVICE_MOUNTED_PARTITION # E.g. /dev/sdb1</syntaxhighlight>
#Replace ''/dev/YOUR_USB_DEVICE'' with the device that corresponds to your USB stick. '''Warning:''' You risk losing your own data if you specify a wrong drive.
#:<syntaxhighlight lang="shell">sudo dd if=~/Downloads/disk.img of=/dev/YOUR_USB_DEVICE bs=1M</syntaxhighlight>

===Windows===
#Download and install [https://rufus.ie/en/ Rufus Portable]
#Start Rufus
#Select the USB stick under device and select the previously downloaded IC-OS disk image and press start
#:[[File:05.png|480px|screenshot]]
#You may see some warnings. Make sure you don't have any other USBs in your computer and chose OK
#:[[File:06.png|480px|screenshot]]
#:[[File:07.png|480px|screenshot]]
# The "Ready" bar will go from left to right as it completes.

==6. Add configuration==

===A. Open Config.ini in a text editor===

===='''Mac OS X'''====

#Open Finder. You should now be able to see the CONFIG partition. If it's not visible, remove the USB and insert it again.
#:[[File:mac_01.png|580px|screenshot]]
#Double-click <code>config.ini</code> to open it in TextEdit.

===='''Linux'''====

#Open the File Manager. You should now be able to see the CONFIG partition. If it's not visible, remove the USB and insert it again.
#:[[File:linux_01.png|580px|screenshot]]
#Double-click <code>config.ini</code> to open it in KWrite.

===='''Windows'''====

#Open the Disk Management utility with a right click on the Start menu
#:[[File:09-b.png|300px|screenshot]]#:
#Right click the CONFIG partition
# Select Change drive letter or paths...
#:[[File:10-b.png|780px|screenshot]]
# Select any letter from the drop-down list
#:[[File:11-b.png|480px|screenshot]]
#Click OK.
#You should now be able to see the CONFIG partition in your Windows Explorer. Select the <code>config.ini</code> configuration file
#:[[File:12-b.png|780px|screenshot]]
#Click on Edit to open it.

===B. Edit Config.ini===

#Insert your IPv6 prefix, subnet and gateway.
#:[[File:Edit config ini.png|580px|screenshot]]
#:*The IPv6 prefix should consist of four groups of hexadecimal digits, separated by colons (':'). Each group can contain up to four hex digits.
#:*For example, a valid prefix could look like this: <code>2a00:fb01:400:100</code>
#:*'''Important:'''
#:**The prefix should not have a trailing ':'
#:**IPv6 CIDR notation allows for a double colon ('::') to represent consecutive groups of zeroes in an address. However, the prefix configuration in this context does '''not''' support '::'. Hence, the '::' shorthand should '''not''' be used: even if some groups are all zeros, they must be explicitly written out.
#Save the changes.
#:* If you have trouble saving this file directly, you may need to save to a known location first, then copy the file into place.
#:* If you need help, please do not hesitate to post your issue in the [[Node Provider Matrix channel]].
#:*:[[File:mac_03.png|580px|screenshot]]

=== C. Copy Node Operator private key to config partition===

#Copy <code>node_operator_private_key.pem</code> (created in [[Node Provider Onboarding#6. Setup the Node Operator keys|Node Provider Onboarding step 6]]) to the <code>CONFIG</code> partition. This file should have the name <code>node_operator_private_key.pem</code>, and sit next to <code>config.ini</code>, NOT inside the <code>ssh_authorized_keys</code> folder.

==7. Connect Crash Cart==
#In order to configure the UEFI and initiate the installation of the IC-OS, please connect a crash cart to the physical machine.
#Plug-in the VGA/Video, keyboard and IC-OS USB stick
#:[[File:08.png|580px|screenshot]]

==8. UEFI Setup and Boot Menu==

Make sure that server date/time is set to UTC (Universal Time Coordinated)

Use the related page below to set up the BIOS/UEFI according to your hardware vendor.

*[[Node Provider Machine Hardware Guide#Gen 2 Node Machine requirements|Gen2 hardware]]
**[[IC-OS Installation - UEFI Configuration - Gen2 Dell]]
** [[IC-OS Installation - UEFI Configuration - Gen2 Supermicro]]
** [[IC-OS Installation - UEFI Configuration - Gen2 Gigabyte]]
**[[IC-OS Installation - UEFI Configuration - Gen2 ASUS]]
*[[Node Provider Machine Hardware Guide#Gen 1 Node Machine requirements|Gen1 hardware]]
**[[IC-OS Installation - UEFI Configuration - Gen1 Dell|IC-OS Installation - UEFI Configuration - Gen1 Dell (Poweredge R6525)]]
**[[IC-OS Installation - UEFI Configuration - Gen1 Supermicro]]
***
'''Important:''' Do NOT enable the RAID bios setting. Doing so will cause issues with the IC-OS installation.

Resume from this point when you are finished configuring the BIOS.

==9. IC-OS Installation==
#Please wait while the USB Installer is booting up. This process can take up to 3 minutes.
#:[[File:35-sm.png|580px|screenshot]]
#The IC-OS installation starts. Please keep an eye on the progress. This part can take up to 10 minutes. Please remember to check the [[Possible Node Onboarding Errors]] page if you encounter any errors.
#:[[File:36-sm.png|580px|screenshot]]
#If the installation finished successfully, it will initiate a reboot.
#:[[File:38-sm.png|580px|screenshot]]<br />
==10. First Boot==
Please remember to check the [[Possible Node Onboarding Errors]] page if you encounter any errors onboarding.

'''🚨 Do NOT re-try the IC-OS installation after completing this section, as this can cause duplication within the registry.'''
#Once you see this message, you may unplug the USB stick and VGA/Video.
#:[[File:Node join message.png|580px|screenshot]]

Congratulations! Your machine successfully joined the Internet Computer! The machine has joined the IC and the Node Provider will start receiving rewards!

🚨 Again: Once you reach this stage and see this message, '''do not attempt to restart the onboarding process.''' Doing so may cause duplicate entries in the registry.

==11. Verify node onboarding==

#Verify that your node was successfully onboarded by checking its status on the [https://dashboard.internetcomputer.org/ dashboard] is set to either “Awaiting Subnet” or “Active in Subnet”.
#*The dashboard can be searched by your Node Provider principal. There, you should see the Node ID of your node (Node ID is outputted in step 10).
#*If the status of your node is not either “Awaiting Subnet” or “Active in Subnet”, or if it is not listed under your Node Provider principal, you should contact the [[Node Provider Matrix channel]] for assistance.
#*:[[File:Dashboard-node-verification.png|thumb|998x998px]]

IC-OS Installation - UEFI Configuration - Gen2 Supermicro

2023-12-13T07:42:44Z

Lerak: Adding "Restore Optimized Defaults" step. Making sure our starting point is the same on all servers.

== 1. UEFI - Enter Setup ==

# Reboot or power on the server.
# Watch for the screen with the Supermicro logo and boot options underneath it. Press DEL (delete) a couple of times, once the options are listed.
#: [[File:UEFI_Gen2_Supermicro_Post_Screen.png|580px]]
# If your option is recognized by the system, it will say: "Entering Setup…"

== 2. UEFI - Check Version ==

# Check the version number at the bottom of the screen. Ensure the UEFI/BIOS version is '''2.21.1280''' or higher. This version was tested and found to support features required for IC-OS.
#* Is your version lower than '''2.21.1280'''? Download the latest version and follow the included instructions at the [https://www.supermicro.com/en/support/resources/downloadcenter/swdownload Supermicro support site]

== 3. Restore Optimized Defaults ==

* Select "Restore Optimized Defaults" from the '''Save & Exit''' menu before makimg the changes below.
* Save changes

== 4. UEFI - Advanced Settings ==

# Change to the '''Advanced''' menu. Then, select '''Boot Feature''' and press enter.
#:[[File:UEFI_Gen2_Supermicro_Advanced_Menu_Boot_Feature_Highlighted.jpg|580px]]
# Set '''Wait For "F1" If Error''' to '''Disabled'''; press ESC (escape) to return to previous menu.
#:[[File:UEFI_Gen2_Supermicro_Advanced_Boot_Feature_Menu_Wait_For_F1.jpg|580px]]
# Select '''ACPI Settings''' and press enter.
#:[[File:UEFI_Gen2_Supermicro_Advanced_Menu_ACPI_Highlighted.jpg|580px]]
# Set '''PCI AER Support''' to '''Enabled'''; set '''NUMA Nodes Per Socket''' to '''NPS0'''; press ESC (escape) to return to previous menu.
#:[[File:UEFI_Gen2_Supermicro_Advanced_ACPI_Menu_ACPI_SRAT_L3_Cache.jpg|580px]]
# Select '''CPU Configuration''' and press enter.
#:[[File:UEFI_Gen2_Supermicro_Advanced_Menu_CPU_Configuration_Highlighted.jpg|580px]]
# Set '''SMT Control''' to '''Enabled'''.
# Set '''SMEE''' to '''Enabled'''.
# Set '''SEV ASID Count''' to '''253 ASIDs'''.
# Set '''SEV-ES ASID Space Limit Control''' to '''Manual'''.
# Set '''SEV-ES ASID Space Limit''' to '''100'''.
# Set '''SNP Memory (RMP Table) Coverage''' to '''Enabled'''.
# Set '''SVM Mode''' to '''Enabled'''.
#:[[File:SPM bios CPU Config.png|580px]]
# Press ESC (escape) to return to previous menu.
# Select '''NB Configuration''' and press enter.
#:[[File:UEFI_Gen2_Supermicro_Advanced_Menu_NB_Configuration_Highlighted.jpg|580px]]
# Set '''SEV-SNP Support''' to '''Enabled'''.
#:[[File:UEFI_Gen2_Supermicro_Advanced_North_Bridge_Configuration_Menu_SEV-SNP_Support.jpg|580px]]
#Select '''xGMI Configuration''' and press enter.
#:[[File:UEFI_Gen2_Supermicro_Advanced_North_Bridge_Configuration_Menu_xGMI_Highlighted.jpg|580px]]
#Set '''xGMI Max Speed''' to '''22Gbps'''; press escape to return to the previous menu.
#:[[File:Screenshot 2023-08-04 at 9.55.32 AM.png|580px]]
#Select '''Memory Configuration''' and press '''enter'''.
#:[[File:Screenshot 2023-08-03 at 5.19.46 PM.png|580px]]
#Set '''TSME''' to '''Disabled'''.
#:[[File:Screenshot 2023-08-03 at 5.20.27 PM.png|580px]]
#Press '''ESC''' twice (escape) to return to the '''Advanced''' main menu.
#Select '''PCIe/PCI/PnP Configuration''' and press enter.
#:[[File:UEFI_Gen2_Supermicro_Advanced_Menu_PCIe_Configuration_Highlighted.jpg|580px]]
#Set '''SR-IOV Support''' to '''Enabled'''.
#:[[File:UEFI_Gen2_Supermicro_Advanced_PCIe_Menu_SR-IOV_Support.jpg|580px]]
#Select '''Trusted Computing'''.
#:[[File:UEFI Menu Supermicro TPM Settings.png|580px]]
#Set '''Security Device Support''' to '''Enabled'''.
#Set '''SHA256 PCR Bank''' to '''Enabled'''.
#Set '''Platform Hierarchy''' to '''Enabled'''.
#Set '''Storage Hierarchy''' to '''Enabled'''.
#Set '''Endorsement Hierarchy''' to '''Enabled'''.
#:[[File:UEFI Menu Supermicro Advanced TrustedComputing.png|580px]]

==5. UEFI - Boot Menu==

#Change to the '''Boot''' menu; Set '''Boot Mode Select''' to '''UEFI'''.
#:[[File:UEFI_Gen2_Supermicro_Boot_Menu_Boot_Mode_Select.jpg|580px]]
#Change to '''Save & Exit''' menu, then select '''Save Changes and Reset''' and press enter.
#Select '''Yes''' at the prompt and press enter.
#:[[File:UEFI_Gen2_Supermicro_Save_And_Exit_Prompt.jpg|580px]]
The system will now reboot. Please do not unplug the IC-OS USB stick at this point.

== 6. Boot the IC-OS USB image==
# Watch for the screen with the Supermicro logo and boot options underneath it. Press F11 a couple of times, once the options are listed.
#:[[File:UEFI_Gen2_Supermicro_Post_Screen.png|580px]]
#If the keypress was recognized by the system, it will print '''Invoking Boot Menu'''.
#:[[File:SM-33.png]]
#In the boot menu, select the ''first partition on the USB device'' and press enter, e.g.:
#:[[File:SM-35.png|580px]]
<br>

==Return to the Installation Runbook==

*If using the '''non-HSM''' onboarding procedure, return to the [[IC-OS Installation Runbook#8. UEFI Setup and Boot Menu|IC-OS Installation Runbook]].

*If using the '''legacy, HSM''' onboarding procedure, return to the [[NitroKey HSM installation runbook#8. UEFI Setup and Boot Menu|NitroKey HSM installation runbook.]]

File:Restore Optimized Defaults.png

2023-12-13T07:38:02Z

Lerak:

First step of the UEFI configuration

Troubleshooting Unhealthy Nodes

2023-12-13T06:53:46Z

Lerak: Added a possible error OFFLINE node due the RAM module that failed

=== '''Use [https://dashboard.internetcomputer.org/centers the dashboard] to verify that the node is healthy''' ===
*The node count for your data center should match the number of nodes in that data center.
* Look for the principal ID for the node which you are servicing. Status explanations are [https://wiki.internetcomputer.org/wiki/Node_Provider_Troubleshooting#Node_Status_on_the_Dashboard here].
* If the node isn't listed at all, then it needs to be [[IC-OS Installation Runbook|redeployed the node with a fresh IC-OS image]].

=== '''Degraded nodes statuses''' ===

==== IC_OrchestratorFlapping ====
'''Explanation:''' Orchestrator coordinates the execution of many IC GuestOS processes, including the IC Replica. If the orchestrator is repeatedly restarting, then the GuestOS and the Replica process likely do not operate as expected.

'''Possible causes:'''

* Networking issues
* Hardware issues
* Software problems

'''Troubleshooting and remediation:'''

* Check if any NNS proposals were recently executed for the nodes in question https://dashboard.internetcomputer.org/governance
* Check if there are some bandwidth limitations on these nodes, and ensure any traffic shaping, QoS, DOS protection, etc, are ''disabled'' on the ISP side
* Perform other [https://wiki.internetcomputer.org/w/index.php?title=Unhealthy_Nodes&veaction=edit#Setting_Up_an_Auxiliary_Machine_for_Network_Diagnostics network diagnostics checks]
* Inspect node logs and metrics, if possible
* Consult other node providers and DFINITY if there are any known software problems with the latest revision that the node(s) are running

==== IC_Replica_Behind ====
'''Explanation:''' IC Replica is the main process, that runs canisters (smart contracts). If the Replica process cannot catch up, then the replica (node) cannot be a productive member of the IC subnet.

'''Possible causes:'''

* Networking issues
* Hardware issues
* Software problems

'''Troubleshooting and remediation:'''

* Check if there are any hardware issues reported by the machine's BMC (Baseboard Management Controller)
* Perform [https://wiki.internetcomputer.org/wiki/Updating_Firmware firmware upgrade]
* Check if any NNS proposals were recently executed for the nodes in question https://dashboard.internetcomputer.org/governance
* Check if there are some bandwidth limitations on these nodes, and ensure any traffic shaping, QoS, DOS protection, etc, are ''disabled'' on the ISP side
* Perform other [https://wiki.internetcomputer.org/w/index.php?title=Unhealthy_Nodes&veaction=edit#Setting_Up_an_Auxiliary_Machine_for_Network_Diagnostics network diagnostics checks]
* Inspect node logs and metrics, if possible
* Consult other node providers and DFINITY if there are any known software problems with the latest revision that the node(s) are running

=== '''"Orchestrator Started" message''' ===
This message is not an error, nor is it confirmation that the node is running properly. This must be determined in other ways:

*'''Check [https://dashboard.internetcomputer.org/ the dashboard]''' to check the status of the node. (Status explanations are [https://wiki.internetcomputer.org/wiki/Node_Provider_Troubleshooting#Node_Status_on_the_Dashboard here].) Use the principal ID that was assigned to the node when it was onboarded to identify it.
*If the node is not visible on the dashboard, then it has not registered with the Internet Computer.
**If you have recently installed a current IC-OS image, then you can try inserting the HSM and/or a reboot to see if it joins. This would work if a recent IC-OS installation was successful and only the registration and joining was interrupted.
**If you have ''not'' recently installed a current IC-OS image, then do ''not'' insert the HSM. You do not want the node to rejoin with an old IC-OS image, as it will only fail again. Instead, you should consider [[Updating Firmware|upgrading the firmware]] if it is running on old versions, and then redeploy the node with [[IC-OS Installation Runbook|a fresh/current IC-OS image]] (which will assign a new principal to the node so that you can identify it in the dashboard.)

=== '''Offline nodes''' ===
Network issues are the main reason why nodes are in an "OFFLINE" state.

* Your node may not be reachable from the IC or
* you may not be able to reach other nodes the IC from your node or
* you may note be able to reach the monitoring servers from your node.

Please refer to the '''Networking Troubleshooting Steps''' below.

Another possible reason for an OFFLINE node may be that your GuestOS failed to start due to a RAM failure.

=== '''Server Troubleshooting Steps''' ===
These steps may help when a server is unhealthy or has been removed from the network, but the connectivity in the data center is functioning correctly:

#Verify if the server is up and running:
#*Check the power status of the server.
#*Check if the server is displaying any error messages or indicators.
#*If possible, access the server remotely or physically to ensure it is functioning properly.
#Hook up a crash cart and check for errors on the screen, troubleshoot as needed.
#Consider [[Updating Firmware|updating the firmware]] if it has been a long time, and/or if you have recently had other nodes that needed firmware upgrades to become healthy again.
# If no known error is found, please [[IC-OS Installation Runbook|redeploy the node with a fresh IC-OS image]].
#* The deployment process identifies/fixes many software issues.
#* Note that if an old IC-OS image is used, the node will "appear" to be healthy at first, but it will not be able to catch up to the blockchain and will therefore fall behind and become unhealthy again. '''Thus, a current IC-OS image must be used.'''
#* At the end, obtain the new principal ID for the node from the crash cart screen so you can check the dashboard status.
#*'''If a node is healthy ("Awaiting Subnet" status) for a while and then changes to "Offline," then whatever the issue was originally still exists.''' Troubleshoot hardware, upgrade firmware, etc to resolve the issue.

=== '''Networking Troubleshooting Steps''' ===

# Inspect network hardware in the rack for any visible signs of malfunction (e.g. red lights) or incorrect setup
# Verify the cabling and port status on the switch:
#*Check the physical connection of the network cable between the server and the switch.
#*Ensure that the cable is securely plugged into the correct port on both ends.
#*Look for any signs of damage or loose connections.
#*Test the connectivity by trying a different network cable or using the same cable on a different port.
#Check for recent port flaps/link failures or any other activities which might cause it:
#*Check the logs or monitoring systems for any indications of port flapping or link failures.
#*Investigate any recent changes or activities that could have affected the network connection.
#*Consider any software updates, configuration changes, or physical alterations made recently.
#Try to perform a re-seat of cable/breakout/SFP/QSFP toward the affected machine:
#*Disconnect and reconnect the network cable at both ends (server and switch).
#*If applicable, re-seat any breakout cables, SFP modules, or QSFP modules used in the connection.
#*Ensure a secure and proper connection is established.
#Check with the switch vendor:
#*If the issue persists, contact the switch vendor's support team for further assistance.
#*Provide them with detailed information about the problem and any troubleshooting steps you have already taken.
#*Follow vendor guidance to troubleshoot and resolve the issue.
#**If your vendor requires a TSR log, see [[IDRAC access and TSR logs]] for an example of how to retrieve one from a Dell server.
#**[[Updating_Firmware|Updating the firmware]] might also resolve the issue.
# Utilize an auxiliary machine within the same rack with full network access to run diagnostics tools like <code>ping</code>, <code>traceroute</code>, and <code>nmap</code>
# Work with the ISP to troubleshoot and resolve any network routing issues identified during diagnostics
# Prepare for future incidents by establishing network redundancy and failover mechanisms

=== '''Best practices''' ===

# Keep a separate machine in the same rack with appropriate tools for network diagnostics and troubleshooting
# Engage with the node provider community for support and to share effective troubleshooting techniques

=== '''Setting Up an Auxiliary Machine for Network Diagnostics''' ===
Robust Internet connectivity is essential. Without access to internal node logs and metrics, troubleshooting requires alternative strategies, including the use of an auxiliary machine within the same rack. Here's a brief outline for setting up an auxiliary machine in the same rack, while following best security practices:
# Hardware Setup:
#* Choose a server with sufficient resources to run diagnostic tools without impacting its performance. There is no need to follow the gen1/gen2 hardware requirements for this server (since this node would not be joining the IC network) but make sure the server is performant enough to run network tests.
#* Ensure physical security measures are in place to prevent unauthorized access.
# Operating System and Software:
#* Install a secure operating system, like a minimal installation of Linux (we prefer Ubuntu 22.04), which reduces the attack surface.
#* Keep the system updated with the latest security patches and firmware updates.
# Network Configuration:
#* Configure the machine with an IPv6 address in the same range as the IC nodes, for accurate testing.
#* Set up a restrictive firewall on the machine to allow ''only the necessary'' inbound and outbound traffic. Consider allowing Internet access for this machine only during troubleshooting sessions, and keeping the machine behind a VPN at other times.
# Diagnostic Tools:
#* Install network diagnostic tools such as <code>ping</code>, <code>traceroute</code>, <code>nmap</code>, <code>tcpdump</code>, and <code>iperf</code>.
#* Configure monitoring tools to simulate node activities and track responsiveness.
# Security Measures:
#* Use strong, unique passwords for all accounts and change them regularly. Or, preferably, do not use passwords at all, and use key-based access instead.
#* Implement key-based SSH authentication and disable root login over SSH.
#* Regularly review logs for any unusual activities that might indicate a security breach.
# Maintenance and Updates:
#* Regularly update all software to the latest versions.
#* Periodically test your network diagnostic tools to ensure they are functioning as expected.

=== '''Frequently Asked Questions''' ===
Q: Is the monitoring system open-sourced? How does it communicate with the nodes?

A: The monitoring system configuration is not currently open-sourced. However, the node configuration that is required for proper node operation is fully open source. For more information about the node-to-node and node-to-monitoring communication, refer to the [https://sourcegraph.com/github.com/dfinity/ic/-/blob/ic-os/hostos/rootfs/etc/nftables.conf?L54-112 nftables configuration], which is the definitive guide for required open ports on Host OS, and [https://sourcegraph.com/github.com/dfinity/ic/-/blob/ic-os/guestos/rootfs/opt/ic/share/ic.json5.template?L322 similar nftables configuration] for the Guest OS. You can find the configuration for DFINITY-owned DCs and Gen1 node providers here: nftables configuration. We use [https://docs.victoriametrics.com/ Victoria Metrics] for metrics scraping (documentation) and [https://vector.dev vector.dev] for log scraping.

Q: What are the destination IPs and Ports for Frankfurt, Chicago, and San Francisco for connectivity troubleshooting?

A: At present, detailed node/port information is not publicly accessible, as disclosing this information is considered a security risk. To effectively troubleshoot connectivity issues with your nodes, we recommend setting up a "spot instance" or a temporary virtual machine (VM) with a cloud provider in each of the geographical regions. This approach allows you to test both connectivity and connection stability to your nodes, providing a practical solution for identifying and resolving network-related issues.

Q: The dashboard shows offline or degraded status for <DC>, but everything seems fine. What should we do?

A: Check if any of these issues are present: firewall restrictions, traffic shaping, DOS protection, or QOS features enabled on the ISP side. These should all be disabled for optimal node operation.

Back to [[Node Provider Troubleshooting]]

Back to [[Node Provider Documentation]]

File:GuestOS failed to start.png

2023-12-13T06:51:29Z

Lerak:

One RAM module failed - thus just 480GB memory available and not the expected 512GB

Node Provider Matrix channel

2023-11-30T06:01:22Z

Lerak: adding my dc's to the table

There is an open Matrix room (channel) that is intended to be a forum for potential and existing Node Providers as well as anybody else who would like to follow the node provider discussion: https://app.element.io/#/room/#ic-node-providers:matrix.org

The runs runs on the open, public, and decentralized Matrix network. Matrix [https://matrix.org/clients/ has many clients] you can use. One of the popular clients is element.io, which is very similar to Slack, and offers a web UI, a [https://element.io/download desktop app], and a mobile app.

We strongly recommend that you add [https://ems-docs.element.io/books/element-cloud-documentation/page/element-settings an email address in the Element Profile settings] and to [https://element.io/help#settings3 enable notifications for missed messages].

If you are a Node Provider, please also add your Node Provider contact details on the Matrix room, to facilitate communication.
{| class="wikitable"
|+
!Node Provider Name
!Node Provider Id
!DC list
!Matrix aliases
|-
|1G
|7k7b7-4pzhf-aivy6-y654t-uqyup-2auiz-ew2cm-4qkl4-nsl4v-bul5k-5qe
|fm1
|
|-
|43rd Big Idea Films
|sqhxa-h6ili-qkwup-ohzwn-yofnm-vvnp5-kxdhg-saabw-rvua3-xp325-zqe
|hu1
|
|-
|87m Neuron, LLC
|eipr5-izbom-neyqh-s3ec2-52eww-cyfpg-qfomg-3dpwj-4pffh-34xcu-7qe
|pl1, dl1, lv1
|
|-
|9Yards Capital
|spp3m-vawt7-3gyh6-pjz5d-6zidf-up3qb-yte62-otexv-vfpqg-n6awf-lqe
|jv1, ch2, ny1
|
|-
|ACCUSET SOLUTIONS
|cp5ib-twnmx-h4dvd-isef2-tu44u-kb2ka-fise5-m4hta-hnxoq-k45mm-hqe
|gn1
|
|-
|Adam Dymecki
|n32q7-33lmk-m33tr-o5ltb-po6cb-tqqrr-2x6wp-pzhw7-ymizu-o3fyp-sqe
|
|
|-
|A Dog's Boutique, LLC
|uvawj-nnt3i-4ch3a-hruy4-hyfjn-7owvl-wgkli-uq2hd-e6lkq-v4b2o-4ae
|
|
|-
|Allusion
|rbn2y-6vfsb-gv35j-4cyvy-pzbdu-e5aum-jzjg6-5b4n5-vuguf-ycubq-zae
|
|
|-
|Anonstake
|kos24-5xact-6aror-uofg2-tnvt6-dq3bk-c2c5z-jtptt-jbqvc-lmegy-qae
|
|
|-
|ANYPOINT PTY LTD
|fwnmn-zn7yt-5jaia-fkxlr-dzwyu-keguq-npfxq-mc72w-exeae-n5thj-oae
|
|
|-
|Archery Blockchain SCSp
|7ryes-jnj73-bsyu4-lo6h7-lbxk5-x4ien-lylws-5qwzl-hxd5f-xjh3w-mqe
|
|
|-
|Arjay LLC
|l2kri-jarwr-7whc4-pjdpn-n6hlb-45ltr-l6ghm-twttl-pcsvt-rynko-dqe
|
|
|-
|Artem Horodyskyi
|diyay-s4rfq-xnx23-zczwi-nptra-5254n-e4zn6-p7tqe-vqhzr-sd4gd-bqe
|
|
|-
|Aspire Properties
|2byzn-q2crt-hgczo-eruff-6p7af-pemor-n2z4z-6d2sd-wvdqa-yqvxb-mqe
|
|
|-
|Bigger Capital
|7a4u2-gevsy-5c5fs-hsgri-n2kdz-dxxwf-btcfp-jykro-l4y7c-7xky2-aqe
|me3
|Dallas W
|-
|Bitmoon
|mjnyf-lzqq6-s7fzb-62rqm-xzvge-5oa26-humwp-dvwxp-jxxkf-hoel7-fqe
|
|
|-
|Blockchain Development Labs
|7at4h-nhtvt-a4s55-jigss-wr2ha-ysxkn-e6w7x-7ggnm-qd3d5-ry66r-cae
|
|
|-
|BlockTech Ventures, LLC
|ks7ow-zvs7i-ratdk-azq34-zio2b-gbekj-qjicg-pfhp3-ovhgu-k5qql-dae
|
|
|-
|BLP22, LLC
|sma3p-ivkif-hz7nu-ngmvq-ibnjg-nubke-zf6gh-wbnfc-2dlng-l3die-zqe
|
|
|-
|Bohatyrov Volodymyr
|dhywe-eouw6-hstpj-ahsnw-xnjxq-cmqks-47mrg-nnncb-3sr5d-rac6m-nae
|
|
|-
|Boolean Bit, LLC
|qdj4d-76lh3-w2q5i-kwjcd-643pq-pk42d-cziag-4hkau-35gib-m7s33-6qe
|
|
|-
|Brener, Inc.
|2wxxr-qwylo-n7dhz-6co6m-iektd-vl7dn-ocvyc-xazaf-hbfxq-66spe-aae
|
|
|-
|Conic Ventures
|i3cfo-s2tgu-qe5ym-wk7e6-y7ura-pptgu-kevuf-2feh7-z4enq-5hz4s-mqe
|
|
|-
|Coplus Limited
|jz47c-irtey-dr2nb-wienh-emhaz-jo6ua-gsbho-t2z5j-l7kbf-5i7p5-5ae
|
|
|-
|CRM52 Systems, LLC
|7nxxb-6qgm4-fftx3-xkwpj-sjrcm-tzmk5-dvuqk-l4ei4-3hvii-scwnj-tae
|
|
|-
|DFINITY Operations SA
|bvcsg-3od6r-jnydw-eysln-aql7w-td5zn-ay5m6-sibd2-jzojt-anwag-mqe
|mr1, sh1, zh2
|Katie Peters, sasa-tomic, Sven Fischer, eerokelly, garym-dfn, Radek Zakrzyk, Samuel Howard, Tomas Hlavacek, dfinityian, andrewbattat, Rudd-O, tthebst
|-
|DFINITY USA Research LLC
|lq5ra-f4ibl-t7wpy-hennc-m4eb7-tnfxe-eorgd-onpsl-wervo-7chjj-6qe
|
|Katie Peters, sasa-tomic, Sven Fischer, eerokelly, garym-dfn, Radek Zakrzyk, Samuel Howard, Tomas Hlavacek, dfinityian, andrewbattat, Rudd-O, tthebst
|-
|DFINITY USA Research LLC
|r2qgy-abcek-yc2ot-3powq-gayr6-4dibc-jvvfl-ri376-gmfmw-u2v7w-4qe
|bo1
|Katie Peters, sasa-tomic, Sven Fischer, eerokelly, garym-dfn, Radek Zakrzyk, Samuel Howard, Tomas Hlavacek, dfinityian, andrewbattat, Rudd-O, tthebst
|-
|DRMxTech Enterprises, LLC
|olgti-2hegv-ya7pd-ky2wt-of57j-tzs6q-ydrpy-hdxyy-cjnwx-ox5t4-3qe
|
|
|-
|Exaion
|xo7ih-nswlt-hbq3n-v5ixi-etu7j-sasg6-fjf4p-zx6or-cc7c3-pnh7t-2ae
|
|
|-
|Fidgitville Ventures LLC
|ob633-g55bt-y6pu5-5iby6-jmcvi-oylqs-q6ahw-cvecq-5ckeh-m4wws-nae
|
|
|-
|Finteck LLC
|x3zyd-pkcbf-5n3w2-n7uov-2qrbt-d3kfn-ojdd7-pxog5-vpqnt-6lex5-fqe
|
|
|-
|Fractal Labs AG
|wdjjk-blh44-lxm74-ojj43-rvgf4-j5rie-nm6xs-xvnuv-j3ptn-25t4v-6ae
|
|
|-
|Fritz Huie
|67gkg-gkgzz-g2ubz-3cc6h-jr3zm-twsii-7i325-r3gzr-kp2kh-dwxg6-pqe
|
|
|-
|FUM Capital, LLC
|zgupw-boshs-mg6kz-2ciwm-upk7g-igjpn-75t3p-np6g4-47l7q-nuefa-6qe
|
|
|-
|Geodd Pvt Ltd
|eybf4-6t6bb-unfb2-h2hhn-rrfi2-cd2vs-phksn-jdmbn-i463m-4lzds-vqe
|
|
|-
|GeoNodes LLC
|5wedd-dwi2c-ybr2g-benav-qlgc7-vifbt-qz3ml-ymopp-uaide-ilpkb-tae
|
|
|-
|George Bassadone
|lksfo-j2smg-76m4c-jwasz-e5exl-gptsk-jvtc7-zvm34-ezk4g-p2xjt-dqe
|
|
|-
|Giant Leaf, LLC
|wwdbq-xuqhf-eydzu-oyl7p-ga565-zm7s7-yrive-ozgsy-zzgh3-qwb3j-cae
|
|
|-
|Goat, LLC
|p6fou-ngmgk-rxc6t-7ckzz-hojr2-kk6r3-xnlrk-ewzvu-g6xms-rfafz-zae
|
|
|-
|Goodsir, LLC
|myrs2-bc6j6-mydpr-2jmli-l45mu-35ybt-c34mo-kjpve-zmaao-ajusy-nqe
|
|
|-
|Honeycomb Capital (Pty) Ltd
|nmdd6-rouxw-55leh-wcbkn-kejit-njvje-p4s6e-v64d3-nlbjb-vipul-mae
|
|honeycombcapital
|-
|Iancu Aurel
|i7dto-bgkj2-xo5dx-cyrb7-zkk5y-q46eh-gz6iq-qkgyc-w4qte-scgtb-6ae
|
|
|-
|Icaria Systems Pty Ltd
|ihbuj-erwnc-tkjux-tqtnv-zkoar-uniy2-sk2go-xfpkc-znbb4-seukm-wqe
|mn2
|Icarus
|-
|IC Pros
|srga3-cikqa-srnxx-rwejf-672jj-5o6qy-tuzsa-khds3-2ofjw-5gnew-mae
|
|
|-
|Illusions In Art (Pty) Ltd
|optdi-nwa4m-hly3k-6ua4n-sqyxf-yahvb-wps77-ddayn-r7zcz-edla5-7qe
|
|
|-
|InfoObjects
|7ws2n-wqorv-vmo4m-5e222-n42c3-hk43s-ei3kp-4hpbn-xlkzo-jgv7i-tqe
|
|
|-
|Internet Computer Explorer
|i7v7g-cwjtl-gzd2s-nt2ko-4d5su-vh64h-bqu4h-rio26-tbaej-dtj6d-fae
|
|
|-
|Ivanov Oleksandr
|ivf2y-crxj4-y6ewo-un35q-a7pum-wqmbw-pkepy-d6uew-bfmff-g5yxe-eae
|
|
|-
|Jeffrey Schnettler
|5zqo2-omblo-i7knq-qyrfu-mjccn-tljyd-qslab-b7ukn-7tshi-pbeke-pae
|
|
|-
|Jimmy Quach
|dzxyh-fo4sw-pxckk-kwqvc-xjten-3yqon-fm62b-2hz4s-raa4g-jzczg-iqe
|
|
|-
|Jonathan Ziskind
|qcs4o-yswwp-7ozhg-m2ago-ytjyl-zlckb-raykw-fi5hl-cflyt-4beyv-zqe
|
|
|-
|Joseph Stella
|f5wg2-kl4aq-cj4ym-nq7ul-i4kxy-t4x3x-yiyc5-3wlu6-opzwd-o5qes-hae
|
|
|-
|Karel Frank
|unqqg-no4b2-vbyad-ytik2-t3vly-3e57q-aje2t-sjb5l-bd4ke-chggn-uqe
|jb1, jb2, sc1
|lerak
|-
|KLAW Media, LLC
|pcwis-xaq3p-xvasz-5cfws-oelni-xs72v-acbwz-umxnq-nb2bc-ziyuu-sqe
|
|
|-
|Kontrapunt (Pty) Ltd
|py2kr-ipr2p-ryh66-x3a3v-5ts6u-7rfhf-alkna-ueffh-hz5ox-lt6du-qqe
|
|
|-
|Krishna Enterprises
|zy4m7-z5mhs-zfkpl-zlsjl-blrbx-mvvmq-5z4zu-mf7eq-hhv7o-ezfro-3ae
|
|
|-
|Krishna Sriram
|sajvr-l5iok-rj6c2-r76co-k2mde-d3kir-pnmjs-443f6-lxto4-pu55q-uae
|
|
|-
|Lauren Dymecki Chickvara
|abscc-3lezh-oezci-5i3kz-pkwlc-ozz3r-5wv4n-htujn-rtajh-6cgyv-jae
|
|
|-
|Lukas Helebrandt
|efem5-kmwaw-xose7-zzhgg-6bfif-twmcw-csg7a-lmqvn-wrdou-mjwlb-vqe
|bn1
|@lukas:greensteps.cn
|-
|Luke Jacobson
|6dwst-olsa5-tagsr-jylgj-oicju-rf42e-o7job-gedre-fgoxt-7br62-jqe
|
|
|-
|Maksym Ishchenko
|4r6qy-tljxg-slziw-zoteo-pboxh-vlctz-hkv2d-7zior-u3pxm-mmuxb-cae
|
|
|-
|Marc Johnson
|xfvlz-qkgs4-xyeyx-lbke5-aarza-ovi52-wzpxl-3p27l-ov6ki-csxv6-qqe
|
|
|-
|Marvelous Web3
|7uioy-xitfw-yqcko-5gpya-3lpsw-dw7zt-dyyyf-wfqif-jvi76-fdbkg-cqe
|
|
|-
|Mary Ren
|vdzyg-amckj-thvl5-bsn52-2elzd-drgii-ryh4c-izba3-xaehb-sohtd-aae
|
|
|-
|MB Patrankos šūvis
|4jjya-hlyyc-s766p-fd6gr-d6tvv-vo3ah-j5ptx-i73gw-mwgyd-rw6w2-rae
|
|
|-
|Mika Properties, LLC
|3siog-htc6j-ed3wz-sguhu-2objz-g5qct-npoma-t3wwt-bd6wy-chwsi-4ae
|
|
|-
|MI Servers
|izmhk-lpjum-uo4oy-lviba-yctpc-arg4b-2ywim-vgoiu-gqaj2-gskmw-2qe
|
|
|-
|Mitchel Guerra
|4anlt-yam7x-eodmx-ik7mo-nl3kx-t35fj-52hfy-uv4jj-u2iea-ntg76-pqe
|
|
|-
|Moon Block Ventures
|sdal5-w2c3d-p3buy-zieck-2wyuj-eu5bn-rkfe6-uuspi-o4n2b-gpei7-iae
|
|
|-
|Mostly Wholesome, Inc.
|ou3o7-akyjc-ldwd5-anyjn-l2buz-cwhbg-nehlc-abkde-qtc7w-fozdi-hae
|
|
|-
|Neptune Partners
|4dibr-2alzr-h6kva-bvwn2-yqgsl-o577t-od46o-v275p-a2zov-tcw4f-eae
|
|
|-
|NoviSystems, LLC
|hk7eo-22zam-kqmsx-dtfbj-k5i6f-jg65h-micpf-2cztc-t2eqk-efgvx-vqe
|ny2, fm1
|novisystems
|-
|OneSixtyTwo Digital Capital
|6nbcy-kprg6-ax3db-kh3cz-7jllk-oceyh-jznhs-riguq-fvk6z-6tsds-rqe
|
|
|-
|Origin Game
|cgmhq-c4zja-yov4u-zeyao-64ua5-idlhb-ezcgr-cultv-3vqjs-dhwo7-rqe
|
|
|-
|Palnu_Logistics
|pfcmh-uqedc-ijkjz-mdo35-bvprd-2vtlw-ktod5-npgfc-pgbq4-akaen-uqe
|
|
|-
|Paul Legato
|yr4eg-kwk3m-q44vj-ale35-2mtxk-5dyn7-vgppx-z6tcw-kzo4o-ezpm5-fqe
|
|
|-
|Peggy Shafaghi
|72idx-a7c3y-nrcwc-lboj4-mmsas-sfdpm-gq23i-h2yuy-lykcj-vrxn2-jqe
|
|
|-
|Philippe Chapparone
|2c4m6-25hos-qroi3-mk4aj-nog6s-zbzcv-ccpcc-cbv7s-sjy6p-bv3g5-fae
|
|
|-
|Pindar Technology Limited
|r3yjn-kthmg-pfgmb-2fngg-5c7d7-t6kqg-wi37r-j7gy6-iee64-kjdja-jae
|
|
|-
|Power Meta Corporation
|4fedi-eu6ue-nd7ts-vnof5-hzg66-hgzl7-liy5n-3otyp-h7ipw-owycg-uae
|
|
|-
|Prayit Jain
|waj5k-wlyvv-jbj4n-vxwjm-dmkyg-uw2nl-ggojp-34kln-wgx3n-d7xih-5qe
|
|
|-
|Privoxy Solutions, LLC
|trxbq-wy5xi-3y27q-bkpaf-mhi2m-puexs-yatgt-nhwiy-dh6jy-rolw5-zqe
|
|
|-
|Rachel Dymecki
|chnsu-yaqt5-6osy5-au4zn-li6yu-nufmw-dewrt-utkiu-twd76-ujypw-rae
|
|
|-
|Richard Ma
|egb3e-rzi2e-vpsmm-akysp-l2owk-4dgst-b5hmg-xrkwa-cr3uk-zlzds-mae
|
|
|-
|Richard Suarez
|cmcjw-6c5ve-4zjnt-lipnl-2lp43-oh5wk-ewciz-xyvnv-m2rz5-hkm6a-hqe
|
|
|-
|Ricky Sidhu
|q22bo-3uyqa-jvtpt-gapjk-pseor-esx4a-zyb74-vzea4-o7nx2-tafgq-hae
|
|
|-
|Rishi Sachdev
|6tg64-cdfoh-kl35i-p6qti-sose3-746lr-jk5ex-phuvu-jfu3d-5svwa-7qe
|
|
|-
|Rivram Inc
|ulyfm-vkxtj-o42dg-e4nam-l4tzf-37wci-ggntw-4ma7y-d267g-ywxi6-iae
|
|
|-
|Rodney Zorrilla
|bgprp-b2mnt-ci5in-57vuk-p7qvo-tj2tb-5w5su-qwenk-gbe77-mnuiq-sqe
|
|
|-
|Ronnie Pellizzari
|wwxec-c2gd2-bu5on-ktpwz-z2ph3-vlr4p-m7ztf-6ck7r-nt3r4-fxbdq-mae
|
|
|-
|Russell Ford
|p5jx4-lsrog-ep5o3-5uudg-opdg3-ur45z-xeefi-4ejfz-47t2k-riwfv-dae
|
|
|-
|Scott Hallock
|i6sxi-fks25-viets-mboa7-3i23b-qeocf-e57qj-ar6vy-2mchu-xb5vp-aqe
|
|
|-
|Shelburne Ventures, LLC
|a24zv-2ndbz-hqogc-ev63f-qxnpb-7ramd-usexl-ennaq-4om4k-sod6u-gae
|
|
|-
|Staking Facilities
|niw4y-easue-l3qvz-sozsi-tfkvb-cxcx6-pzslg-5dqld-ooudp-hsuui-xae
|
|
|-
|Starbase
|sixix-2nyqd-t2k2v-vlsyz-dssko-ls4hl-hyij4-y7mdp-ja6cj-nsmpf-yae
|
|
|-
|Starseed Technology, LLC
|ruxoj-jnqql-uau6o-xwrtb-ufde4-geddn-mnhni-wpew4-zhzi5-xjrxi-lqe
|
|
|-
|Sygnum Bank
|6r5lw-l7db7-uwixn-iw5en-yy55y-ilbtq-e6gcv-g22r2-j3g6q-y37jk-jqe
|
|
|-
|Tomahawk.vc
|ucjqj-jmbj3-rs4aq-ekzpw-ltjs3-zrcma-t6r3t-m5wxc-j5yrj-unwoj-mae
|
|
|-
|Virtual Hive Ltd
|wdnqm-clqti-im5yf-iapio-avjom-kyppl-xuiza-oaz6z-smmts-52wyg-5ae
|
|
|-
|Vladyslav Popov
|3oqw6-vmpk2-mlwlx-52z5x-e3p7u-fjlcw-yxc34-lf2zq-6ub2f-v63hk-lae
|
|
|-
|Wancloud limited
|g2ax6-jrkmb-3zuh3-jibtb-q5xoq-njrgo-5utbc-j2o7g-zfq2w-yyhky-dqe
|
|
|-
|Web3game
|64xe5-tx2s3-4gjmj-pnozr-fejw2-77y5y-rhcjk-glnmx-62brf-qin5q-pqe
|
|
|-
|William Zelver
|usau7-upgoh-sg464-6qnso-lud42-nxho6-ith26-a2jhq-q5bgy-ajeou-4ae
|
|
|-
|Wolfhound LLC
|6mifr-stcqy-w5pzr-qpijh-jopft-p6jl3-n2sww-jhmzg-uzknn-hte4m-pae
|
|
|-
|Wolkboer (Pty) Ltd
|mme7u-zxs3z-jq3un-fbaly-nllcz-toct2-l2kp3-larrb-gti4r-u2bmo-dae
|
|
|-
|Zondax AG
|hzqcb-iiagd-4erjo-qn7rq-syqro-zztl6-cpble-atnkd-2c6bg-bxjoa-qae
|
|
|-
|ZTLC PTE LTD
|xsrwt-tl3tk-n3aya-rafh3-ta6xu-eviw5-ae5dg-2f3bf-siaab-wdwdo-dqe
|
|
|}