Node Provider Network Setup Guide
NP Network Requirements - EZ Guide
Who is this for? Node Providers (NP’s) who need to set up their servers into a rack and set up a functioning network.
What skills are necessary? You should be familiar with IP networking, network equipment and network cabling.
The Bare Minimum Network Requirements
To join your servers to the Internet Computer (IC) you will need:
- 10G Network equipment
- “Gen-2” node hardware
- Rackspace in a data center
- Internet connection
- ~300Mbps per node
- Ingress/egress ratio is currently 1:1. We expect egress (serving responses to client queries) to increase faster than ingress in the future.
- This should guide how many servers to deploy and the appropriate ISP connection speed
- E.g. a 1Gbps connection will support up to 3 IC nodes.
- One IPv6 /64 subnet - each node gets multiple IPv6 addresses
- One IPv4 address for every 4 nodes. See Appendix 1 for table.
- All IP addresses are assigned statically and automatically by IC-OS
- This is configured in the IC-OS Installation Runbook
When racking and stacking your servers, ensure the first two 10G network ports on each server are connected to the 10G switch.
For example, on a Supermicro 1U server, the bottom two ports are considered ports 1 and 2 and will be enumerated by Linux in this order. Connect the bottom two ports to the switch.
Servers from other vendors will differ! See the server documentation for guidance.
This is subject to change - the IC-OS network configuration logic is undergoing improvements to make it more flexible.
Connect the 10G switch to the ISP endpoint - this could be the Top Of Rack (TOR) switch or other box.
BMC Setup Recommendations
What’s a BMC?
The Baseboard Management Controller (BMC) grants control of the underlying server hardware.
BMC’s have notoriously poor security. Vendors may name their implementation differently (Dell -> iDRAC, HPE -> iLO, etc.).
Change the password
BMC’s usually come with a common password. Log in via crash cart, KVM or the web interface and change it to something strong.
No broad internet access
It is highly recommended: do not expose your BMC to the broad internet. This is a safety precaution against attackers.
- Don’t connect the BMC to the internet.
- Any BMC activities occur via SSH on the host (unreliable on many mainboard vendors) or via crash cart (requires physical interaction with the machine).
- Connect the BMC to a separate dumb switch, dumb switch connects to a Rack Mounted Unit (RMU).
- Connect the BMC to a managed switch, separate VLAN
This can get complicated. It’s outside the scope of this document to explain how to do this.
- StackExchange - Best practice for accessing management port of firewall
- Supermicro Guidance
- Unicom Guidance
What NOT to do
Don’t use external firewalls, packet filters, rate limiters
Don’t block or interfere with any traffic to the node machines. This can disrupt node machine functionality. Occasionally ports are opened for incoming (and outgoing) connections when new versions of node software are deployed.
What about network security?
IC-OS manages its own firewall(s) and rate limiters very strictly and is designed with security as a primary principle.
How DFINITY manages its servers
See reference DFINITY data center runbook.
- Did you deploy a 10G switch?
- Do the first and second 10G ports on each server plug into the 10G switch?
- Do you have one IPv6 /64 prefix allocated from your ISP?
- Do you have at least one IPv4 address for every four nodes allocated?
- Does each node have ~300Mbps bandwidth?
- Is your BMC inaccessible from the broad internet?
- Gen2 Network Requirements - more detailed, possibly out of date.
Appendix 1: Number of IPv4 Addresses Required
|# Nodes||# IPv4 Addresses|