Difference between revisions of "Node Provider Machine Hardware Guide"

From Internet Computer Wiki
Jump to: navigation, search
(Added Dell section. Edited SPM+GBY sections.)
(Adjusted node names)
(41 intermediate revisions by 8 users not shown)
Line 1: Line 1:
−
== What are the Hardware Requirements to be a Node Provider? ==
+
Node Providers operate one or more node machines than run in the Internet Computer network.
−
Hardware requirements are specified for all nodes that were purchased prior to Genesis - these are 'Gen 1'. <br>
+
 
−
Gen 2 hardware requirements are currently [https://forum.dfinity.org/t/draft-motion-proposal-new-hardware-specification-and-remuneration-for-ic-nodes/14202 under discussion] as they must be approved by the NNS. <br>
+
Gen1 hardware requirements have been used by Node Providers to set up node machines during the Genesis launch.  
−
== Gen 1 ==
+
 
−
=== Node Type 1 - Dell ===
+
The Gen2 hardware requirements have been defined for the further growth of the Internet Computer network. The specifications for the Gen2 node machines are generic (instead of vendor specific) and support VM memory encryption and attestation, which will be needed in future features on the Internet Computer.  
 +
 
 +
While Gen2 Node Providers are only strictly required to follow the generic specifications '''(the IC-OS installation will fail if the generic specifications are not met)''', it is strongly recommended for Node Providers to purchase one of the validated configurations listed below.  
 +
== Gen 2 Node Machine requirements ==
 +
==== Gen-2.3 Generic specification ====
 
{| class="wikitable"
 
{| class="wikitable"
 
|-
 
|-
−
| 1 || R6525
+
| Dual Socket AMD EPYC Milan CPU - Recommended: [https://en.wikichip.org/wiki/amd/epyc/7313 7313] (16C/32T 3 Ghz)
 +
[https://en.wikichip.org/wiki/amd/epyc#7003_Series_.28Zen_3.29 optionally] 7343, 7373, 73F3
 +
|-
 +
| 16x 32GB RDIMM, 3200MT/s, Dual Rank
 +
|-
 +
| 5x 6.4TB NVMe Mixed Mode (DWPD >= 3) with U.2 or U.3 interface
 +
|-
 +
| Dual Port 10G SFP or BASE-T
 +
|-
 +
| TPM 2.0
 +
|}
 +
'''Important:''' Do NOT order a RAID controller for your node machine. If a RAID controller is included in your setup, it will cause issues with the installation of IC-OS on your machine.
 +
 
 +
=== Dfinity-validated configurations ===
 +
DFINITY has [https://forum.dfinity.org/t/draft-motion-proposal-new-hardware-specification-and-remuneration-for-ic-nodes/14202/14?u=garym validated] the following Gen2 hardware configurations.
 +
==== Dell ====
 +
{| class="wikitable"
 +
|-
 +
| 2 || AMD EPYC 7343 3.2GHz, 16C/32T, 128M Cache (190W)
 +
|-
 +
| 16 || 32GB RDIMM, 3200MT/s, Dual Rank 16Gb BASE x8
 
|-
 
|-
−
| 1 || Chassis - Supports Up to 10 NVMe drives, 12 drives total
+
| 5 || 6.4TB Enterprise NVMe Mixed Use AG Drive U.2 Gen4 with carrier
 
|-
 
|-
−
| 1 || Dual 1 GB on Motherboard
+
| 1 || PowerEdge R6525 Motherboard, with 2 x 1Gb Onboard LOM (BCM5720)MLK V2
 
|-
 
|-
−
| 3 || Low Profile PCIe Slots
+
| 2 || Dual, Hot-plug, Redundant Power Supply (1+1) 1100W, Mixed Mode Titanium
 
|-
 
|-
−
| - || 3 Year Basic NBD Support
+
| 1 || Intel X710 Dual Port 10GbE SFP+, OCP NIC 3.0
 
|-
 
|-
−
| - || iDrac Enterprise
+
| 1 || Trusted Platform Module 2.0 V3
 +
|}
 +
 
 +
==== ASUS ====
 +
{| class="wikitable"
 
|-
 
|-
−
| 1 || Dual port 10GbE Base - T Adapter Broadcom, PCIe Low Profile
+
| 2 || AMD EPYC 7313 (3,00 GHz, 16-Core, 128 MB)
 
|-
 
|-
−
| 10 || 3.2TB NVMe, Mixed Use, 2.5" with Carrier
+
| 16 || 32GB ECC Reg ATP DDR4 3200 RAM
 
|-
 
|-
−
| 16 || 32GB RDIMM (3200MT/s)
+
| 5 || 6.4 TB NVMe Kioxia SSD 3D-NAND TLC U.3 (Kioxia CM6-V)
 
|-
 
|-
−
| 2 || AMD 7302 3GHz, 16C/32T, 128M, 155W, 3200
+
| 1 || Asus Mainboard KMPP-D32 Series (without OCP 3.0, without Pike)
 
|-
 
|-
−
| 1 || Single Power Supply (800W)
+
| 2 || 1600 Watt redundant PSU
 
|-
 
|-
−
| 1 || C13-C14, 3M, 125V 15A Power Cored
+
| 1 || Broadcom 25 Gigabit P225P SFP28 Dual Port Network Card
 
|-
 
|-
 +
|0
 +
|TPM 2.0*
 
|}
 
|}
−
<br>
+
<nowiki>*</nowiki> We (Dfinity) forget to order the TPM 2.0 on the ASUS test machine we validated. Your machine DOES need the TPM 2.0. 
−
=== Node Type 1 - SuperMicro ===
+
 
 +
==== Supermicro ====
 
{| class="wikitable"
 
{| class="wikitable"
 
|-
 
|-
−
| 1 || AS - 1023US - TR4
+
| 2 || AMD EPYC 7313 (3,00 GHz, 16-Core, 128 MB)
 
|-
 
|-
−
| 2 || Rome 7302 DP/UP 16C/32T 3.0
+
| 16 || 32GB ECC Reg DDR4 3200 RAM
 
|-
 
|-
−
| 16 || 32GB DDR4-3200 2Rx4 ECC REG DIMM
+
| 5 || 6.4 TB NVMe TLC SSD, PCIe 4.0 x4, U.3 2.5", 3 DWPD
 
|-
 
|-
−
| 5 || Samsung PM983 3.2TB NVMe PCIE/SATA Hybrid M.2 & 1 PCIE
+
| 1 || SuperMicro H12DSU-iN
 
|-
 
|-
−
| 2 || 800W Power Supply
+
| 2 || 1200 Watt redundant PSU
 
|-
 
|-
−
| 1 || Std LP 2-port 10G RJ45, Intel x540
+
| 1 || Dual Port 10GBase-T Network Adapter Intel® X710-TM4*
 
|-
 
|-
−
| 5 || Micron 5300 PRO 7.4TB, SATA, 2.5', 3D TLC, .6DWPD (with Caddie)
+
|1
 +
|Dual Port 10GbE Network Adapter Intel® X710-TM4, SFP+*
 
|-
 
|-
−
| 1 || C13/C14 13A Power Cord
+
| 1 || Trusted Platform Module 2.0
 
|}
 
|}
−
<br>
+
<nowiki>*</nowiki> Only one dual port network adaptor is required.
−
=== Node Type 2 - Dell ===
+
 
 +
==== Gigabyte ====
 
{| class="wikitable"
 
{| class="wikitable"
 
|-
 
|-
−
| 1 || R6515
+
| 2 || AMD EPYC 7313 (3,00 GHz, 16-Core, 128 MB)
 +
|-
 +
| 16 || 32GB ECC Reg DDR4 3200 RAM
 
|-
 
|-
−
| 1 || 3.5" Chassis with up to 4 Hot-Plug Hard Drives and OS RAID
+
| 5 || 6.4 TB NVMe TLC SSD, PCIe 4.0 x4, U.3 2.5", 3 DWPD
 
|-
 
|-
−
| 1 || Dual 1 Gb on Motherboard
+
| 1 || Gigabyte MZ92-FS1 Rev. 3.0
 
|-
 
|-
−
| 3 || Low Profile PCIe Slots
+
| 2 || 1200 Watt redundant PSU
 
|-
 
|-
−
| 1 || Standard Fan
+
| 1 || Dual Port 1000Base-T Network Adapter Intel® I350-AM2*
 
|-
 
|-
−
| - || 3 Year Basic NBD Support
+
|1
 +
|Dual Port 10GbE Network Adapter, Broadcom NetXtreme E-Series*
 
|-
 
|-
−
| - || iDrac Enterprise
+
| 1 || Trusted Platform Module 2.0
 +
|}
 +
<nowiki>*</nowiki> Only one dual port network adaptor is required.
 +
 
 +
=== Community-validated configurations ===
 +
Node Providers have validated the following Gen2 hardware configurations:
 +
 
 +
==== Dell PowerEdge R7525 ====
 +
{| class="wikitable"
 +
|2
 +
|AMD 7313 3.0GHz,16C/32T,128M,155W,3200
 
|-
 
|-
−
| 2 || Dual Port 10GbE Base - T Adapter Broadcom, PCIe LOw Profile
+
|16
 +
|32GB RDIMM, 3200MT/s, Dual Rank 16Gb BASE x8
 
|-
 
|-
−
| 2 || 480GB SSD SATA Mix Use 6Gbps 512 2.5in Hot-Plug AG Drive, 3.5in
+
|5
 +
|6.4TB Enterprise NVMe Mixed Use AG Drive U.2 Gen4 with carrier
 
|-
 
|-
−
| 4 || 8GB RDIMM, 3200 MT/s, Single Rank
+
|1
 +
|PowerEdge R7525 Motherboard, with 2 x 1Gb Onboard LOM (BCM5720)MLK V2
 
|-
 
|-
−
| 1 || AMD EPYC 7232P 3.10GHz, 8C/16T, 64M Cache (120W) DDR4-3200
+
|2
 +
|Dual, Hot-plug, Power Supply Redundant (1+1), 800W, Mixed Mode, NAF
 
|-
 
|-
−
| 1 || Dual Hot-Plug Redundant Power Supply (1+1), 550W
+
|1
 +
|Intel X710 Dual Port 10GbE SFP+, OCP NIC 3.0
 
|-
 
|-
−
| 2 || Jumper Cord - C13/C14, .6M, 250V, 13A
+
|1
 +
|Trusted Platform Module 2.0 V3
 
|}
 
|}
  
−
== Gen 2 ==
+
== Gen 1 Node Machine requirements ==
 +
If you're a Node Provider acquiring machines to join the IC, use the Gen2 specifications listed above. This section is for Gen1 Node Providers who joined before Genesis.
  
−
=== Generic specification ===
+
==== SuperMicro - Gen-1.1  ====
 
{| class="wikitable"
 
{| class="wikitable"
 
|-
 
|-
−
| Dual Socket AMD EPYC 7313 Milan 16C/32T 3 Ghz, 32K/512K/128M
+
| 1 || AS - 1023US - TR4
−
optionally 7343, 7373, 73F3
+
|-
 +
| 2 || Rome 7302 DP/UP 16C/32T 3.0
 +
|-
 +
| 16 || 32GB DDR4-3200 2Rx4 ECC REG DIMM
 
|-
 
|-
−
| 16x 32GB RDIMM, 3200MT/s, Dual Rank
+
| 5 || Samsung PM983 3.2TB NVMe PCIE/SATA Hybrid M.2 & 1 PCIE
 
|-
 
|-
−
| 5x 6.4TB NVMe Mixed Mode
+
| 2 || 800W Power Supply
 
|-
 
|-
−
| Dual Port 10G SFP or BASE-T
+
| 1 || Std LP 2-port 10G RJ45, Intel x540
 +
|-
 +
| 5 || Micron 5300 PRO 7.4TB, SATA, 2.5', 3D TLC, .6DWPD (with Caddie)
 
|-
 
|-
−
| TPM 2.0
+
| 1 || C13/C14 13A Power Cord
 
|}
 
|}
−
<br>
+
 
−
=== Validated Configurations ===
+
==== Dell - Gen-1.1  ====
−
DFINITY has [https://forum.dfinity.org/t/draft-motion-proposal-new-hardware-specification-and-remuneration-for-ic-nodes/14202/14?u=garym validated] the following Gen2 hardware configurations.
 
−
<br><br>
 
−
==== Validated configuration: Dell ====
 
 
{| class="wikitable"
 
{| class="wikitable"
 
|-
 
|-
−
| 2 || AMD EPYC 7343 3.2GHz, 16C/32T, 128M Cache (190W)
+
| 1 || R6525
 +
|-
 +
| 1 || Chassis - Supports Up to 10 NVMe drives, 12 drives total
 +
|-
 +
| 1 || Dual 1 GB on Motherboard
 +
|-
 +
| 3 || Low Profile PCIe Slots
 +
|-
 +
| - || 3 Year Basic NBD Support
 +
|-
 +
| - || iDrac Enterprise
 +
|-
 +
| 1 || Dual port 10GbE Base - T Adapter Broadcom, PCIe Low Profile
 
|-
 
|-
−
| 16 || 32GB RDIMM, 3200MT/s, Dual Rank 16Gb BASE x8
+
| 10 || 3.2TB NVMe, Mixed Use, 2.5" with Carrier
 
|-
 
|-
−
| 5 || 6.4TB Enterprise NVMe Mixed Use AG Drive U.2 Gen4 with carrier
+
| 16 || 32GB RDIMM (3200MT/s)
 
|-
 
|-
−
| 1 || PowerEdge R6525 Motherboard, with 2 x 1Gb Onboard LOM (BCM5720)MLK V2
+
| 2 || AMD 7302 3GHz, 16C/32T, 128M, 155W, 3200
 
|-
 
|-
−
| 2 || Dual, Hot-plug, Redundant Power Supply (1+1) 1100W, Mixed Mode Titanium
+
| 1 || Single Power Supply (800W)
 
|-
 
|-
−
| 1 || Intel X710 Dual Port 10GbE SFP+, OCP NIC 3.0
+
| 1 || C13-C14, 3M, 125V 15A Power Cored
 
|-
 
|-
−
| 1 || Trusted Platform Module 2.0 V3
 
 
|}
 
|}
−
<br>
 
  
−
==== Validated configuration: ASUS ====
+
==== Dell - Gen-1.2 ====
−
Note: SEV-SNP validation is still pending for the ASUS configuration.
 
 
{| class="wikitable"
 
{| class="wikitable"
 
|-
 
|-
−
| 2 || AMD EPYC 7313 (3,00 GHz, 16-Core, 128 MB)
+
| 1 || R6515
 +
|-
 +
| 1 || 3.5" Chassis with up to 4 Hot-Plug Hard Drives and OS RAID
 +
|-
 +
| 1 || Dual 1 Gb on Motherboard
 +
|-
 +
| 3 || Low Profile PCIe Slots
 +
|-
 +
| 1 || Standard Fan
 +
|-
 +
| - || 3 Year Basic NBD Support
 +
|-
 +
| - || iDrac Enterprise
 +
|-
 +
| 2 || Dual Port 10GbE Base - T Adapter Broadcom, PCIe LOw Profile
 
|-
 
|-
−
| 16 || 32GB ECC Reg ATP DDR4 3200 RAM
+
| 2 || 480GB SSD SATA Mix Use 6Gbps 512 2.5in Hot-Plug AG Drive, 3.5in
 
|-
 
|-
−
| 5 || 6.4 TB NVMe Kioxia SSD 3D-NAND TLC U.3 (Kioxia CM6-V)
+
| 4 || 8GB RDIMM, 3200 MT/s, Single Rank
 
|-
 
|-
−
| 1 || Asus Mainboard KMPP-D32 Series (without OCP 3.0, without Pike)
+
| 1 || AMD EPYC 7232P 3.10GHz, 8C/16T, 64M Cache (120W) DDR4-3200
 
|-
 
|-
−
| 2 || 1600 Watt redundant PSU
+
| 1 || Dual Hot-Plug Redundant Power Supply (1+1), 550W
 
|-
 
|-
−
| 1 || Broadcom 25 Gigabit P225P SFP28 Dual Port Network Card
+
| 2 || Jumper Cord - C13/C14, .6M, 250V, 13A
 
|}
 
|}
−
<br>
+
 
−
==== Validated configurations: Supermicro & Gigabyte ====
+
== Purchasing Hardware ==
−
We're re-running validation on Supermicro and Gigabyte machines which match the spec. This section will be updated when those results are ready.
+
# '''Choose your node machine configuration'''
 +
#* Stick with the configurations above. Anything else is unsupported.
 +
# '''Use local and trusted supply chains'''
 +
#* If possible, purchase hardware locally from a trusted vendor to avoid global single points of failure and to reduce the risk that somebody tampers with your hardware during delivery
 +
# '''Where to buy?'''
 +
#*[https://www.supermicro.com/en/wheretobuy Supermicro]
 +
#*[https://www.dell.com/en-us/shop/dell-poweredge-servers/sc/servers Dell]
 +
#*[https://www.gigabyte.com/WheretoBuy/Enterprise Gigabyte]
 +
#*[https://servers.asus.com/support/wheretobuy/117 ASUS]
 +
# '''Place your order'''
 +
#* Double check the components meet the generic specs above.
 +
 
 +
 
 +
 
 +
Lead times can be weeks to '''months'''. Plan accordingly with respect to onboarding.
 +
 
 +
Note: Ordering the hardware directly after the acceptance of your Node Provider and Data Center registration by the NNS is the safest option. However, in practice, if you are fully aware of the associated risks, ordering the hardware at an earlier stage can be done as well.

Revision as of 16:58, 29 September 2023

Node Providers operate one or more node machines than run in the Internet Computer network.

Gen1 hardware requirements have been used by Node Providers to set up node machines during the Genesis launch.

The Gen2 hardware requirements have been defined for the further growth of the Internet Computer network. The specifications for the Gen2 node machines are generic (instead of vendor specific) and support VM memory encryption and attestation, which will be needed in future features on the Internet Computer.

While Gen2 Node Providers are only strictly required to follow the generic specifications (the IC-OS installation will fail if the generic specifications are not met), it is strongly recommended for Node Providers to purchase one of the validated configurations listed below.

Gen 2 Node Machine requirements

Gen-2.3 Generic specification

Dual Socket AMD EPYC Milan CPU - Recommended: 7313 (16C/32T 3 Ghz)

optionally 7343, 7373, 73F3

16x 32GB RDIMM, 3200MT/s, Dual Rank
5x 6.4TB NVMe Mixed Mode (DWPD >= 3) with U.2 or U.3 interface
Dual Port 10G SFP or BASE-T
TPM 2.0

Important: Do NOT order a RAID controller for your node machine. If a RAID controller is included in your setup, it will cause issues with the installation of IC-OS on your machine.

Dfinity-validated configurations

DFINITY has validated the following Gen2 hardware configurations.

Dell

2 AMD EPYC 7343 3.2GHz, 16C/32T, 128M Cache (190W)
16 32GB RDIMM, 3200MT/s, Dual Rank 16Gb BASE x8
5 6.4TB Enterprise NVMe Mixed Use AG Drive U.2 Gen4 with carrier
1 PowerEdge R6525 Motherboard, with 2 x 1Gb Onboard LOM (BCM5720)MLK V2
2 Dual, Hot-plug, Redundant Power Supply (1+1) 1100W, Mixed Mode Titanium
1 Intel X710 Dual Port 10GbE SFP+, OCP NIC 3.0
1 Trusted Platform Module 2.0 V3

ASUS

2 AMD EPYC 7313 (3,00 GHz, 16-Core, 128 MB)
16 32GB ECC Reg ATP DDR4 3200 RAM
5 6.4 TB NVMe Kioxia SSD 3D-NAND TLC U.3 (Kioxia CM6-V)
1 Asus Mainboard KMPP-D32 Series (without OCP 3.0, without Pike)
2 1600 Watt redundant PSU
1 Broadcom 25 Gigabit P225P SFP28 Dual Port Network Card
0 TPM 2.0*

* We (Dfinity) forget to order the TPM 2.0 on the ASUS test machine we validated. Your machine DOES need the TPM 2.0.

Supermicro

2 AMD EPYC 7313 (3,00 GHz, 16-Core, 128 MB)
16 32GB ECC Reg DDR4 3200 RAM
5 6.4 TB NVMe TLC SSD, PCIe 4.0 x4, U.3 2.5", 3 DWPD
1 SuperMicro H12DSU-iN
2 1200 Watt redundant PSU
1 Dual Port 10GBase-T Network Adapter Intel® X710-TM4*
1 Dual Port 10GbE Network Adapter Intel® X710-TM4, SFP+*
1 Trusted Platform Module 2.0

* Only one dual port network adaptor is required.

Gigabyte

2 AMD EPYC 7313 (3,00 GHz, 16-Core, 128 MB)
16 32GB ECC Reg DDR4 3200 RAM
5 6.4 TB NVMe TLC SSD, PCIe 4.0 x4, U.3 2.5", 3 DWPD
1 Gigabyte MZ92-FS1 Rev. 3.0
2 1200 Watt redundant PSU
1 Dual Port 1000Base-T Network Adapter Intel® I350-AM2*
1 Dual Port 10GbE Network Adapter, Broadcom NetXtreme E-Series*
1 Trusted Platform Module 2.0

* Only one dual port network adaptor is required.

Community-validated configurations

Node Providers have validated the following Gen2 hardware configurations:

Dell PowerEdge R7525

2 AMD 7313 3.0GHz,16C/32T,128M,155W,3200
16 32GB RDIMM, 3200MT/s, Dual Rank 16Gb BASE x8
5 6.4TB Enterprise NVMe Mixed Use AG Drive U.2 Gen4 with carrier
1 PowerEdge R7525 Motherboard, with 2 x 1Gb Onboard LOM (BCM5720)MLK V2
2 Dual, Hot-plug, Power Supply Redundant (1+1), 800W, Mixed Mode, NAF
1 Intel X710 Dual Port 10GbE SFP+, OCP NIC 3.0
1 Trusted Platform Module 2.0 V3

Gen 1 Node Machine requirements

If you're a Node Provider acquiring machines to join the IC, use the Gen2 specifications listed above. This section is for Gen1 Node Providers who joined before Genesis.

SuperMicro - Gen-1.1

1 AS - 1023US - TR4
2 Rome 7302 DP/UP 16C/32T 3.0
16 32GB DDR4-3200 2Rx4 ECC REG DIMM
5 Samsung PM983 3.2TB NVMe PCIE/SATA Hybrid M.2 & 1 PCIE
2 800W Power Supply
1 Std LP 2-port 10G RJ45, Intel x540
5 Micron 5300 PRO 7.4TB, SATA, 2.5', 3D TLC, .6DWPD (with Caddie)
1 C13/C14 13A Power Cord

Dell - Gen-1.1

1 R6525
1 Chassis - Supports Up to 10 NVMe drives, 12 drives total
1 Dual 1 GB on Motherboard
3 Low Profile PCIe Slots
- 3 Year Basic NBD Support
- iDrac Enterprise
1 Dual port 10GbE Base - T Adapter Broadcom, PCIe Low Profile
10 3.2TB NVMe, Mixed Use, 2.5" with Carrier
16 32GB RDIMM (3200MT/s)
2 AMD 7302 3GHz, 16C/32T, 128M, 155W, 3200
1 Single Power Supply (800W)
1 C13-C14, 3M, 125V 15A Power Cored

Dell - Gen-1.2

1 R6515
1 3.5" Chassis with up to 4 Hot-Plug Hard Drives and OS RAID
1 Dual 1 Gb on Motherboard
3 Low Profile PCIe Slots
1 Standard Fan
- 3 Year Basic NBD Support
- iDrac Enterprise
2 Dual Port 10GbE Base - T Adapter Broadcom, PCIe LOw Profile
2 480GB SSD SATA Mix Use 6Gbps 512 2.5in Hot-Plug AG Drive, 3.5in
4 8GB RDIMM, 3200 MT/s, Single Rank
1 AMD EPYC 7232P 3.10GHz, 8C/16T, 64M Cache (120W) DDR4-3200
1 Dual Hot-Plug Redundant Power Supply (1+1), 550W
2 Jumper Cord - C13/C14, .6M, 250V, 13A

Purchasing Hardware

  1. Choose your node machine configuration
    • Stick with the configurations above. Anything else is unsupported.
  2. Use local and trusted supply chains
    • If possible, purchase hardware locally from a trusted vendor to avoid global single points of failure and to reduce the risk that somebody tampers with your hardware during delivery
  3. Where to buy?
  4. Place your order
    • Double check the components meet the generic specs above.


Lead times can be weeks to months. Plan accordingly with respect to onboarding.

Note: Ordering the hardware directly after the acceptance of your Node Provider and Data Center registration by the NNS is the safest option. However, in practice, if you are fully aware of the associated risks, ordering the hardware at an earlier stage can be done as well.

Retrieved from "https://wiki.internetcomputer.org/w/index.php?title=Node_Provider_Machine_Hardware_Guide&oldid=6470"