Difference between revisions of "Node Provider Machine Hardware Guide"
Gary.mcelroy (talk | contribs) (ASUS Validation Complete!) |
m (Add quantity / item description table headings) |
||
(41 intermediate revisions by 9 users not shown) | |||
Line 1: | Line 1: | ||
− | + | Node Providers operate one or more node machines than run in the Internet Computer network. | |
− | + | ||
− | + | Gen1 hardware requirements have been used by Node Providers to set up node machines during the Genesis launch. | |
− | == Gen | + | |
− | === | + | The Gen2 hardware requirements have been defined for the further growth of the Internet Computer network. The specifications for the Gen2 node machines are generic (instead of vendor specific) and support VM memory encryption and attestation, which will be needed in future features on the Internet Computer. |
+ | |||
+ | While Gen2 Node Providers are only strictly required to follow the generic specifications '''(the IC-OS installation will fail if the generic specifications are not met)''', it is strongly recommended for Node Providers to purchase one of the validated configurations listed below. | ||
+ | == Gen 2 Node Machine requirements == | ||
+ | ==== Gen-2.3 Generic specification ==== | ||
{| class="wikitable" | {| class="wikitable" | ||
+ | |'''Quantity''' | ||
+ | |'''Item description''' | ||
+ | |- | ||
+ | |2 | ||
+ | | Dual Socket AMD EPYC Milan CPU - Recommended: [https://en.wikichip.org/wiki/amd/epyc/7313 7313] (16C/32T 3 Ghz) | ||
+ | [https://en.wikichip.org/wiki/amd/epyc#7003_Series_.28Zen_3.29 optionally] 7343, 7373, 73F3 | ||
+ | |- | ||
+ | |16 | ||
+ | | 32GB RDIMM, 3200MT/s, Dual Rank | ||
+ | |- | ||
+ | |5 | ||
+ | | 6.4TB NVMe Mixed Mode (DWPD >= 3) with U.2 or U.3 interface | ||
|- | |- | ||
− | | 1 || | + | |1 |
+ | | Dual Port 10G SFP or BASE-T | ||
+ | |- | ||
+ | |1 | ||
+ | | TPM 2.0 | ||
+ | |} | ||
+ | '''Important:''' Do NOT order a RAID controller for your node machine. If a RAID controller is included in your setup, it will cause issues with the installation of IC-OS on your machine. | ||
+ | |||
+ | === Dfinity-validated configurations === | ||
+ | DFINITY has [https://forum.dfinity.org/t/draft-motion-proposal-new-hardware-specification-and-remuneration-for-ic-nodes/14202/14?u=garym validated] the following Gen2 hardware configurations. | ||
+ | ==== Dell ==== | ||
+ | {| class="wikitable" | ||
+ | |'''Quantity''' | ||
+ | |'''Item description''' | ||
+ | |- | ||
+ | | 2 || AMD EPYC 7343 3.2GHz, 16C/32T, 128M Cache (190W) | ||
+ | |- | ||
+ | | 16 || 32GB RDIMM, 3200MT/s, Dual Rank 16Gb BASE x8 | ||
|- | |- | ||
− | | | + | | 5 || 6.4TB Enterprise NVMe Mixed Use AG Drive U.2 Gen4 with carrier |
|- | |- | ||
− | | 1 || | + | | 1 || PowerEdge R6525 Motherboard, with 2 x 1Gb Onboard LOM (BCM5720)MLK V2 |
|- | |- | ||
− | | | + | | 2 || Dual, Hot-plug, Redundant Power Supply (1+1) 1100W, Mixed Mode Titanium |
|- | |- | ||
− | | | + | | 1 || Intel X710 Dual Port 10GbE SFP+, OCP NIC 3.0 |
|- | |- | ||
− | | | + | | 1 || Trusted Platform Module 2.0 V3 |
+ | |} | ||
+ | |||
+ | ==== ASUS ==== | ||
+ | {| class="wikitable" | ||
+ | |'''Quantity''' | ||
+ | |'''Item description''' | ||
|- | |- | ||
− | | | + | | 2 || AMD EPYC 7313 (3,00 GHz, 16-Core, 128 MB) |
|- | |- | ||
− | | | + | | 16 || 32GB ECC Reg ATP DDR4 3200 RAM |
|- | |- | ||
− | | | + | | 5 || 6.4 TB NVMe Kioxia SSD 3D-NAND TLC U.3 (Kioxia CM6-V) |
|- | |- | ||
− | | | + | | 1 || Asus Mainboard KMPP-D32 Series (without OCP 3.0, without Pike) |
|- | |- | ||
− | | | + | | 2 || 1600 Watt redundant PSU |
|- | |- | ||
− | | 1 || | + | | 1 || Broadcom 25 Gigabit P225P SFP28 Dual Port Network Card |
|- | |- | ||
+ | |0 | ||
+ | |TPM 2.0* | ||
|} | |} | ||
− | < | + | <nowiki>*</nowiki> We (Dfinity) forget to order the TPM 2.0 on the ASUS test machine we validated. Your machine DOES need the TPM 2.0. |
− | === | + | |
+ | ==== Supermicro ==== | ||
{| class="wikitable" | {| class="wikitable" | ||
+ | |'''Quantity''' | ||
+ | |'''Item description''' | ||
|- | |- | ||
− | | | + | | 2 || AMD EPYC 7313 (3,00 GHz, 16-Core, 128 MB) |
|- | |- | ||
− | | | + | | 16 || 32GB ECC Reg DDR4 3200 RAM |
|- | |- | ||
− | | | + | | 5 || 6.4 TB NVMe TLC SSD, PCIe 4.0 x4, U.3 2.5", 3 DWPD |
|- | |- | ||
− | | | + | | 1 || SuperMicro H12DSU-iN |
|- | |- | ||
− | | 2 || | + | | 2 || 1200 Watt redundant PSU |
|- | |- | ||
− | | 1 || | + | | 1 || Dual Port 10GBase-T Network Adapter Intel® X710-TM4* |
|- | |- | ||
− | | | + | |1 |
+ | |Dual Port 10GbE Network Adapter Intel® X710-TM4, SFP+* | ||
|- | |- | ||
− | | 1 || | + | | 1 || Trusted Platform Module 2.0 |
|} | |} | ||
− | < | + | <nowiki>*</nowiki> Only one dual port network adaptor is required. |
− | === | + | |
+ | ==== Gigabyte ==== | ||
{| class="wikitable" | {| class="wikitable" | ||
+ | |'''Quantity''' | ||
+ | |'''Item description''' | ||
+ | |- | ||
+ | | 2 || AMD EPYC 7313 (3,00 GHz, 16-Core, 128 MB) | ||
|- | |- | ||
− | | | + | | 16 || 32GB ECC Reg DDR4 3200 RAM |
+ | |- | ||
+ | | 5 || 6.4 TB NVMe TLC SSD, PCIe 4.0 x4, U.3 2.5", 3 DWPD | ||
|- | |- | ||
− | | 1 || 3. | + | | 1 || Gigabyte MZ92-FS1 Rev. 3.0 |
|- | |- | ||
− | | | + | | 2 || 1200 Watt redundant PSU |
|- | |- | ||
− | | | + | | 1 || Dual Port 1000Base-T Network Adapter Intel® I350-AM2* |
|- | |- | ||
− | | 1 | | + | |1 |
+ | |Dual Port 10GbE Network Adapter, Broadcom NetXtreme E-Series* | ||
|- | |- | ||
− | | - || | + | | 1 || Trusted Platform Module 2.0 |
+ | |} | ||
+ | <nowiki>*</nowiki> Only one dual port network adaptor is required. | ||
+ | |||
+ | === Community-validated configurations === | ||
+ | Node Providers have validated the following Gen2 hardware configurations: | ||
+ | |||
+ | ==== Dell PowerEdge R7525 ==== | ||
+ | {| class="wikitable" | ||
+ | |'''Quantity''' | ||
+ | |'''Item description''' | ||
|- | |- | ||
− | | | + | |2 |
+ | |AMD 7313 3.0GHz,16C/32T,128M,155W,3200 | ||
|- | |- | ||
− | | | + | |16 |
+ | |32GB RDIMM, 3200MT/s, Dual Rank 16Gb BASE x8 | ||
|- | |- | ||
− | | | + | |5 |
+ | |6.4TB Enterprise NVMe Mixed Use AG Drive U.2 Gen4 with carrier | ||
|- | |- | ||
− | | | + | |1 |
+ | |PowerEdge R7525 Motherboard, with 2 x 1Gb Onboard LOM (BCM5720)MLK V2 | ||
|- | |- | ||
− | | | + | |2 |
+ | |Dual, Hot-plug, Power Supply Redundant (1+1), 800W, Mixed Mode, NAF | ||
|- | |- | ||
− | | 1 | | + | |1 |
+ | |Intel X710 Dual Port 10GbE SFP+, OCP NIC 3.0 | ||
|- | |- | ||
− | | 2 | + | |1 |
+ | |Trusted Platform Module 2.0 V3 | ||
|} | |} | ||
− | == Gen | + | == Gen 1 Node Machine requirements == |
+ | If you're a Node Provider acquiring machines to join the IC, use the Gen2 specifications listed above. This section is for Gen1 Node Providers who joined before Genesis. | ||
− | === | + | ==== SuperMicro - Gen-1.1 ==== |
{| class="wikitable" | {| class="wikitable" | ||
+ | |'''Quantity''' | ||
+ | |'''Item description''' | ||
|- | |- | ||
− | | | + | | 1 || AS - 1023US - TR4 |
− | |||
|- | |- | ||
− | | | + | | 2 || Rome 7302 DP/UP 16C/32T 3.0 |
+ | |- | ||
+ | | 16 || 32GB DDR4-3200 2Rx4 ECC REG DIMM | ||
+ | |- | ||
+ | | 5 || Samsung PM983 3.2TB NVMe PCIE/SATA Hybrid M.2 & 1 PCIE | ||
+ | |- | ||
+ | | 2 || 800W Power Supply | ||
|- | |- | ||
− | | | + | | 1 || Std LP 2-port 10G RJ45, Intel x540 |
|- | |- | ||
− | | | + | | 5 || Micron 5300 PRO 7.4TB, SATA, 2.5', 3D TLC, .6DWPD (with Caddie) |
|- | |- | ||
− | | | + | | 1 || C13/C14 13A Power Cord |
|} | |} | ||
− | + | ||
− | === | + | ==== Dell - Gen-1.1 ==== |
− | |||
− | |||
− | |||
{| class="wikitable" | {| class="wikitable" | ||
+ | |'''Quantity''' | ||
+ | |'''Item description''' | ||
+ | |- | ||
+ | | 1 || R6525 | ||
+ | |- | ||
+ | | 1 || Chassis - Supports Up to 10 NVMe drives, 12 drives total | ||
+ | |- | ||
+ | | 1 || Dual 1 GB on Motherboard | ||
+ | |- | ||
+ | | 3 || Low Profile PCIe Slots | ||
+ | |- | ||
+ | | - || 3 Year Basic NBD Support | ||
+ | |- | ||
+ | | - || iDrac Enterprise | ||
|- | |- | ||
− | | | + | | 1 || Dual port 10GbE Base - T Adapter Broadcom, PCIe Low Profile |
|- | |- | ||
− | | | + | | 10 || 3.2TB NVMe, Mixed Use, 2.5" with Carrier |
|- | |- | ||
− | | | + | | 16 || 32GB RDIMM (3200MT/s) |
|- | |- | ||
− | | | + | | 2 || AMD 7302 3GHz, 16C/32T, 128M, 155W, 3200 |
|- | |- | ||
− | | | + | | 1 || Single Power Supply (800W) |
|- | |- | ||
− | | 1 || | + | | 1 || C13-C14, 3M, 125V 15A Power Cored |
|- | |- | ||
− | |||
|} | |} | ||
− | |||
− | ==== | + | ==== Dell - Gen-1.2 ==== |
{| class="wikitable" | {| class="wikitable" | ||
+ | |'''Quantity''' | ||
+ | |'''Item description''' | ||
+ | |- | ||
+ | | 1 || R6515 | ||
+ | |- | ||
+ | | 1 || 3.5" Chassis with up to 4 Hot-Plug Hard Drives and OS RAID | ||
+ | |- | ||
+ | | 1 || Dual 1 Gb on Motherboard | ||
+ | |- | ||
+ | | 3 || Low Profile PCIe Slots | ||
+ | |- | ||
+ | | 1 || Standard Fan | ||
+ | |- | ||
+ | | - || 3 Year Basic NBD Support | ||
+ | |- | ||
+ | | - || iDrac Enterprise | ||
|- | |- | ||
− | | 2 || | + | | 2 || Dual Port 10GbE Base - T Adapter Broadcom, PCIe LOw Profile |
|- | |- | ||
− | | | + | | 2 || 480GB SSD SATA Mix Use 6Gbps 512 2.5in Hot-Plug AG Drive, 3.5in |
|- | |- | ||
− | | | + | | 4 || 8GB RDIMM, 3200 MT/s, Single Rank |
|- | |- | ||
− | | 1 || | + | | 1 || AMD EPYC 7232P 3.10GHz, 8C/16T, 64M Cache (120W) DDR4-3200 |
|- | |- | ||
− | | | + | | 1 || Dual Hot-Plug Redundant Power Supply (1+1), 550W |
|- | |- | ||
− | | | + | | 2 || Jumper Cord - C13/C14, .6M, 250V, 13A |
|} | |} | ||
− | + | ||
− | ==== | + | == Purchasing Hardware == |
− | + | # '''Choose your node machine configuration''' | |
+ | #* Stick with the configurations above. Anything else is unsupported. | ||
+ | # '''Use local and trusted supply chains''' | ||
+ | #* If possible, purchase hardware locally from a trusted vendor to avoid global single points of failure and to reduce the risk that somebody tampers with your hardware during delivery | ||
+ | # '''Where to buy?''' | ||
+ | #*[https://www.supermicro.com/en/wheretobuy Supermicro] | ||
+ | #*[https://www.dell.com/en-us/shop/dell-poweredge-servers/sc/servers Dell] | ||
+ | #*[https://www.gigabyte.com/WheretoBuy/Enterprise Gigabyte] | ||
+ | #*[https://servers.asus.com/support/wheretobuy/117 ASUS] | ||
+ | # '''Place your order''' | ||
+ | #* Double check the components meet the generic specs above. | ||
+ | |||
+ | |||
+ | |||
+ | Lead times can be weeks to '''months'''. Plan accordingly with respect to onboarding. | ||
+ | |||
+ | Note: Ordering the hardware directly after the acceptance of your Node Provider and Data Center registration by the NNS is the safest option. However, in practice, if you are fully aware of the associated risks, ordering the hardware at an earlier stage can be done as well. |
Latest revision as of 15:45, 28 May 2024
Node Providers operate one or more node machines than run in the Internet Computer network.
Gen1 hardware requirements have been used by Node Providers to set up node machines during the Genesis launch.
The Gen2 hardware requirements have been defined for the further growth of the Internet Computer network. The specifications for the Gen2 node machines are generic (instead of vendor specific) and support VM memory encryption and attestation, which will be needed in future features on the Internet Computer.
While Gen2 Node Providers are only strictly required to follow the generic specifications (the IC-OS installation will fail if the generic specifications are not met), it is strongly recommended for Node Providers to purchase one of the validated configurations listed below.
Gen 2 Node Machine requirements
Gen-2.3 Generic specification
Quantity | Item description |
2 | Dual Socket AMD EPYC Milan CPU - Recommended: 7313 (16C/32T 3 Ghz)
optionally 7343, 7373, 73F3 |
16 | 32GB RDIMM, 3200MT/s, Dual Rank |
5 | 6.4TB NVMe Mixed Mode (DWPD >= 3) with U.2 or U.3 interface |
1 | Dual Port 10G SFP or BASE-T |
1 | TPM 2.0 |
Important: Do NOT order a RAID controller for your node machine. If a RAID controller is included in your setup, it will cause issues with the installation of IC-OS on your machine.
Dfinity-validated configurations
DFINITY has validated the following Gen2 hardware configurations.
Dell
Quantity | Item description |
2 | AMD EPYC 7343 3.2GHz, 16C/32T, 128M Cache (190W) |
16 | 32GB RDIMM, 3200MT/s, Dual Rank 16Gb BASE x8 |
5 | 6.4TB Enterprise NVMe Mixed Use AG Drive U.2 Gen4 with carrier |
1 | PowerEdge R6525 Motherboard, with 2 x 1Gb Onboard LOM (BCM5720)MLK V2 |
2 | Dual, Hot-plug, Redundant Power Supply (1+1) 1100W, Mixed Mode Titanium |
1 | Intel X710 Dual Port 10GbE SFP+, OCP NIC 3.0 |
1 | Trusted Platform Module 2.0 V3 |
ASUS
Quantity | Item description |
2 | AMD EPYC 7313 (3,00 GHz, 16-Core, 128 MB) |
16 | 32GB ECC Reg ATP DDR4 3200 RAM |
5 | 6.4 TB NVMe Kioxia SSD 3D-NAND TLC U.3 (Kioxia CM6-V) |
1 | Asus Mainboard KMPP-D32 Series (without OCP 3.0, without Pike) |
2 | 1600 Watt redundant PSU |
1 | Broadcom 25 Gigabit P225P SFP28 Dual Port Network Card |
0 | TPM 2.0* |
* We (Dfinity) forget to order the TPM 2.0 on the ASUS test machine we validated. Your machine DOES need the TPM 2.0.
Supermicro
Quantity | Item description |
2 | AMD EPYC 7313 (3,00 GHz, 16-Core, 128 MB) |
16 | 32GB ECC Reg DDR4 3200 RAM |
5 | 6.4 TB NVMe TLC SSD, PCIe 4.0 x4, U.3 2.5", 3 DWPD |
1 | SuperMicro H12DSU-iN |
2 | 1200 Watt redundant PSU |
1 | Dual Port 10GBase-T Network Adapter Intel® X710-TM4* |
1 | Dual Port 10GbE Network Adapter Intel® X710-TM4, SFP+* |
1 | Trusted Platform Module 2.0 |
* Only one dual port network adaptor is required.
Gigabyte
Quantity | Item description |
2 | AMD EPYC 7313 (3,00 GHz, 16-Core, 128 MB) |
16 | 32GB ECC Reg DDR4 3200 RAM |
5 | 6.4 TB NVMe TLC SSD, PCIe 4.0 x4, U.3 2.5", 3 DWPD |
1 | Gigabyte MZ92-FS1 Rev. 3.0 |
2 | 1200 Watt redundant PSU |
1 | Dual Port 1000Base-T Network Adapter Intel® I350-AM2* |
1 | Dual Port 10GbE Network Adapter, Broadcom NetXtreme E-Series* |
1 | Trusted Platform Module 2.0 |
* Only one dual port network adaptor is required.
Community-validated configurations
Node Providers have validated the following Gen2 hardware configurations:
Dell PowerEdge R7525
Quantity | Item description |
2 | AMD 7313 3.0GHz,16C/32T,128M,155W,3200 |
16 | 32GB RDIMM, 3200MT/s, Dual Rank 16Gb BASE x8 |
5 | 6.4TB Enterprise NVMe Mixed Use AG Drive U.2 Gen4 with carrier |
1 | PowerEdge R7525 Motherboard, with 2 x 1Gb Onboard LOM (BCM5720)MLK V2 |
2 | Dual, Hot-plug, Power Supply Redundant (1+1), 800W, Mixed Mode, NAF |
1 | Intel X710 Dual Port 10GbE SFP+, OCP NIC 3.0 |
1 | Trusted Platform Module 2.0 V3 |
Gen 1 Node Machine requirements
If you're a Node Provider acquiring machines to join the IC, use the Gen2 specifications listed above. This section is for Gen1 Node Providers who joined before Genesis.
SuperMicro - Gen-1.1
Quantity | Item description |
1 | AS - 1023US - TR4 |
2 | Rome 7302 DP/UP 16C/32T 3.0 |
16 | 32GB DDR4-3200 2Rx4 ECC REG DIMM |
5 | Samsung PM983 3.2TB NVMe PCIE/SATA Hybrid M.2 & 1 PCIE |
2 | 800W Power Supply |
1 | Std LP 2-port 10G RJ45, Intel x540 |
5 | Micron 5300 PRO 7.4TB, SATA, 2.5', 3D TLC, .6DWPD (with Caddie) |
1 | C13/C14 13A Power Cord |
Dell - Gen-1.1
Quantity | Item description |
1 | R6525 |
1 | Chassis - Supports Up to 10 NVMe drives, 12 drives total |
1 | Dual 1 GB on Motherboard |
3 | Low Profile PCIe Slots |
- | 3 Year Basic NBD Support |
- | iDrac Enterprise |
1 | Dual port 10GbE Base - T Adapter Broadcom, PCIe Low Profile |
10 | 3.2TB NVMe, Mixed Use, 2.5" with Carrier |
16 | 32GB RDIMM (3200MT/s) |
2 | AMD 7302 3GHz, 16C/32T, 128M, 155W, 3200 |
1 | Single Power Supply (800W) |
1 | C13-C14, 3M, 125V 15A Power Cored |
Dell - Gen-1.2
Quantity | Item description |
1 | R6515 |
1 | 3.5" Chassis with up to 4 Hot-Plug Hard Drives and OS RAID |
1 | Dual 1 Gb on Motherboard |
3 | Low Profile PCIe Slots |
1 | Standard Fan |
- | 3 Year Basic NBD Support |
- | iDrac Enterprise |
2 | Dual Port 10GbE Base - T Adapter Broadcom, PCIe LOw Profile |
2 | 480GB SSD SATA Mix Use 6Gbps 512 2.5in Hot-Plug AG Drive, 3.5in |
4 | 8GB RDIMM, 3200 MT/s, Single Rank |
1 | AMD EPYC 7232P 3.10GHz, 8C/16T, 64M Cache (120W) DDR4-3200 |
1 | Dual Hot-Plug Redundant Power Supply (1+1), 550W |
2 | Jumper Cord - C13/C14, .6M, 250V, 13A |
Purchasing Hardware
- Choose your node machine configuration
- Stick with the configurations above. Anything else is unsupported.
- Use local and trusted supply chains
- If possible, purchase hardware locally from a trusted vendor to avoid global single points of failure and to reduce the risk that somebody tampers with your hardware during delivery
- Where to buy?
- Place your order
- Double check the components meet the generic specs above.
Lead times can be weeks to months. Plan accordingly with respect to onboarding.
Note: Ordering the hardware directly after the acceptance of your Node Provider and Data Center registration by the NNS is the safest option. However, in practice, if you are fully aware of the associated risks, ordering the hardware at an earlier stage can be done as well.