Gen-2 Data Center runbook

ICR Gen2 build - DFINITY-managed reference design

This is a work-in-progress

This runbook illustrates how DFINITY manages a typical rack of IC nodes (an 'ICR'). It is published for the benefit of node providers to show one possible implementation of the ICR Gen2 networking requirements.

This runbook is NOT mandatory for ICR builds that are not managed by DFINITY.

Prerequisites

Collect the following information for the variables that will need to be adjusted for your installation.

• site-ID (i.e. “zh2” or “mr1”)
• PDU outlet type in racks (IEC 60320 C13 or national power outlet)
• IPv4 subnet assignment for management network - a new /25 subnet has to be assigned by the node provider based on NP’s addressing plan (DFINITY-owned ICRs use 10.10.X.128/25 addressing scheme where X is assigned to each ICR)
• RMU or jump-box HW (server, power cords and 5x 2m (7ft) Cat6 cables and the required installation artifacts are available

Uplink configuration

Collect information about the uplinks and verify the minimum requirements:

1. Management Port
   • Assigned public IPv4 range (min /31): [FILL IN]
   • Default GW address: [FILL IN]
   • 1G/10G, media type fiber/copper: [FILL IN]
     • if fiber: multi-mode/single-mode: [FILL IN]
     • if fiber: patch panel connector type (SC/PC or LC/PC or E2000/APC …): [FILL IN]
     • if 1G fiber: required transceiver type (LX/SX/other): [FILL IN]
     • if 10G fiber: required transceiver type (LR/SR/other): [FILL IN]
2. Production Port
   • Assigned public IPv6 range (/64): [FILL IN]
   • IPv6 Default GW address: [FILL IN]
   • (optional) assigned public IPv4 range (min /29): [FILL IN]
   • (optional) IPv4 Default GW address: [FILL IN]
   • 1G/10G, media type fiber/copper: [FILL IN]
     • if fiber: multi-mode/single-mode: [FILL IN]
     • if fiber: patch panel connector type (SC/PC or LC/PC or E2000/APC …): [FILL IN]
     • if 1G fiber: required transceiver type (LX/SX/other): [FILL IN]
     • if 10G fiber: required transceiver type (LR/SR/other): [FILL IN]
     • if other fiber: required transceiver type: [FILL IN]
3. Location and the circuit IDs or patch-panel positions of the management and production ports: [FILL IN]
4. Number and location of the racks (1 or 2): [FILL IN]

HW requirements

Verify BOM for each rack:

• 1x Dell EMC S3048-ON with 2x AC PSU and PSU->ports airflow
• 1x Dell EMC S4148T-ON with 2x AC PSU and PSU->ports airflow (or alternative - see below)
• 4x power cords for the switches (PDU outlet type to C13)

Alternatives for S4148T-ON (can be used when servers have SFP+ or SFP28 cages):

• 1x Dell EMC S4148F-ON with 2x AC PSU and PSU->ports airflow
• 1x Dell EMC S5248F-ON with 2x AC PSU and PSU->ports airflow

Verify BOM for ICR:

• If management port is single-mode fiber:
• 1x management port fiber patch cord 9/125 LC/PC to patch-panel type connector
• 1x transceiver (SFP or SFP+) matching the management port required transceiver type (probably -LX or -LR)

If production port is single-mode fiber:

• 1x production port fiber patch cord 9/125 LC/PC to patch-panel type connector
• 1x transceiver (SFP or SFP+ or QSFP+ or QSFP28) matching the production port required transceiver type
• 40G QSFP+ to 10G SFP+ Adapter Converter Module if switch is S4148T-ON and the transceiver is SFP or SFP+

If management port is multi-mode fiber:

• 1x management port fiber patch cord 50/125 LC/PC to patch-panel type connector
• 1x transceiver (SFP or SFP+) matching the management port required transceiver type (probably -SX or -SR)

If production port is multi-mode fiber:

• 1x production port fiber patch cord 50/125 LC/PC to patch-panel type connector
• 1x transceiver (SFP or SFP+ or QSFP+ or QSFP28) matching the production port required transceiver type if production port is fiber (probably -SX or -SR)
• 40G QSFP+ to 10G SFP+ Adapter Converter Module if switch is S4148T-ON and the transceiver is SFP or SFP+

If management port is copper:

• 2m (7ft) Cat6 copper cable

If production port is copper:

• 2m (7ft) Cat6 copper cable
• 40G QSFP+ to 10G SFP+ Adapter Converter Module if switch is S4148T-ON and the transceiver is SFP or SFP+
• 10GBASE-T SFP+ if production switch is Dell EMC S4148F-ON or Dell EMC S5248F-ON

Verify BOM for 2-rack ICR:

If the DC rules allows running DAC or AOC cables between the racks and the position of the racks and the maximum length of the available AOC or DAC allows connecting the switches:

• 1x 100G QSFP28 AOC or DAC cable (matching the required length from one production switch to another - with neighboring racks it is usually 5m (16ft) )
• 2x Cat6 interconnects between the racks (matching the required length from one management switch to another and from RMU to the switch in the second rack, usually 5m (16ft))

Otherwise:

• 2x single-mode (9/125) optical paths between the racks (four fibers or two pairs in total)
• 2x 100G QSFP28 -LR4, -LR or -CWDM4 transceivers
• 2x 9/125 2m (7ft) patch cords LC/PC to connector of the optical path termination
• 2x Cat6 interconnects between the racks
• 4x Cat6 2m (7ft) patch cables

Marking

• (virtually) mark the rack with both management and production uplink Rack 1
• (virtually) mark the rack with no uplinks Rack 2 (if the rack exists)
• Mark the production switch in Rack 1: {site-ID}-sw02
• Mark the production switch in Rack 2: {site-ID}-sw04 (if the rack exists)
• Mark the management switch in the Rack 1: {site-ID}-msw01
• Mark the management switch in the Rack 2: {site-ID}-msw02 (if the rack exists)

Rack and stack devices

Racking and stacking of devices is beyond the scope of this runbook.

The site should include the following components:

• Rack 1
  • {site-ID}-sw02
  • {site-ID}-msw01
  • RMU / jump-box
  • 0-14x servers (IC nodes)
• Rack 2 (if present)
  • {site-ID}-sw04
  • {site-ID}-msw02
  • 0-14x servers (IC nodes)

Cabling

Minimum required steps

• In each rack connect PSUs of each switch the PDUs using the power cords, select different power rails for PSU1 and PSU2
• In Rack 1 connect the production uplink ([carrier] Internet) to {site-ID}-sw02 port 25 using the selected transceiver and 40G QSFP+ to 10G SFP+ Adapter Converter Module if switch is S4148T-ON and the transceiver is SFP or SFP+
• In Rack 1 connect the management uplink to RMU/jump-box port wan (for jump-box TBD runbook)
• In Rack 1 connect Cat6 cable from RMU port management1 to {site-ID}-msw01 port 46
• In Rack 1 connect Cat6 cable from RMU port management2 to {site-ID}-msw01 management port
• In Rack 1 connect Cat6 cable from RMU port lan1 to {site-ID}-sw02 port 54
• In Rack 1 connect Cat6 cable from {site-ID}-sw02 management port to {site-ID}-msw01 port 47

If Rack 2 is present:

• Connect AOC/DAC or fiber optical path with QSFP28 transceivers from Rack 1 {site-ID}-sw02 port 30 to Rack 2 {site-ID}-sw04 port 30
• Connect Cat6 cable from Rack 1 {site-ID}-mw01 port 48 to Rack 2 {site-ID}-msw02 port 48
• Connect Cat6 cable from Rack 1 RMU port management3 to Rack 2 {site-ID}-msw02 management port
• Connect Cat6 cable from port {site-ID}-msw02 port 47 to Rack 2 {site-ID}-sw04 management port

Wiring diagram

Servers

The exact cabling instructions for servers are not part of this runbook because the servers differ in the port type, number of ports and location of ports. High-level requirements for each server:

• Connect the BMC / IPMI / iLO port of the server to the management switch in the same rack (allocate ports from left to right, take the first free ports with the matching speed and port type)
• Connect the first 10G/25G port to the production switch in the same rack (allocate ports from left to right, take the first free ports with the matching speed and port type)
• Use Cat6 cables or PatchBox Cat6 modules for connecting the servers if 10GBASE-T is supported by the switch and the server or at least one side; use 10GBASE-T SFP+ transceivers to adapt switch or server side with SFP+ or SFP28 interfaces
• Use AOC or DAC cables in case both the switch and the server have SFP+ or SFP28 interfaces; select the matching speed and type of the cable for the interface and verify the vendor compatibility on switch side and also on server side

Dell OS10 NOS install

Verify that Dell OS10 (version >=10.4) is installed on all switches.

• If no NOS is installed or older version is installed, use NOS installation guide:

https://www.dell.com/support/manuals/en-us/force10-s4048-on/ee-upgrade-downgrade/installing-smartfabric-os10?guid=guid-9bf59a6c-9be9-4abb-99cf-b2671091f3e0&lang=en-us

• It is suggested to install the OS10 from USB stick as described in the “Manual installation” section:

https://www.dell.com/support/manuals/en-us/smartfabric-os10-emp-partner/ee-upgrade-downgrade/manual-installation?guid=guid-d4a157a0-e1fc-4ad7-bb68-cd98fdcc0025&lang=en-us

• For upgrading OS10 see the instructions in ”Upgrading OS10 software” chapter: https://www.dell.com/support/manuals/en-sg/smartfabric-os10-emp-partner/ee-upgrade-downgrade/upgrading-os10-software?guid=guid-29a7887c-d5ed-4896-9cc6-9dcd614c0aee&lang=en-us

Dell OS10 switch minimum configuration

• All switches are active and can be connected right after the NOS installation. Further configuration is required to set user accounts, re-set default passwords and switch hardening.

Prerequisites:

• Generate a password for admin user
• Generate a password for linuxadmin user
• (optional) Collect SSH key for admin user
• Prepare serial console connection to the switch (see “Log in to an OS10 switch” chapter: https://www.dell.com/support/manuals/en-sg/dell-emc-smartfabric-os10/ee-upgrade-downgrade/log-in-to-an-os10-switch?guid=guid-977e7f9f-3175-49b4-a0bc-5e8a15d8c424&lang=en-us ) or collect the IP address from DHCP server if the switch has been auto-installed
• Assign management network IP addresses (following example is based on 10.10.X.128/25 assignment - DFINITY-only address plan):
  • 10.10.X.254 - RMU or jump-box acting as default GW
  • 10.10.X.140 - {site-ID}-msw01
  • 10.10.X.141 - {site-ID}-msw02
  • 10.10.X.142 - {site-ID}-sw02
  • 10.10.X.144 - {site-ID}-sw04

Procedure:

• Connect to the switch using the serial console and screen {switch tty device - i.e. /dev/ttyUSB0} 115200 (Linux) or PuTTY (Windows)
• Enter configuration mode:

{site-ID}-{switch name}# configure terminal
{site-ID}-{switch name}(config)#

Clear interface mgm1/1/1:

interface mgmt1/1/1
shutdown
no ip address dhcp
no ipv6 address autoconfig

Configure management VRF and the management interface

ip vrf management
interface management
!
interface mgmt1/1/1
no shutdown
no ip address dhcp
ip address {switch IP address - 10.10.X.Y/25}
ipv6 address autoconfig
!
management route 0.0.0.0/0 10.10.X.254

Configure users and basic configuration

ip http vrf management
default mtu 9216
hostname {site-ID}-{switch name}
system-user linuxadmin password {linuxadmin password}
ip name-server vrf management 1.1.1.1
username admin password {admin password} role sysadmin priv-lvl 15
username admin sshkey "{admin SSH key}"
snmp-server community public ro 
snmp-server contact "Contact Support"
ntp server pool.ntp.org
ntp source mgmt1/1/1
ntp enable vrf management

Tests

• Ping from RMU / jump-box to the switch management interfaces:

ping -c4 10.10.X.140
ping -c4 10.10.X.141 (if installed)
ping -c4 10.10.X.142
ping -c4 10.10.X.144 (if installed)

• Test server BMC and IPv6 connectivity
  • Deploy at least one server in each rack, collect BMC IP address in the management network and the server IP in production network
  • Ping the collected server IPs from RMU / jump-box

Dell OS10 operation recommendations

The details of Dell OS10-based switch operation is beyond the scope of this runbook. The ultimate responsibility for the network connection availability, quality and security lies with the node provider. The qualitative parameters depend among others also on the switching fabric performance and health. Therefore it is strongly recommended to implement the following steps:

• • Overall health of all switches should be watched - PSU, fan and memory and CPU load (SNMP-based monitoring or gRPC Streaming Telemetry)
  • Port load of all switches should be watched (using gRPC Streaming Telemetry)
  • New versions of OS10, new known issues and security advisories should be periodically (once a month) evaluated and upgrades should be scheduled when there is a relevant issue or enhancement in the newly available version
  • The HW lifetime upgrade path should be followed according to the vendor’s recommendations

See Also

• The Internet Computer project website (hosted on the IC): internetcomputer.org