RMU build Gen-1.5: IV. Best Practices - C. Dell OpenManage Enterprise

From Internet Computer Wiki
Revision as of 20:39, 29 February 2024 by Katie.peters (talk | contribs) (Finising the page)
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to: navigation, search

This page is part of the Gen-1.5 RMU build runbook.

Go back to the previous section: B. Install RACADM Tool on RMU

C. Dell OpenManage Enterprise (Recommended)

This is a service that allows you to talk to the Baseboard Management Controller(s) (BMC) of your IC nodes. This service will allow you to discover your Dell’s servers iDRACs (Dell’s version of a BMC.) Once configured, you will be able to apply firmware updates remotely.

Create VM

  1. From the main Proxmox page, select Create VM to launch the Create: Virtual Machine dialog window.
  2. Tick the Advanced box and the Start at boot: box, then enter VM ID of 123, set Name: to <dc>-ome. Enter 10 into the Start/Shutdown Order:. Select Next.
    • OME - VM Create - 01 - General.png
  3. Select Do not use any media. Set Type: to Other. Select Next.
    • OME - VM Create - 02 - OS.png
  4. Ensure that Graphic card: is set to Default, SCSI Controller: is set to VirtIO SCSI single, Machine: is set to Default (i440fx), BIOS: is set to Default (SeaBIOS). Select Next.
    • OME - VM Create - 03 - System.png
  5. Ensure that Bus/Device: is set to IDE & 0, Storage: is set to local-zfs, Disk size (GiB): is set to 8, Backup: box is ticked, Async IO: is set to Default (io_uring). Select Next.
    • Screenshot 2024-01-22 at 2.50.42 PM.png
  6. Set Sockets to 1, set Type: to X86-64-v2-AES, set Cores: to 8. Select Next.
    • OME - VM Create - 05 - CPU.png
  7. Set Memory(MiB): to 16767 and Minimum memory (MB): to 2048. Select Next.
    • OME - VM Create - 06 - Memory.png
  8. Set Bridge: to vmbr1, set Mode: to Intel E1000, Disable the Firewall: tickbox. Select Next.
    • OME - VM Create - 07 - Network.png
  9. Do NOT tick the Start after created because we need to map the (soon to be downloaded) OME image to a disk. Select Finish.
    • OME - VM Create - 08 - Confirm.png

Download & Import OME

  1. On the RMU, download OME virtual image
  2. Open a shell console on the RMU
    wget --user-agent="Mozilla" -O ome.zip 'https://dl.dell.com/FOLDER07474001M/1/openmanage_enterprise_kvm_format_3.6.1.zip?uid=c802e350-6536-4f28-7a66-93b4f844cd30&fn=openmanage_enterprise_kvm_format_3.6.1.zip'
    
  3. Unzip the downloaded file
    apt update && apt install unzip -y
    unzip ome.zip
    
  4. Import the QCOW2 image into the OEM VM’s space
    cd appliance/qemu-kvm/
    qm importdisk 123 openmanage_enterprise.qcow2 local-zfs
    
    • Screenshot 2023-12-19 at 1.31.39 PM.png
  5. Configure the imported disk as the boot disk for the VM
  6. Select the OME VM on the left side
  7. Select Hardware
  8. Double click Unused Disk
    • OME - VM Create - 09 - Add Unused Disk 01.png
  9. Set Bus/Device: to VirtIO Block. Select Add
    • OME - VM Create - 10 - Add Unused Disk 02.png
  10. Select Options and double Click Boot Order
    • OME - VM Create - 11 - Update Boot Order 01.png
  11. Move the device with description local-zfs:vm-123-disk-1… to position 1. Tick the enable box next to it.
    • OME - VM Create - 12 - Update Boot Order 02.png
  12. Untick the enable box for all other boot devices Select OK
    • OME - VM Create - 13 - Update Boot Oder 03.png
  13. When complete, your options screen should look like the following
    • OME - VM Create - 14 - Start VM.png
  14. Select the OME VM and start it.

Configure OME

  1. Select the OME VM and Select Console
  2. Wait for the VM to start, please be patient as this is it’s first boot. (You can use your tab key to scroll through the options available and hit enter to select)
  3. Select Accept
    • OME - VM Create - 15 - Accept EULA.png
  4. Select your keyboard type
    • OME - VM Create - 16 - Select Keyboard.png
  5. Create a non trivial password and store it in your password manager.
  6. Once you enter in your password, hit tab to scroll over apply and then enter to select.
    • OME - VM Create - 17 - Set Default Password.png
  7. Use your arrow keys to scroll down to Set Networking Parameters and hit enter
    • OME - VM Create - 18 - Configure OME Networking 01.png
  8. Hit enter on the available Network adapter
    • OME - VM Create - 19 - Configure OME Networking 02.png
  9. Enter in your new non trivial password you just made.
  10. Hit tab to scroll over Continue and hit enter
    • OME - VM Create - 20 - Configure OME Networking 03.png
  11. Set static IPv4 IP Address to 10.10.100.23
    • You can use the arrow keys to get to DHCP and then hit tab to enter the editable text fields
  12. Set Static Gateway to 10.10.100.1
  13. Set Static Subnet Mask to 255.255.255.0
  14. Set Static Preferred DNS server to 1.1.1.1
  15. Select Apply
    • Screenshot 2023-12-19 at 2.47.16 PM.png
  16. Enter in your non trivial password and select Continue
    • Screenshot 2023-12-19 at 2.47.34 PM.png

Share Services with Teleport

  1. Open a shell on your RMU server and edit /etc/teleport.yaml with your favorite text editor
  2. Add the following section under the app_service -> apps stanza
    • Make sure your indention follows the previous name: bo1-rmu section
    • Replace the instances of “bo1” and .dfinity.network with the placeholders used earlier in the runbook
      - name: bo1-ome
            uri: https://10.10.100.23:443
            public_addr: "bo1-ome.teleport.bo1.dfinity.network"
            insecure_skip_verify: true
            rewrite:
              redirect:
              - "10.10.100.23"
              - "bo1-ome.teleport.bo1.dfinity.network"
            labels:
              dc: "bo1"
      
  3. Then, reload the teleport services (WARNING: if you are accessing the RMU shell via teleport, your connection will get disconnected):
    systemctl restart teleport
    

Configure OME Web GUI

  1. Go to your Teleport Resources page and find the tile for OME.
  2. Select it to launch the web page, you should be brought to the following screen.
  3. Input your credentials NOTE: admin is the default user for this service, input the password you made during this step.
    • OME - VM Create - 23 - OME Initial Configuration 01.png
  4. Select Initial Settings under Step 1
    • OME - VM Create - 24 - OME Initial Configuration 02.png
  5. Expand Time Configuration and enter in your timezone settings
    • We recommend to use UTC. That's what DFINITY as uses, and if you have more than 1 data center, it's nice to use one timezone for all. You may, however use whatever you like.
    • OME - VM Create - 25 - OME Initial Configuration 03.png

Discover Your Nodes

  1. On Step 2 Select Discover Devices which should bring up the Create Discovery Job Dialog Box.
  2. Under Device Type, click the drop down and select Server.
    • Ensure that Dell iDRAC is selected and click OK.
    • Screenshot 2024-01-03 at 1.27.17 PM.png
  3. Under IP/Hostname/Range enter 10.10.100.53-10.10.100.99.
  4. Under Services API Credentials enter in the user name and password for the iDRACs. Note: If you have a different password for each iDRAC, each one will need its own box, hit add and only put the single IP
  5. Select Finish
    • If not all nodes are discovered on the first attempt, you can try again using a narrower range or you can add each host separate with its own password (if not all servers have the same iDRAC password)
      • Screenshot 2024-01-03 at 1.25.43 PM.png
  6. You can monitor the status of the discovery by selecting monitor in the top navigation bar and then jobs
    • Screenshot 2024-01-03 at 1.23.33 PM.png
    • Screenshot 2024-01-03 at 1.32.11 PM.png

Create a baseline

  1. Select Configuration and then Firmware/Driver Compliance
    • Screenshot 2024-02-12 at 3.04.13 PM.png
  2. Select Create Baseline
    • Screenshot 2024-02-12 at 3.08.10 PM.png
  3. In the Create Update Baseline dialog box, select Add next to Catalog
  4. Input a name for the catalog, IE Dell Catalog
  5. Ensure that Latest Component versions on Dell.com is enabled
  6. Select Finish
    • Screenshot 2024-02-12 at 3.19.09 PM.png
  7. Input a Baseline name, we recommend including the <dc> name
  8. Select Next
    • Screenshot 2024-02-12 at 3.33.47 PM.png
  9. Select Select Devices
    • Screenshot 2024-02-12 at 3.36.06 PM.png
  10. Select all devices by clicking the small box, and then ticking Across all pages
  11. Select OK
    • Screenshot 2024-02-12 at 3.41.47 PM.png
  12. Select Finish

Applying Firmware Updates

Single Server

  1. Select Devices
    • Screenshot 2024-02-12 at 4.28.17 PM.png
  2. Select the IP address within the Name column to select a single device
  3. Select Firmware/Drivers
  4. Select the baseline you made earlier in the dropdown
    • Screenshot 2024-02-12 at 4.32.16 PM.png
  5. Tick all the firmware updates you wish to apply
    • Screenshot 2024-02-12 at 4.33.27 PM.png
  6. Select Update
  7. Select Update Now
  8. Set Reboot Server immediately. to Graceful Reboot with Forced Shutdown
  9. Tick Reset idrac
  10. Tick Clear Job Queue
  11. Select Update
    • Screenshot 2024-02-12 at 4.35.19 PM.png

.

Multiple Servers

  1. Select Configuration and then Firmware/Driver Compliance
    • Screenshot 2024-02-12 at 4.59.30 PM.png
  2. Tick the box next to the baseline we made earlier
  3. Select Check Compliance to ensure that available firmware updates are up to date
  4. Select View Report
    • Screenshot 2024-02-12 at 5.01.31 PM.png
  5. Tick the boxes next to the servers you wish to update
  6. Select Make Compliant
  7. Select Update Now
  8. Set Reboot Server immediately. to Graceful Reboot with Forced Shutdown
  9. Tick Reset idrac
  10. Tick Clear Job Queue
  11. Select Update
    • Screenshot 2024-02-12 at 5.03.18 PM.png

_

Continue to next section: D. Update Firmware on SuperMicro Nodes

Or skip and jump to E. Enable Remote Access to Servers Console

Go to the index of the Gen-1.5 RMU build runbook.