Skip to content

Lab rebuild for IBM Fusion HCI Rack #411

Closed

Description

@rbo

Hey everyone,

Quick heads-up regarding the power outage of Munich Office on Friday, September 25th: neither Daniel M. nor I will be in the office that day.

Here is the plan for the lab:

Thursday afternoon (Sep 24): I will shut down the lab completely and power everything off.
Tuesday & Wednesday (Sep 29–30): Since everything will already be off, I’d like to use this window to do the planned lab rebuild (swapping out the racks) once I'm back in the office.
Thursday (Oct 1): The lab will be fully booted back up and operational by Thursday at the latest.

Note: If anyone urgently needs access earlier, we could potentially bring parts of the lab back online sooner with a bit of extra effort—just reach out!

Does anyone see any issues with this schedule or have critical dependencies? Please let me know!

Slack: https://redhat-internal.slack.com/archives/C04J8QF8Y83/p1789646062716079

Planning

Documentation of some "special cases":

Server/Env Shutdown Power On
BR Server 2026-09-24
Metal 5 2026-09-24 Try to power on Friday 2026-09-25, including relocation)
vPACK 2026-09-24
ISAR 2026-09-24 - late as possible
storm1 2026-09-24
storm4 2026-09-24 - late as possible because PoC

Time line

Thu 2026-09-24 Fri 2026-09-45 Tue 2026-09-29 & Wed 2026-09-30
Shutdown entire lab Power maintenance in the Office Rebuild the lab, because if new IBM Rack

Activity

  1. self-assigned this
    on Sep 18, 2026
  2. rbo commented on Sep 18, 2026

    @rbo
    MemberAuthor

    Current rack layout

    Image
  3. rbo commented on Sep 23, 2026

    @rbo
    MemberAuthor

    inf20 / inf3 - telco lab cableing

    Image
    Server Switch Port
    inf20-1 redhat-switch 25
    inf20-2 arista-sfp 33
    inf20-3 arista-sfp 35
    inf20-4 redhat-switch 29
    inf20-5 arista-sfp 36
    inf20-6 arista-sfp 34
    inf20-7 arista-sfp 11-1
    inf20-8 arista-sfp 11-4
    inf3-1 arista-sfp 28-2
    inf3-4 arista-sfp 28-3
  4. rbo commented on Sep 24, 2026

    @rbo
    MemberAuthor

    Lab is off

    NetApp shutdown:

    Warning: Permanently added '10.32.104.23' (ED25519) to the list of known hosts.
    (admin@10.32.104.23) Password:
    (admin@10.32.104.23) Password:
    
    Last login time: 6/17/2026 11:42:44
    Unsuccessful login attempts since last login: 2
    fas2552::> system node halt -node * -inhibit-takeover true -skip-lif-migration-before-shutdown true -ignore-quorum-warnings true
    
    Warning: Are you sure you want to halt node "fas2552-01"? {y|n}:
    {y|n}: y
    
    Warning: Are you sure you want to halt node "fas2552-02"? {y|n}: y
    2 entries were acted on.
    
    
    Connection to 10.32.104.23 closed.
    
  5. changed the title [-]Planning: Power Outage & Planned Lab Maintenance (Sep 24)[/-] [+]Lab rebuild for IBM Fusion HCI Rack[/+] on Oct 18, 2026
  6. rbo commented on Oct 18, 2026

    @rbo
    MemberAuthor

    Report of Switch (arista-sfp & arista-rj45) configuration

    ⚡ Special Ports (arista-rj45)

    Port Description Mode Config Details
    Ethernet17 broken shutdown Port is administratively disabled
    Ethernet27 power91-1 mgmt - hmc1 access VLAN 241 (ibm-power-hmc1)
    Ethernet28 power91-2 mgmt - hmc2 access VLAN 242 (ibm-power-hmc2)
    Ethernet29 power92-1 mgmt - hmc1 access VLAN 241 (ibm-power-hmc1)
    Ethernet30 power92-2 mgmt - hmc2 access VLAN 242 (ibm-power-hmc2)
    Ethernet38 (none) access VLAN 2025 (user-vlan-2025)
    Ethernet52/1–52/4 port-group-11, link to arista-sfp LAG member channel-group 11 mode active (Port-Channel 11)
    Management1 — routed IP 10.32.104.2/20

    ⚡ Special Ports (arista-sfp)

    Port Description Mode Config Details
    Ethernet5/1–5/4 (none) switchport only No trunk/access config — effectively default VLAN 1
    Ethernet13/1 inf20 MLX enp66s0f0np0 — keep separated bcf vlan trunk Native VLAN 32, allowed VLANs 2-54,56-238,240,243-4094 (nearly all VLANs!), VLAN translation 3333→32
    Ethernet24/4 inf20 MLX enp66s0f0np0 trunk Standard allowed VLANs, speed forced 10000full
    Ethernet29 Link to IBM Fusion HCI LAG member channel-group 250 mode active (Port-Channel 250)
    Ethernet30 Link to IBM Fusion HCI LAG member channel-group 250 mode active (Port-Channel 250)
    Ethernet34 inf20 e810 enp5s0f0 trunk Standard allowed VLANs, speed forced 10000full
    Ethernet36 inf20 e810 enp5s0f1 trunk Native VLAN tag (not 32), speed forced 10000full, PTP enabled (layer2, master role)
    Ethernet19/1–19/4 Link to arista-rj45 LAG member channel-group 11 mode active (Port-Channel 11)
    Management1 — routed IP 10.32.104.3/20

    All Details

    Arista Switch Port Configuration Report

    Arista Switch Port Configuration Report

    arista-rj45 arista-sfp
    Model DCS-7050TX-64 DCS-7050QX-32S
    EOS 4.27.0F 4.27.0F
    Hostname arista-rj45 arista-sfp
    Management IP 10.32.104.2/20 10.32.104.3/20
    Domain coe.muc.redhat.com coe.muc.redhat.com
    Spanning-tree none none
    PTP — domain 24, boundary mode, prio1=20, prio2=10
    Default gateway 10.32.111.254 10.32.111.254

    VLANs

    VLAN Name arista-rj45 arista-sfp
    9 storage ✅ ✅
    32 coe ✅ ✅
    55 bla_ocp ✅ ✅
    69 coe-air-gapped ✅ ✅
    70 hcp-tests ✅ ✅
    100–109 sriov-test-100 … 109 ✅ ✅
    141 (unnamed) ✅ ❌
    239 ceph-internal ✅ ✅
    241 ibm-power-hmc1 ✅ ✅
    242 ibm-power-hmc2 ✅ ✅
    1004 stormshift-ocp4 ✅ ✅
    2000–2025 user-vlan-2000 … 2025 ✅ ✅

    Inter-Switch Link

    Both switches are connected via a Port-Channel 11 (LACP active):

    arista-rj45 arista-sfp
    Member ports Ethernet52/1–52/4 Ethernet19/1–19/4
    Native VLAN 32 32
    Allowed VLANs 9,32,69-70,100-109,241-242,1004,2000-2025 9,32,69-70,100-109,1004,2000-2025

    ⚠️ Mismatch: Port-Channel 11 on arista-rj45 includes VLANs 241-242 in the allowed list, but arista-sfp does not. Traffic for VLANs 241/242 will not cross the inter-switch link from the arista-sfp side.


    arista-rj45 — Port Configuration

    Standard Trunk Profile

    The vast majority of ports share this identical configuration:

    switchport trunk native vlan 32
    switchport trunk allowed vlan 9,32,69-70,100-109,241-242,1004,2000-2025
    switchport mode trunk
    
    Ports Description
    Ethernet2–8 (none)
    Ethernet10, 12–14, 16 (none)
    Ethernet20, 26 (none)
    Ethernet31, 33 (none)
    Ethernet37, 39–40, 42–43, 46 (none)
    Ethernet49/1–49/4, 50/1–50/4, 51/1–51/4 (none)
    Ethernet1 Link to RedHatSwitch
    Ethernet9 ucs56-primary
    Ethernet11 storm1
    Ethernet15 ucs55-secondary
    Ethernet18 ucs58
    Ethernet19 ceph10 - secondary
    Ethernet21 ceph12 - secondary
    Ethernet22 ucs56-secondary
    Ethernet23 ucs58
    Ethernet24 storm6-secondary
    Ethernet25 ceph12 - secondary
    Ethernet32 storm5-primary
    Ethernet34 storm5-secondary
    Ethernet35 ucs57-secondary
    Ethernet36 ucs57-primary
    Ethernet41 ceph11
    Ethernet44 onlogic-metal5
    Ethernet45 ceph10
    Ethernet47 ceph12
    Ethernet48 onlogic-metal4

    ⚡ Special Ports

    Port Description Mode Config Details
    Ethernet17 broken shutdown Port is administratively disabled
    Ethernet27 power91-1 mgmt - hmc1 access VLAN 241 (ibm-power-hmc1)
    Ethernet28 power91-2 mgmt - hmc2 access VLAN 242 (ibm-power-hmc2)
    Ethernet29 power92-1 mgmt - hmc1 access VLAN 241 (ibm-power-hmc1)
    Ethernet30 power92-2 mgmt - hmc2 access VLAN 242 (ibm-power-hmc2)
    Ethernet38 (none) access VLAN 2025 (user-vlan-2025)
    Ethernet52/1–52/4 port-group-11, link to arista-sfp LAG member channel-group 11 mode active (Port-Channel 11)
    Management1 — routed IP 10.32.104.2/20

    arista-sfp — Port Configuration

    Standard Trunk Profile

    The majority of ports share this identical configuration:

    switchport trunk native vlan 32
    switchport trunk allowed vlan 9,32,69-70,100-109,1004,2000-2025
    switchport mode trunk
    

    Note: Unlike arista-rj45, the standard trunk profile on arista-sfp does not include VLANs 241-242.

    Ports Description
    Ethernet1–4 (none)
    Ethernet7/1–7/4 (none)
    Ethernet9/1–9/4 (none)
    Ethernet10/1–10/4 (none)
    Ethernet11/1–11/4 (none)
    Ethernet12/1–12/4 (none)
    Ethernet13/2–13/3 (none)
    Ethernet14/1–14/4 (none)
    Ethernet15/1–15/4 (none)
    Ethernet16/1–16/4 (none)
    Ethernet17/1–17/4 (none)
    Ethernet18/1–18/4 (none)
    Ethernet20/1–20/4 (none)
    Ethernet21/1–21/4 (none)
    Ethernet22/1–22/4 (none)
    Ethernet23/1–23/4 (none)
    Ethernet24/3 (none)
    Ethernet25/1–25/4 (none)
    Ethernet26/1–26/4 (none)
    Ethernet27/1–27/4 (none)
    Ethernet28/1–28/2, 28/4 (none)
    Ethernet31–32 (none)
    Ethernet6/1 inf44 secondary eno2
    Ethernet6/2 power92
    Ethernet6/3 inf3
    Ethernet6/4 inf44 primary eno1
    Ethernet8/1 inf6
    Ethernet8/2 power91
    Ethernet8/3 inf5
    Ethernet8/4 inf4
    Ethernet13/4 inf20 MLX enp66s0f1np1 — keep separated bcf vlan
    Ethernet24/1–24/2 NetApp
    Ethernet28/3 inf20 MLX enp66s0f1np1
    Ethernet33 inf20 XL710 enp67s0f0
    Ethernet35 inf20 XL710 enp67s0f1

    ⚡ Special Ports

    Port Description Mode Config Details
    Ethernet5/1–5/4 (none) switchport only No trunk/access config — effectively default VLAN 1
    Ethernet13/1 inf20 MLX enp66s0f0np0 — keep separated bcf vlan trunk Native VLAN 32, allowed VLANs 2-54,56-238,240,243-4094 (nearly all VLANs!), VLAN translation 3333→32
    Ethernet24/4 inf20 MLX enp66s0f0np0 trunk Standard allowed VLANs, speed forced 10000full
    Ethernet29 Link to IBM Fusion HCI LAG member channel-group 250 mode active (Port-Channel 250)
    Ethernet30 Link to IBM Fusion HCI LAG member channel-group 250 mode active (Port-Channel 250)
    Ethernet34 inf20 e810 enp5s0f0 trunk Standard allowed VLANs, speed forced 10000full
    Ethernet36 inf20 e810 enp5s0f1 trunk Native VLAN tag (not 32), speed forced 10000full, PTP enabled (layer2, master role)
    Ethernet19/1–19/4 Link to arista-rj45 LAG member channel-group 11 mode active (Port-Channel 11)
    Management1 — routed IP 10.32.104.3/20

    Port-Channel 250 — IBM Fusion HCI

    Setting Value
    Member ports Ethernet29, Ethernet30
    Allowed VLANs 9,32,69-70,100-109,1004,2000-2025
    Native VLAN (not set — defaults to 1)
    LACP fallback static (timeout 20s)

    ⚠️ Note: Port-Channel 250 has no native VLAN 32 configured, unlike all other trunk ports. Untagged traffic will land on VLAN 1.


    Key Findings & Observations

    # Finding
    1 VLAN 241/242 mismatch: arista-rj45 trunk ports allow VLANs 241-242 (IBM Power HMC), but arista-sfp trunk ports do not. This means HMC management traffic cannot traverse the inter-switch link from the sfp side.
    2 Ethernet17 on arista-rj45 is shut down and marked as "broken".
    3 Ethernet38 on arista-rj45 is an access port on VLAN 2025 with no description — potentially undocumented.
    4 Ethernet5/1–5/4 on arista-sfp have switchport enabled but no mode or VLAN configuration — they default to VLAN 1 and appear unused.
    5 Ethernet13/1 on arista-sfp has a vastly different VLAN set (nearly all VLANs) plus VLAN translation (3333→32), intentionally separated for BCF.
    6 Ethernet36 on arista-sfp is the only port with PTP enabled (layer2, master role) and uses native vlan tag instead of native vlan 32.
    7 Port-Channel 250 (IBM Fusion HCI) is missing native vlan 32 — untagged traffic goes to VLAN 1 instead of the network default (32). This may be intentional.
    8 VLAN 141 exists only on arista-rj45 with no name and no ports assigned to it.
    9 VLAN 55 (bla_ocp) is defined on both switches but not referenced in any port's allowed VLAN list.
    10 VLAN 239 (ceph-internal) is defined on both switches but not allowed on any port.
  7. rbo commented on Oct 21, 2026

    @rbo
    MemberAuthor

    Almost done, I have to rewire inf20 - I used the wrong cables.

  8. rbo commented on Oct 29, 2026

    @rbo
    MemberAuthor

    inf20 wired done.

    We can close the ticket

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions