Skip to content

KubemacpoolMACCollisionDetected is firing #408

Closed

Description

@DanielFroehlich

subject says it all.

Activity

  1. DanielFroehlich commented on Sep 1, 2026

    @DanielFroehlich
    Author

    Following runbook procedure: https://github.com/openshift/runbooks/blob/master/alerts/openshift-virtualization-operator/KubemacpoolMACCollisionDetected.md

    mac-Adresses:

    • 0a:58:0a:2a:00:02
    • 0a:58:0a:80:00:02
    • 0a:58:0a:80:02:02
    • 0a:58:0a:81:00:02
    • 0a:58:0a:81:02:02
    • 0a:58:0a:82:00:02
    • 0a:58:0a:83:00:02
    • 0a:59:00:00:00:01

    Lets take a look at one:

     % export MAC="0a:58:0a:80:00:02"
    dfroehli@dfroehli-mac21 stormshift-automation % oc get vmi -A -o jsonpath='{range .items[*]}{.metadata.namespace}{"\t"}{.metadata.name}{"\t"}{.status.interfaces[*].mac}{"\n"}{end}' | grep -i "$MAC"
    rguske-ocp42	rguske-ocp42-cp2	02:06:b6:02:4d:b2 f2:58:95:b9:a2:e1 ba:d6:3f:de:f0:52 0a:58:0a:80:00:02 12:e9:85:03:40:1e 76:9d:c7:b6:3e:cc 8e:9e:fc:0b:2a:32 22:1e:3a:4d:ba:42 f2:17:32:bc:f8:dd ce:49:a4:78:3b:8a c2:1e:a3:3d:ad:f7
    stormshift-ocp1-infra	ocp1-cp-1	0e:c0:ef:20:69:43 b2:6f:eb:62:e3:9d 0a:58:0a:80:00:02 36:e2:f3:ef:c9:57 16:94:e6:01:47:e9 06:95:9a:58:3a:bc ea:59:d2:cf:12:19 4a:28:13:5f:89:12 52:ed:87:fc:a1:d8 ee:f9:96:f2:5b:3d 2e:78:47:2b:12:fe
    stormshift-ocp11-infra	ocp11-cp-0	0e:c0:ef:20:6f:80 9e:b0:23:04:dc:3f 0a:58:0a:80:00:02 16:75:5f:ed:e8:29 c2:f0:8c:61:10:29 8e:3d:50:88:1b:1a d2:a6:84:e7:96:b3 a2:41:a4:ac:e8:e6 ba:cb:2f:a8:ee:3b 6e:81:6e:e6:33:ec 1a:98:a6:17:c0:9d
    stormshift-ocp14-infra	ocp14-cp-0	0e:c0:ef:20:6f:83 f6:ae:f0:0e:a6:fc 0a:58:0a:80:00:02 9a:da:c4:72:12:44 d2:c7:35:bd:81:62 82:48:ab:25:c8:23 e6:86:c2:6f:1d:74 4e:14:e7:eb:ab:47 d2:ea:f9:c0:d8:ad 2a:57:c5:9c:72:64 82:e1:47:24:5d:79
    stormshift-ocp16-infra	ocp16-cp-0	0e:c0:ef:20:6f:85 12:ca:46:ed:eb:07 0a:58:0a:80:00:02 a2:45:f3:43:21:10 12:e7:32:d9:6a:f7 6a:f1:ca:26:fa:a9 b6:cf:97:12:fc:80 b2:08:99:a1:05:51 6e:b1:c1:d6:37:70 4e:b9:68:30:c3:1d 32:64:4a:fa:65:21
    stormshift-ocp19-infra	ocp19-cp-0	0e:c0:ef:20:6f:88 a6:59:16:93:5b:1b 0a:58:0a:80:00:02 ba:cb:8a:fd:06:a1 d2:17:cd:00:8f:23 22:0c:d8:6e:e5:19 b2:19:a9:04:bc:09 36:19:35:30:a0:4a 5a:22:36:6a:37:1d 82:2b:a7:84:12:9d 02:29:01:71:cf:3e
    stormshift-ocp20-infra	ocp20-cp-0	0e:c0:ef:20:6f:89 52:2b:77:a9:b1:01 0a:58:0a:80:00:02 76:a0:23:fa:41:eb 5a:0f:c2:2e:dc:06 86:bc:f7:8f:69:10 06:8e:1d:60:65:f1 3e:f0:5c:91:e1:6b 9e:f2:7e:03:af:fb 6a:e3:5f:a1:37:cf da:a4:a5:ec:27:47
    stormshift-ocp21-infra	ocp21-cp-1	0e:c0:ef:20:69:7d 2e:8e:d4:52:40:1c 0a:58:0a:80:00:02 7e:be:13:be:47:2a be:a6:e4:d0:47:d0 5a:9f:75:e0:b2:29 e2:da:27:bd:84:41 26:b9:e0:ac:0a:d8 c6:aa:f9:b4:91:a7 16:41:3d:b1:07:73 3e:7e:18:1b:76:ef
    stormshift-ocp22-infra	ocp22-cp-1	0e:c0:ef:20:6f:c2 72:57:30:3c:ee:5a 0a:58:0a:80:00:02 6a:4c:e5:a7:b9:4c 52:34:e6:a8:c5:0f 4e:aa:e2:25:f8:c1 3e:00:3d:c7:ea:f6 7a:b4:69:a6:6d:6c 46:c8:ba:16:5e:46 9a:91:72:88:7a:dc f2:da:c2:31:d2:66
    stormshift-ocp6-infra	ocp6-cp-2	0e:c0:ef:20:69:5e 12:8a:86:3c:54:72 0a:58:0a:80:00:02 b6:cc:b2:a4:cb:32 12:c8:33:0e:ca:c2 9e:00:24:33:85:83 36:c5:a0:34:c4:32 d2:9a:59:49:c9:41 8a:77:0a:50:b5:86 ba:ba:12:c2:b1:18 ca:6e:21:39:b0:6c
    
    

    Looking at ocp1-cp-1, thats the ovn-k8s-mp0 bridge with that MAC address.
    Same on ocp11-cp-0.

    Strange. Asking claude for help...

  2. DanielFroehlich commented on Sep 1, 2026

    @DanielFroehlich
    Author

    Claude analysis:

    Short answer: Safe to ignore — it's a false positive

    What's actually happening

    The MAC 0a:58:0a:80:00:02 decodes like this:

    • 0a:58 = OVN-Kubernetes's hardcoded prefix for its management port MAC addresses
    • 0a:80:00:02 = hex for 10.128.0.2 (OpenShift's default pod network starts at 10.128.0.0/14)

    OVN-Kubernetes deterministically derives the ovn-k8s-mp0 MAC from the interface's IP address using the formula: 0a:58 + IP-in-hex. So any OCP node whose ovn-k8s-mp0 gets assigned 10.128.0.2 will always have this exact MAC.

    Both ocp1-cp-1 and ocp11-cp-0 are running nested OCP clusters with the same default pod CIDR (10.128.0.0/14). Inside each VM, OVN-Kubernetes independently assigns 10.128.0.2 to its first ovn-k8s-mp0 interface — producing the same MAC in two separate VMs.

    Why KubeMacPool is wrong to flag this

    KubeMacPool's job is to prevent collisions on the host L2 network (ISAR's pod/VM network). But ovn-k8s-mp0 is an in-guest virtual interface — it lives inside the VM's network namespace, never touches ISAR's network fabric, and is completely invisible to anything outside the VM. The actual KubeVirt NIC MACs your inventory assigns are 0E:C0:EF:* — those are what KubeMacPool should care about.

    KubeMacPool is scanning VMI status and picking up guest-internal OVN-K8s interfaces it has no business managing. This is a known false positive in nested OCP-on-OCP-Virt setups.

  3. DanielFroehlich commented on Sep 1, 2026

    @DanielFroehlich
    Author

    @rbo do you agree? I would silence this alert for the stormshift-*infra namespaces

  4. rbo commented on Sep 7, 2026

    @rbo
    Member

    @DanielFroehlich Agree!

    IMHO it's also an kind of a bug, because with openshift on openshift we are not the only who hits the alert.

  5. DanielFroehlich commented on Sep 7, 2026

    @DanielFroehlich
    Author

    Seems there is already a very releated bug existing:
    https://redhat.atlassian.net/browse/CNV-95971
    I added a comment and will go ahead and silence this alert for our infra-projects.

  6. DanielFroehlich commented on Sep 7, 2026

    @DanielFroehlich
    Author

    Alert is silenced now:
    https://console-openshift-console.apps.isar.coe.muc.redhat.com/monitoring/silences/f771e774-57f1-4d30-9d95-438c51f2f4a9
    Downside: cant be silence per project regexp - is global now. We need to remember to remove this once the fix is delivered.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

bugSomething isn't workingcluster/isarBareMetal COE Cluter

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions