VOOZH about

URL: https://www.ibm.com/support/pages/linux-power-ethernet-adapters-which-which

⇱ Linux on POWER Ethernet adapters: Which is Which?

Linux on POWER Ethernet adapters: Which is Which?

How To


Summary

PCIe adapter slots and Ethernet port name can easily get confusing, this article might help.

Objective

👁 Nigels Banner

Steps

So I have a Linux on Power Scale-Out S822L Server with VM with physical adapters but which Ethernet adapter is which after I added two more?

First port of call is the regular lspci command:


# lspci
0000:70:00.0 RAID bus controller: IBM PCI-E IPR SAS Adapter (ASIC) (rev 02)
0001:80:00.0 Ethernet controller: Broadcom Corporation NetXtreme BCM5719 Gigabit Ethernet PCIe (rev 01)
0001:80:00.1 Ethernet controller: Broadcom Corporation NetXtreme BCM5719 Gigabit Ethernet PCIe (rev 01)
0001:80:00.2 Ethernet controller: Broadcom Corporation NetXtreme BCM5719 Gigabit Ethernet PCIe (rev 01)
0001:80:00.3 Ethernet controller: Broadcom Corporation NetXtreme BCM5719 Gigabit Ethernet PCIe (rev 01)
0002:01:00.0 Ethernet controller: Mellanox Technologies MT27520 Family [ConnectX-3 Pro]
0003:01:00.0 Ethernet controller: Mellanox Technologies MT27520 Family [ConnectX-3 Pro]
#

My two new Mellanox adapters are there at the end but which is in Slot C2 and in C5 and what are they new fangled Ethernet device names (like the old eth0 etc)?

The old fashioned eth0 sort of names could be renumbered if added the adapters in to earlier PCI slot numbers and cause chaos.

The probably mean something - the manuals say the format is:

  • domain:bus:device.function Class: Vendor  Device and optional PhySlot.

So

  • Vendor is 
  • Device is 

So the Physical Slot number is missing. Darn!

I installed the lsvpd command, which comes in the IBM Linux on Power repository extra toolset.  I then ran it but get many 100's of lines of output - too much to paste here but the answer was hidden away in the data. Here is the section for the first Mellanox adapter: Note the Device name is in the lsvpd output but not the Vendor (Mellanox), so the lspci output saved the day:

*TM MT27520 Family [ConnectX-3 Pro]
*CD 10141007
*YC 0
*YL U78CB.001.WZS0FW8-P1-C5
*FC
*DS Ethernet PCI Adapter
*AX enP2p1s0
*NA e41d2d2d0840
*RM 2.36.5000
*YL U78CB.001.WZS0FW8-P1-C5
*FC
*DS Ethernet PCI Adapter

. . .

*DS 10GbE RoCE SFP+ PCIe 3.0 Adapter
*AX ethernet
*TM network
*YL U78CB.001.WZS0FW8-P1-C5-T1
*FC
*DS 10GbE RoCE SFP+ PCIe 3.0 Adapter
*AX ethernet
*TM network
*YL U78CB.001.WZS0FW8-P1-C5-T2
  1. = The Mellanox adapter

many Ethernet adapters have more ports.  The Second Mellanox adapter looks like this:

*TM MT27520 Family [ConnectX-3 Pro]
*CD 10141007
*YC 0
*YL U78CB.001.WZS0FW8-P1-C2
*FC
*DS Ethernet PCI Adapter
*AX enP3p1s0
*NA e41d2d2d1730
*RM 2.36.5000
*YL U78CB.001.WZS0FW8-P1-C2
*FC
*DS Ethernet PCI Adapter
*AX enP3p1s0d1
. . .
*DS 10GbE RoCE SFP+ PCIe 3.0 Adapter
*AX ethernet
*TM network
*YL U78CB.001.WZS0FW8-P1-C2-T1
*FC
*DS 10GbE RoCE SFP+ PCIe 3.0 Adapter
*AX ethernet
*TM network
*YL U78CB.001.WZS0FW8-P1-C2-T2 

I hope this helps and stops you having to use the "empirical method" = try every adapter and every port at least twice until it suddenly blurts in to life!!

KelleyCook commented July 6 2017 lsslot -c pci is very useful too:
[root@gssio1 ~]# lsslot -c pci
# Slot Description Device(s)
U78CB.001.WZS02GN-P1-C2 PCI-E capable, Rev 3, 8x lanes with 8x lanes connected 000a:01:00.0
U78CB.001.WZS02GN-P1-C3 PCI-E capable, Rev 3, 16x lanes with 16x lanes connected 000b:01:00.0
U78CB.001.WZS02GN-P1-C5 PCI-E capable, Rev 3, 16x lanes with 16x lanes connected 0009:01:00.0
U78CB.001.WZS02GN-P1-C6 PCI-E capable, Rev 3, 16x lanes with 16x lanes connected 0004:01:00.0
U78CB.001.WZS02GN-P1-C7 PCI-E capable, Rev 3, 16x lanes with 16x lanes connected 0000:01:00.0
U78CB.001.WZS02GN-P1-C9 PCI-E capable, Rev 3, 8x lanes with 8x lanes connected Empty
U78CB.001.WZS02GN-P1-C10 PCI-E capable, Rev 3, 8x lanes with 8x lanes connected 0007:80:00.0
0007:80:00.1
0007:80:00.2
0007:80:00.3
U78CB.001.WZS02GN-P1-C11 PCI-E capable, Rev 3, 8x lanes with 8x lanes connected 0001:50:00.0
U78CB.001.WZS02GN-P1-C12 PCI-E capable, Rev 3, 8x lanes with 8x lanes connected Empty
[root@gssio1 ~]#

Additional Information

Other places to find content from Nigel Griffiths IBM (retired)

Document Location

Worldwide

[{"Line of Business":{"code":"","label":""},"Business Unit":{"code":"BU054","label":"Systems w\/TPS"},"Product":{"code":"HW1W1","label":"Power -\u003EPowerLinux"},"ARM Category":[{"code":"","label":""}],"Platform":[{"code":"PF016","label":"Linux"}],"Version":"All Versions"}]

Was this topic helpful?

Document Information

Modified date:
14 June 2023

UID

ibm11115631