ethtool 只對實體網卡有作用
所以我先把系統的p3p1 , p3p1定速(Intel X540 10G NIC)
DEVICE=p3p1
#HWADDR=A0:36:9F:7C:63:68
TYPE=Ethernet
UUID=24819cec-d318-49bf-854f-a321be47013e
MASTER=bond0
SLAVE=yes
ONBOOT=yes
NM_CONTROLLED=no
BOOTPROTO=none
ETHTOOL_OPTS="speed 10000 duplex full autoneg off"

一共做了p3p1 , p3p2兩個ports

單一port定速是沒有任何問題，但是bond之後看起來就失效了
(Auto是on , 直接用指令關auto也沒用)

Settings for p3p1:
        Supported ports: [ TP ]
        Supported link modes:   100baseT/Full
                                1000baseT/Full
                                10000baseT/Full
        Supported pause frame use: Symmetric
        Supports auto-negotiation: Yes
        Advertised link modes:  10000baseT/Full
        Advertised pause frame use: Symmetric
        Advertised auto-negotiation: Yes
        Speed: 10000Mb/s
        Duplex: Full
        Port: Twisted Pair
        PHYAD: 0
        Transceiver: external
        Auto-negotiation: on
        MDI-X: Unknown
        Supports Wake-on: d
        Wake-on: d
        Current message level: 0x00000007 (7)
                               drv probe link
        Link detected: yes




switch端已經設定定速完成，系統內也沒連線問題
但是根本就不知道有沒有定速










起因是因為在系統上常常會看到下列訊息(p3p1,p3p2輪著來)
kernel: ixgbe 0000:04:00.0: p3p1: NIC Link is Down
kernel: bond0: link status definitely down for interface p3p1, disabling it
kernel: bond0: first active interface up!
ixgbe 0000:04:00.0: p3p1: NIC Link is Up 10 Gbps, Flow Control: None
kernel: bond0: link status definitely up for interface p3p1, 10000 Mbps full d
up


網卡換過/線換過(Cat.6)/重開過/firmware更新/driver更新/swithc換port, 都沒改善
於是想做個定速測試試試


在同個switch有其他台server 也有插Intel 540 10G,

1台是Oracle Linux 6.7 (無問題)
1台是Oracle Linux 6.8 (有類似問題)
這台是RHEL 6.8 (此篇提的機器)


測到現在我覺得是不是Linux 6.8有問題...
伺服器是Dell R630

→ galic: https://bugzilla.redhat.com/show_bug.cgi?id=696337
→ galic: 是這個issues嗎?
→ galic: 不對 照你的版本應該被fixed 我再查查
→ galic: 你環境描述再清楚一點 像是你更新後的版號之類的...
這個issue有查過，但是系統上沒有rx miss

更新:
ethtool -i p3p1
driver: ixgbe
version: 5.2.4
firmware-version: 0x800005f6, 18.0.17
bus-info: 0000:04:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: yes


Bonding driver version 3.7.1
OS 版本號 : (uname -a去掉日期)
 2.6.32-642.el6.x86_64 #1 SMP x86_64 x86_64 x86_64 GNU/Linux

目前系統上ifconfig狀態如下:(只列出跟bonding有關的,去掉ip)

bond0     Link encap:Ethernet  HWaddr A0:36:9F:E1:5F:C0
          inet addr:xxxxxxxxx  Bcast:172.21.103.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MASTER MULTICAST  MTU:1500  Metric:1
          RX packets:1401076722 errors:0 dropped:0 overruns:0 frame:0
          TX packets:1634636464 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:803655790323 (748.4 GiB)  TX bytes:1140369457904 (1.0 TiB)

p3p1      Link encap:Ethernet  HWaddr A0:36:9F:E1:5F:C0
          UP BROADCAST RUNNING SLAVE MULTICAST  MTU:1500  Metric:1
          RX packets:306679435 errors:0 dropped:0 overruns:0 frame:0
          TX packets:720537271 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:165806032881 (154.4 GiB)  TX bytes:527301564338 (491.0 GiB)

p3p2      Link encap:Ethernet  HWaddr A0:36:9F:E1:5F:C0
          UP BROADCAST RUNNING SLAVE MULTICAST  MTU:1500  Metric:1
          RX packets:701875754 errors:0 dropped:0 overruns:0 frame:0
          TX packets:464806464 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:415406736029 (386.8 GiB)  TX bytes:311857294678 (290.4 GiB)


bonding setup(IP有關的去掉)

DEVICE="bond0"
ONBOOT="yes"
USERCTL="no"
BOOTPROTO="none"
NM_CONTROLLED="no"
BONDING_OPTS="mode=4 miimon=100 xmit_hash_policy=layer3+4 lacp_rate=1"

miimon改過1000, layer嘗試過拿掉或改成2+3 , 3, 4 ,都沒改善。
※ 編輯: GoldDeath (1.163.235.91), 11/24/2017 22:26:54
→ galic: 有點詭異的情況... 我是建議能更新的都更新到最新版本看看
→ galic: 然後看能不能把debug msg. level調高一點 看有什麼有用資訊
→ galic: 問題應該不是出在bond driver
→ galic: https://tinyurl.com/yc8srdsj 你現在msg. level是0x7
→ galic: ethtool -s p3p1 --msglvl 0xffff 試試
→ GoldDeath: 瞭解，下週上班試試，前幾天交換機剛定速
→ GoldDeath: 要是再不行我只能跟老外討論看看升OS
→ GoldDeath: 感謝幫忙，後續我再修文更新
→ GoldDeath: 開了Current message level: 0x0000ffff (65535)
→ GoldDeath: 但是/var/log/messages沒有特別的訊息，還是跟之前一樣
→ galic: 應該看kern.log或syslog喔 messages不會有debug用的訊息
→ galic:         ↑dmesg
→ GoldDeath: 啊，蠢了，忘記看dmesg
→ GoldDeath: 更新，還是沒更多的訊息，跟之前一樣
→ GoldDeath: 問題終於解決，10G switch 韌體更新搞定