Home Forum Software discussion Loopback testing experience between two 10G SFP

Tagged: 

Viewing 4 posts - 1 through 4 (of 4 total)
  • Author
    Posts
  • #7392
    nanasi
    Participant

    I have got two SFP+ transceivers and an optical cable.
    I do an experiment of external loopback.
    There is a log below.

    software:

    mv_ddr:	mv_ddr-devel-18.12.0-g618dadd
    atf:	v1.5(release):1f8ca7e0 (Marvell-devel-18.12.2)
    U-Boot:	2018.03-devel-18.12.3-gc9aa92c
    kernel: mainline-4.19.30
    enabled kernel configs:
     CONFIG_SFP
     CONFIG_MDIO_I2C
    userland: devuan ascii

    kernel dts patch:

    --- armada-8040-mcbin.dts	2018-11-18 05:41:45.477482565 +0900
    +++ armada-8040-mcbin-sfp.dts	2019-03-24 22:46:13.719457662 +0900
    @@ -10,7 +10,7 @@
     #include <dt-bindings/gpio/gpio.h>
     
     / {
    -	model = "Marvell 8040 MACCHIATOBin";
    +	model = "Marvell 8040 MACCHIATOBin SFP+";
     	compatible = "marvell,armada8040-mcbin", "marvell,armada8040",
     			"marvell,armada-ap806-quad", "marvell,armada-ap806";
     
    @@ -250,8 +250,14 @@
     &cp0_eth0 {
     	status = "okay";
     	/* Network PHY */
    +#if 0
     	phy = <&phy0>;
     	phy-mode = "10gbase-kr";
    +#else
    +	sfp = <&sfp_eth0>;
    +	phy-mode = "sgmii";
    +	managed = "in-band-status";
    +#endif
     	/* Generic PHY, providing serdes lanes */
     	phys = <&cp0_comphy4 0>;
     };
    @@ -288,8 +294,14 @@
     &cp1_eth0 {
     	status = "okay";
     	/* Network PHY */
    +#if 0
     	phy = <&phy8>;
     	phy-mode = "10gbase-kr";
    +#else
    +	sfp = <&sfp_eth1>;
    +	phy-mode = "sgmii";
    +	managed = "in-band-status";
    +#endif
     	/* Generic PHY, providing serdes lanes */
     	phys = <&cp1_comphy4 0>;
     };

    bootlog:

    Devuan GNU/Linux ascii mcbin ttyS0
    
    mcbin login: root
    Password: 
    Last login: Wed Mar 27 03:03:08 JST 2019 on ttyS0
    Linux mcbin 4.19.30 aarch64 GNU/Linux
    
    root@mcbin:~# uname -a
    Linux mcbin 4.19.30 #1 SMP PREEMPT Sun Mar 24 09:26:28 JST 2019 aarch64 GNU/Linux
    root@mcbin:~# 
    root@mcbin:~# dmesg | egrep -i ' (eth|sfp)'
    [    0.000000] Machine model: Marvell 8040 MACCHIATOBin SFP+
    [    1.085267] mvpp2 f2000000.ethernet eth0: Using firmware node mac address 00:51:82:11:22:00
    [    1.104349] mvpp2 f4000000.ethernet eth1: Using firmware node mac address 00:51:82:11:22:01
    [    1.114363] mvpp2 f4000000.ethernet eth2: Using firmware node mac address 00:51:82:11:22:02
    [    1.124384] mvpp2 f4000000.ethernet eth3: Using firmware node mac address 00:51:82:11:22:03
    [    3.441927] libphy: SFP I2C Bus: probed
    [    3.445858] sfp sfp-eth0: Host maximum power 1.0W
    [    3.451128] libphy: SFP I2C Bus: probed
    [    3.455055] sfp sfp-eth1: Host maximum power 1.0W
    [    3.460207] libphy: SFP I2C Bus: probed
    [    3.464117] sfp sfp-eth3: Host maximum power 1.0W
    [    3.762077] sfp sfp-eth0: module CISCO-FINISAR    FTLX8571D3BCL-C2 rev A    sn ***********      dc 110823  
    [    3.787759] mvpp2 f2000000.ethernet eth0: switched to inband/10gbase-kr link mode
    [    3.805176] sfp sfp-eth3: module FINISAR CORP.    FTLF8519P2BNL    rev A    sn *******          dc 050310  
    [    3.824456] mvpp2 f4000000.ethernet eth3: switched to inband/1000base-x link mode
    [    3.841941] sfp sfp-eth1: module CISCO-FINISAR    FTLX8571D3BCL-C2 rev A    sn ***********      dc 110824  
    [    3.861252] mvpp2 f4000000.ethernet eth1: switched to inband/10gbase-kr link mode
    [    4.555539] mvpp2 f4000000.ethernet eth2: PHY [f212a200.mdio-mii:00] driver [Generic PHY]
    [    4.577117] mvpp2 f4000000.ethernet eth2: configuring for phy/sgmii link mode
    [    5.065649] IPv6: ADDRCONF(NETDEV_UP): eth2: link is not ready
    [    8.684652] mvpp2 f4000000.ethernet eth2: Link is Up - 1Gbps/Full - flow control off
    [    8.692491] IPv6: ADDRCONF(NETDEV_CHANGE): eth2: link becomes ready
    root@mcbin:~# 

    SFP status:

    root@mcbin:~# ethtool eth0
    Settings for eth0:
            Supported ports: [ FIBRE ]
            Supported link modes:   10000baseSR/Full 
            Supported pause frame use: Symmetric Receive-only
            Supports auto-negotiation: Yes
            Supported FEC modes: Not reported
            Advertised link modes:  10000baseSR/Full 
            Advertised pause frame use: Symmetric Receive-only
            Advertised auto-negotiation: Yes
            Advertised FEC modes: Not reported
            Speed: 10000Mb/s
            Duplex: Full
            Port: FIBRE
            PHYAD: 0
            Transceiver: internal
            Auto-negotiation: on
            Link detected: no
    root@mcbin:~# 
    root@mcbin:~# ethtool -m eth0 | head -34
            Identifier                                : 0x03 (SFP)
            Extended identifier                       : 0x04 (GBIC/SFP defined by 2-wire interface ID)
            Connector                                 : 0x07 (LC)
            Transceiver codes                         : 0x10 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00
            Transceiver type                          : 10G Ethernet: 10G Base-SR
            Encoding                                  : 0x06 (64B/66B)
            BR, Nominal                               : 10300MBd
            Rate identifier                           : 0x00 (unspecified)
            Length (SMF,km)                           : 0km
            Length (SMF)                              : 0m
            Length (50um)                             : 80m
            Length (62.5um)                           : 20m
            Length (Copper)                           : 0m
            Length (OM3)                              : 300m
            Laser wavelength                          : 850nm
            Vendor name                               : CISCO-FINISAR
            Vendor OUI                                : 00:90:65
            Vendor PN                                 : FTLX8571D3BCL-C2
            Vendor rev                                : A
            Option values                             : 0x00 0x1a
            Option                                    : RX_LOS implemented
            Option                                    : TX_FAULT implemented
            Option                                    : TX_DISABLE implemented
            BR margin, max                            : 0%
            BR margin, min                            : 0%
            Vendor SN                                 : ***********
            Date code                                 : 110823
            Optical diagnostics support               : Yes
            Laser bias current                        : 8.584 mA
            Laser output power                        : 0.5281 mW / -2.77 dBm
            Receiver signal average optical power     : 0.0006 mW / -32.22 dBm
            Module temperature                        : 58.75 degrees C / 137.76 degrees F
            Module voltage                            : 3.3272 V
            Alarm/warning flags implemented           : Yes
    root@mcbin:~# 

    enable interface:

    root@mcbin:~# ifconfig eth0 up
    [  846.676846] mvpp2 f2000000.ethernet eth0: configuring for inband/10gbase-kr link mode
    [  846.697120] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
    root@mcbin:~# 
    root@mcbin:~# ifconfig eth1 up
    [  869.305404] mvpp2 f4000000.ethernet eth1: configuring for inband/10gbase-kr link mode
    [  869.326857] IPv6: ADDRCONF(NETDEV_UP): eth1: link is not ready
    root@mcbin:~# 
    [  869.632379] mvpp2 f4000000.ethernet eth1: Link is Up - 10Gbps/Full - flow control rx
    [  869.640174] IPv6: ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready
    root@mcbin:~# 
    root@mcbin:~# ifconfig eth0 down
    root@mcbin:~# ifconfig eth0 up
    [  953.463880] mvpp2 f2000000.ethernet eth0: configuring for inband/10gbase-kr link mode
    [  953.471933] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
    root@mcbin:~# 
    [  953.788886] mvpp2 f2000000.ethernet eth0: Link is Up - 10Gbps/Full - flow control rx
    [  953.796680] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
    root@mcbin:~# 
    root@mcbin:~# ethtool eth0   
    Settings for eth0:
            Supported ports: [ FIBRE ]
            Supported link modes:   10000baseSR/Full 
            Supported pause frame use: Symmetric Receive-only
            Supports auto-negotiation: Yes
            Supported FEC modes: Not reported
            Advertised link modes:  10000baseSR/Full 
            Advertised pause frame use: Symmetric Receive-only
            Advertised auto-negotiation: Yes
            Advertised FEC modes: Not reported
            Speed: 10000Mb/s
            Duplex: Full
            Port: FIBRE
            PHYAD: 0
            Transceiver: internal
            Auto-negotiation: on
            Link detected: yes
    root@mcbin:~# 

    loopback setup:

    root@mcbin:~# 
    root@mcbin:~# ip netns add n0
    root@mcbin:~# ip link set dev eth0 netns n0
    [ 1193.652844] mvpp2 f2000000.ethernet eth0: Link is Down
    root@mcbin:~# 
    root@mcbin:~# ip netns add n1 
    root@mcbin:~# ip link set dev eth1 netns n1
    [ 1221.623608] mvpp2 f4000000.ethernet eth1: Link is Down
    root@mcbin:~# 
    root@mcbin:~# ip netns exec n0 ifconfig eth0 192.168.99.1/24 mtu 8192 up
    [ 1280.285727] mvpp2 f2000000.ethernet eth0: configuring for inband/10gbase-kr link mode
    [ 1280.293970] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
    [ 1280.312718] mvpp2 f2000000.ethernet eth0: configuring for inband/10gbase-kr link mode
    root@mcbin:~# 
    root@mcbin:~# ip netns exec n1 ifconfig eth1 192.168.99.2/24 mtu 8192 up
    [ 1327.240481] mvpp2 f4000000.ethernet eth1: configuring for inband/10gbase-kr link mode
    [ 1327.249029] IPv6: ADDRCONF(NETDEV_UP): eth1: link is not ready
    [ 1327.267647] mvpp2 f4000000.ethernet eth1: configuring for inband/10gbase-kr link mode
    root@mcbin:~# 
    [ 1327.596637] mvpp2 f4000000.ethernet eth1: Link is Up - 10Gbps/Full - flow control rx
    [ 1327.604429] IPv6: ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready
    root@mcbin:~# 
    root@mcbin:~# ip netns exec n0 ifconfig eth0 down
    root@mcbin:~# ip netns exec n0 ifconfig eth0 up
    [ 1401.487313] mvpp2 f2000000.ethernet eth0: configuring for inband/10gbase-kr link mode
    [ 1401.495384] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
    root@mcbin:~# 
    [ 1401.801249] mvpp2 f2000000.ethernet eth0: Link is Up - 10Gbps/Full - flow control rx
    [ 1401.809045] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
    root@mcbin:~# 

    speed test:

    root@mcbin:~# 
    root@mcbin:~# ip netns exec n0 iperf3 -B 192.168.99.1 -s &
    [3] 3727
    -----------------------------------------------------------
    Server listening on 5201
    -----------------------------------------------------------
    root@mcbin:~# 
    root@mcbin:~# ip netns exec n1 iperf3 -B 192.168.99.2 -c 192.168.99.1     
    Connecting to host 192.168.99.1, port 5201
    Accepted connection from 192.168.99.2, port 41227
    [  4] local 192.168.99.2 port 39241 connected to 192.168.99.1 port 5201
    [  5] local 192.168.99.1 port 5201 connected to 192.168.99.2 port 39241
    [ ID] Interval           Transfer     Bandwidth
    [  5]   0.00-1.00   sec   639 MBytes  5.36 Gbits/sec                  
    [ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
    [  4]   0.00-1.00   sec   669 MBytes  5.61 Gbits/sec    0   1002 KBytes       
    [  5]   1.00-2.00   sec   798 MBytes  6.70 Gbits/sec                  
    [  4]   1.00-2.00   sec   799 MBytes  6.70 Gbits/sec    0   1.20 MBytes       
    [  5]   2.00-3.00   sec   793 MBytes  6.66 Gbits/sec                  
    [  4]   2.00-3.00   sec   794 MBytes  6.65 Gbits/sec    0   1.20 MBytes       
    [  5]   3.00-4.00   sec   793 MBytes  6.66 Gbits/sec                  
    [  4]   3.00-4.00   sec   792 MBytes  6.65 Gbits/sec    0   1.20 MBytes       
    [  5]   4.00-5.00   sec   792 MBytes  6.64 Gbits/sec                  
    [  4]   4.00-5.00   sec   792 MBytes  6.64 Gbits/sec    0   1.20 MBytes       
    [  5]   5.00-6.00   sec   795 MBytes  6.67 Gbits/sec                  
    [  4]   5.00-6.00   sec   795 MBytes  6.67 Gbits/sec    0   1.20 MBytes       
    [  5]   6.00-7.00   sec   794 MBytes  6.66 Gbits/sec                  
    [  4]   6.00-7.00   sec   794 MBytes  6.66 Gbits/sec    0   1.20 MBytes       
    [  5]   7.00-8.00   sec   794 MBytes  6.66 Gbits/sec                  
    [  4]   7.00-8.00   sec   794 MBytes  6.66 Gbits/sec    0   1.20 MBytes       
    [  5]   8.00-9.00   sec   794 MBytes  6.66 Gbits/sec                  
    [  4]   8.00-9.00   sec   795 MBytes  6.66 Gbits/sec    0   1.20 MBytes       
    [  5]   9.00-10.00  sec   793 MBytes  6.66 Gbits/sec                  
    [  5]  10.00-10.04  sec  31.2 MBytes  6.69 Gbits/sec                  
    - - - - - - - - - - - - - - - - - - - - - - - - -
    [ ID] Interval           Transfer     Bandwidth
    [  5]   0.00-10.04  sec  0.00 Bytes  0.00 bits/sec                  sender
    [  5]   0.00-10.04  sec  7.63 GBytes  6.53 Gbits/sec                  receiver
    [  4]   9.00-10.00  sec   794 MBytes  6.66 Gbits/sec    0   1.20 MBytes       
    - - - - - - - - - - - - - - - - - - - - - - - - -
    [ ID] Interval           Transfer     Bandwidth       Retr
    [  4]   0.00-10.00  sec  7.63 GBytes  6.56 Gbits/sec    0             sender
    [  4]   0.00-10.00  sec  7.63 GBytes  6.56 Gbits/sec                  receiver
    
    iperf Done.
    -----------------------------------------------------------
    Server listening on 5201
    -----------------------------------------------------------
    root@mcbin:~# 
    #7393
    hbahost
    Participant

    I can confirm my tests with kernel 5.1.0 on eth0 on macchiatobin singleshot.

    SFP devices are: SFP-10GSR-85 from fs.com – connected with 2m LC Fiber cables.

    eth0 seems to require ifdown eth0; ifup eth0; before the link comes up (takes a few seconds). Afterwards it worked perfectly for multiple days with constant streaming/traffic.

    eth1 sometimes goes up, sometimes doesn’t at all. I can’t reliably reproduce it, so I disabled it.

    eth2 as everyone knows (ethernet) works fine.

    eth3 only works at 1gb, sometimes, so I disabled it.

    #7606
    fkpwolf
    Participant

    The port works if the router connects to another Intel 82599 NIC. If it connected to my 10G switch(Quanta LB6M), the eth0 interface can’t be ready.

    Does this issue relate with my switch?

    I have tried different sfp modules and got same result.

    #7645
    nanasi
    Participant

    I have no info other than this HCL. Sorry.

Viewing 4 posts - 1 through 4 (of 4 total)
  • You must be logged in to reply to this topic.

Technical specification tables can not be displayed on mobile. Please view on desktop