Menu
Home › Forum › Software discussion › Loopback testing experience between two 10G SFP
Tagged: SFP loopback
nanasi.
nanasiI have got two SFP+ transceivers and an optical cable.
I do an experiment of external loopback.
There is a log below.
software:
mv_ddr: mv_ddr-devel-18.12.0-g618dadd
atf: v1.5(release):1f8ca7e0 (Marvell-devel-18.12.2)
U-Boot: 2018.03-devel-18.12.3-gc9aa92c
kernel: mainline-4.19.30
enabled kernel configs:
CONFIG_SFP
CONFIG_MDIO_I2C
userland: devuan asciikernel dts patch:
--- armada-8040-mcbin.dts 2018-11-18 05:41:45.477482565 +0900
+++ armada-8040-mcbin-sfp.dts 2019-03-24 22:46:13.719457662 +0900
@@ -10,7 +10,7 @@
#include <dt-bindings/gpio/gpio.h>
/ {
- model = "Marvell 8040 MACCHIATOBin";
+ model = "Marvell 8040 MACCHIATOBin SFP+";
compatible = "marvell,armada8040-mcbin", "marvell,armada8040",
"marvell,armada-ap806-quad", "marvell,armada-ap806";
@@ -250,8 +250,14 @@
&cp0_eth0 {
status = "okay";
/* Network PHY */
+#if 0
phy = <&phy0>;
phy-mode = "10gbase-kr";
+#else
+ sfp = <&sfp_eth0>;
+ phy-mode = "sgmii";
+ managed = "in-band-status";
+#endif
/* Generic PHY, providing serdes lanes */
phys = <&cp0_comphy4 0>;
};
@@ -288,8 +294,14 @@
&cp1_eth0 {
status = "okay";
/* Network PHY */
+#if 0
phy = <&phy8>;
phy-mode = "10gbase-kr";
+#else
+ sfp = <&sfp_eth1>;
+ phy-mode = "sgmii";
+ managed = "in-band-status";
+#endif
/* Generic PHY, providing serdes lanes */
phys = <&cp1_comphy4 0>;
};bootlog:
Devuan GNU/Linux ascii mcbin ttyS0
mcbin login: root
Password:
Last login: Wed Mar 27 03:03:08 JST 2019 on ttyS0
Linux mcbin 4.19.30 aarch64 GNU/Linux
root@mcbin:~# uname -a
Linux mcbin 4.19.30 #1 SMP PREEMPT Sun Mar 24 09:26:28 JST 2019 aarch64 GNU/Linux
root@mcbin:~#
root@mcbin:~# dmesg | egrep -i ' (eth|sfp)'
[ 0.000000] Machine model: Marvell 8040 MACCHIATOBin SFP+
[ 1.085267] mvpp2 f2000000.ethernet eth0: Using firmware node mac address 00:51:82:11:22:00
[ 1.104349] mvpp2 f4000000.ethernet eth1: Using firmware node mac address 00:51:82:11:22:01
[ 1.114363] mvpp2 f4000000.ethernet eth2: Using firmware node mac address 00:51:82:11:22:02
[ 1.124384] mvpp2 f4000000.ethernet eth3: Using firmware node mac address 00:51:82:11:22:03
[ 3.441927] libphy: SFP I2C Bus: probed
[ 3.445858] sfp sfp-eth0: Host maximum power 1.0W
[ 3.451128] libphy: SFP I2C Bus: probed
[ 3.455055] sfp sfp-eth1: Host maximum power 1.0W
[ 3.460207] libphy: SFP I2C Bus: probed
[ 3.464117] sfp sfp-eth3: Host maximum power 1.0W
[ 3.762077] sfp sfp-eth0: module CISCO-FINISAR FTLX8571D3BCL-C2 rev A sn *********** dc 110823
[ 3.787759] mvpp2 f2000000.ethernet eth0: switched to inband/10gbase-kr link mode
[ 3.805176] sfp sfp-eth3: module FINISAR CORP. FTLF8519P2BNL rev A sn ******* dc 050310
[ 3.824456] mvpp2 f4000000.ethernet eth3: switched to inband/1000base-x link mode
[ 3.841941] sfp sfp-eth1: module CISCO-FINISAR FTLX8571D3BCL-C2 rev A sn *********** dc 110824
[ 3.861252] mvpp2 f4000000.ethernet eth1: switched to inband/10gbase-kr link mode
[ 4.555539] mvpp2 f4000000.ethernet eth2: PHY [f212a200.mdio-mii:00] driver [Generic PHY]
[ 4.577117] mvpp2 f4000000.ethernet eth2: configuring for phy/sgmii link mode
[ 5.065649] IPv6: ADDRCONF(NETDEV_UP): eth2: link is not ready
[ 8.684652] mvpp2 f4000000.ethernet eth2: Link is Up - 1Gbps/Full - flow control off
[ 8.692491] IPv6: ADDRCONF(NETDEV_CHANGE): eth2: link becomes ready
root@mcbin:~# SFP status:
root@mcbin:~# ethtool eth0
Settings for eth0:
Supported ports: [ FIBRE ]
Supported link modes: 10000baseSR/Full
Supported pause frame use: Symmetric Receive-only
Supports auto-negotiation: Yes
Supported FEC modes: Not reported
Advertised link modes: 10000baseSR/Full
Advertised pause frame use: Symmetric Receive-only
Advertised auto-negotiation: Yes
Advertised FEC modes: Not reported
Speed: 10000Mb/s
Duplex: Full
Port: FIBRE
PHYAD: 0
Transceiver: internal
Auto-negotiation: on
Link detected: no
root@mcbin:~#
root@mcbin:~# ethtool -m eth0 | head -34
Identifier : 0x03 (SFP)
Extended identifier : 0x04 (GBIC/SFP defined by 2-wire interface ID)
Connector : 0x07 (LC)
Transceiver codes : 0x10 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00
Transceiver type : 10G Ethernet: 10G Base-SR
Encoding : 0x06 (64B/66B)
BR, Nominal : 10300MBd
Rate identifier : 0x00 (unspecified)
Length (SMF,km) : 0km
Length (SMF) : 0m
Length (50um) : 80m
Length (62.5um) : 20m
Length (Copper) : 0m
Length (OM3) : 300m
Laser wavelength : 850nm
Vendor name : CISCO-FINISAR
Vendor OUI : 00:90:65
Vendor PN : FTLX8571D3BCL-C2
Vendor rev : A
Option values : 0x00 0x1a
Option : RX_LOS implemented
Option : TX_FAULT implemented
Option : TX_DISABLE implemented
BR margin, max : 0%
BR margin, min : 0%
Vendor SN : ***********
Date code : 110823
Optical diagnostics support : Yes
Laser bias current : 8.584 mA
Laser output power : 0.5281 mW / -2.77 dBm
Receiver signal average optical power : 0.0006 mW / -32.22 dBm
Module temperature : 58.75 degrees C / 137.76 degrees F
Module voltage : 3.3272 V
Alarm/warning flags implemented : Yes
root@mcbin:~# enable interface:
root@mcbin:~# ifconfig eth0 up
[ 846.676846] mvpp2 f2000000.ethernet eth0: configuring for inband/10gbase-kr link mode
[ 846.697120] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
root@mcbin:~#
root@mcbin:~# ifconfig eth1 up
[ 869.305404] mvpp2 f4000000.ethernet eth1: configuring for inband/10gbase-kr link mode
[ 869.326857] IPv6: ADDRCONF(NETDEV_UP): eth1: link is not ready
root@mcbin:~#
[ 869.632379] mvpp2 f4000000.ethernet eth1: Link is Up - 10Gbps/Full - flow control rx
[ 869.640174] IPv6: ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready
root@mcbin:~#
root@mcbin:~# ifconfig eth0 down
root@mcbin:~# ifconfig eth0 up
[ 953.463880] mvpp2 f2000000.ethernet eth0: configuring for inband/10gbase-kr link mode
[ 953.471933] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
root@mcbin:~#
[ 953.788886] mvpp2 f2000000.ethernet eth0: Link is Up - 10Gbps/Full - flow control rx
[ 953.796680] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
root@mcbin:~#
root@mcbin:~# ethtool eth0
Settings for eth0:
Supported ports: [ FIBRE ]
Supported link modes: 10000baseSR/Full
Supported pause frame use: Symmetric Receive-only
Supports auto-negotiation: Yes
Supported FEC modes: Not reported
Advertised link modes: 10000baseSR/Full
Advertised pause frame use: Symmetric Receive-only
Advertised auto-negotiation: Yes
Advertised FEC modes: Not reported
Speed: 10000Mb/s
Duplex: Full
Port: FIBRE
PHYAD: 0
Transceiver: internal
Auto-negotiation: on
Link detected: yes
root@mcbin:~# loopback setup:
root@mcbin:~#
root@mcbin:~# ip netns add n0
root@mcbin:~# ip link set dev eth0 netns n0
[ 1193.652844] mvpp2 f2000000.ethernet eth0: Link is Down
root@mcbin:~#
root@mcbin:~# ip netns add n1
root@mcbin:~# ip link set dev eth1 netns n1
[ 1221.623608] mvpp2 f4000000.ethernet eth1: Link is Down
root@mcbin:~#
root@mcbin:~# ip netns exec n0 ifconfig eth0 192.168.99.1/24 mtu 8192 up
[ 1280.285727] mvpp2 f2000000.ethernet eth0: configuring for inband/10gbase-kr link mode
[ 1280.293970] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
[ 1280.312718] mvpp2 f2000000.ethernet eth0: configuring for inband/10gbase-kr link mode
root@mcbin:~#
root@mcbin:~# ip netns exec n1 ifconfig eth1 192.168.99.2/24 mtu 8192 up
[ 1327.240481] mvpp2 f4000000.ethernet eth1: configuring for inband/10gbase-kr link mode
[ 1327.249029] IPv6: ADDRCONF(NETDEV_UP): eth1: link is not ready
[ 1327.267647] mvpp2 f4000000.ethernet eth1: configuring for inband/10gbase-kr link mode
root@mcbin:~#
[ 1327.596637] mvpp2 f4000000.ethernet eth1: Link is Up - 10Gbps/Full - flow control rx
[ 1327.604429] IPv6: ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready
root@mcbin:~#
root@mcbin:~# ip netns exec n0 ifconfig eth0 down
root@mcbin:~# ip netns exec n0 ifconfig eth0 up
[ 1401.487313] mvpp2 f2000000.ethernet eth0: configuring for inband/10gbase-kr link mode
[ 1401.495384] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
root@mcbin:~#
[ 1401.801249] mvpp2 f2000000.ethernet eth0: Link is Up - 10Gbps/Full - flow control rx
[ 1401.809045] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
root@mcbin:~# speed test:
root@mcbin:~#
root@mcbin:~# ip netns exec n0 iperf3 -B 192.168.99.1 -s &
[3] 3727
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
root@mcbin:~#
root@mcbin:~# ip netns exec n1 iperf3 -B 192.168.99.2 -c 192.168.99.1
Connecting to host 192.168.99.1, port 5201
Accepted connection from 192.168.99.2, port 41227
[ 4] local 192.168.99.2 port 39241 connected to 192.168.99.1 port 5201
[ 5] local 192.168.99.1 port 5201 connected to 192.168.99.2 port 39241
[ ID] Interval Transfer Bandwidth
[ 5] 0.00-1.00 sec 639 MBytes 5.36 Gbits/sec
[ ID] Interval Transfer Bandwidth Retr Cwnd
[ 4] 0.00-1.00 sec 669 MBytes 5.61 Gbits/sec 0 1002 KBytes
[ 5] 1.00-2.00 sec 798 MBytes 6.70 Gbits/sec
[ 4] 1.00-2.00 sec 799 MBytes 6.70 Gbits/sec 0 1.20 MBytes
[ 5] 2.00-3.00 sec 793 MBytes 6.66 Gbits/sec
[ 4] 2.00-3.00 sec 794 MBytes 6.65 Gbits/sec 0 1.20 MBytes
[ 5] 3.00-4.00 sec 793 MBytes 6.66 Gbits/sec
[ 4] 3.00-4.00 sec 792 MBytes 6.65 Gbits/sec 0 1.20 MBytes
[ 5] 4.00-5.00 sec 792 MBytes 6.64 Gbits/sec
[ 4] 4.00-5.00 sec 792 MBytes 6.64 Gbits/sec 0 1.20 MBytes
[ 5] 5.00-6.00 sec 795 MBytes 6.67 Gbits/sec
[ 4] 5.00-6.00 sec 795 MBytes 6.67 Gbits/sec 0 1.20 MBytes
[ 5] 6.00-7.00 sec 794 MBytes 6.66 Gbits/sec
[ 4] 6.00-7.00 sec 794 MBytes 6.66 Gbits/sec 0 1.20 MBytes
[ 5] 7.00-8.00 sec 794 MBytes 6.66 Gbits/sec
[ 4] 7.00-8.00 sec 794 MBytes 6.66 Gbits/sec 0 1.20 MBytes
[ 5] 8.00-9.00 sec 794 MBytes 6.66 Gbits/sec
[ 4] 8.00-9.00 sec 795 MBytes 6.66 Gbits/sec 0 1.20 MBytes
[ 5] 9.00-10.00 sec 793 MBytes 6.66 Gbits/sec
[ 5] 10.00-10.04 sec 31.2 MBytes 6.69 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth
[ 5] 0.00-10.04 sec 0.00 Bytes 0.00 bits/sec sender
[ 5] 0.00-10.04 sec 7.63 GBytes 6.53 Gbits/sec receiver
[ 4] 9.00-10.00 sec 794 MBytes 6.66 Gbits/sec 0 1.20 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth Retr
[ 4] 0.00-10.00 sec 7.63 GBytes 6.56 Gbits/sec 0 sender
[ 4] 0.00-10.00 sec 7.63 GBytes 6.56 Gbits/sec receiver
iperf Done.
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
root@mcbin:~# 
hbahostI can confirm my tests with kernel 5.1.0 on eth0 on macchiatobin singleshot.
SFP devices are: SFP-10GSR-85 from fs.com – connected with 2m LC Fiber cables.
eth0 seems to require ifdown eth0; ifup eth0; before the link comes up (takes a few seconds). Afterwards it worked perfectly for multiple days with constant streaming/traffic.
eth1 sometimes goes up, sometimes doesn’t at all. I can’t reliably reproduce it, so I disabled it.
eth2 as everyone knows (ethernet) works fine.
eth3 only works at 1gb, sometimes, so I disabled it.
fkpwolfThe port works if the router connects to another Intel 82599 NIC. If it connected to my 10G switch(Quanta LB6M), the eth0 interface can’t be ready.
Does this issue relate with my switch?
I have tried different sfp modules and got same result.
Technical specification tables can not be displayed on mobile. Please view on desktop