10G interfaces not working

[JohnBscvParticipant]

Hello,

I’ve been attempting to get the 10G interfaces working on my DoubleShot. I’ve tried Ubuntu 16.04.3 LTS from the software page, as well as Buildroot – both from the software page and by building my own from the buildroot source – however none of it works.

I get no link regardless if I try using the RJ45 ports or the SFP cages. If I plug in an SFP module into the SFP cage, a little orange light on the RJ45 port comes on. However in software on the board, using ifconfig or ethtool, it always shows that it has a link, even if nothing is plugged in.

So to summarise:
RJ45 10G doesn’t work, no link, no physical indication of anything reacting to being plugged in, no additional LEDs turn on, and no link flapping in software
SFP cages react to me plugging in an SFP+ module or SFP+ copper cable by an orange LED lighting up on the corresponding RJ45 connector belonging to that pair (eth1 -> SFP+/RJ45, eth0 -> SFP+/RJ45), but no link is actually established. I’ve tried plugging the other end into switches and directly into servers, still no link.
The software does react to something being plugged into the SFP cages when running Buildroot by printing out the detected module model, but again, no link.

What am I doing wrong?

[JohnBscvParticipant]

I can’t see a way to edit posts but wanted to add for clarification:

All interfaces have been brought up in software, using for example ifconfig eth0 up

[xorinoxParticipant]

I have the same issue with my recently received DoubleShot. I have tried a couple of “standard” network configurations, the only only port I get up and running find is the 1GbE port. I really wanted this board because of the 2 x 10GbE copper ports. I could not find good instructions on how they were supposed to work..

I would be curious if anyone got them up and running and would be sharing their configuration.

[JohnBscvParticipant]

Yep, eth2 works just fine, but eth0 and eth1 will not link at all

I do believe I got eth3 (up to 2.5Gbps) to link at 1 Gbps via a regular SFP module, but I’d have to check whether that was mainline kernel or not

I’d be great if someone responded with a bit more information because the information we have from Marvell/Solidrun hasn’t been enough to get them to work

[xorinoxParticipant]

I found that with ifconfig eth0 and ifconfig eth1 I am able to activate the 10GbE copper NICs. I have only connected eth1 with a cat7 cable.

cat /sys/class/net/eth1/operstate would then also report “up”

root@macchiatobin-node-1:~# ethtool eth1
Settings for eth1:
Supported ports: [ ]
Supported link modes: Not reported
Supported pause frame use: No
Supports auto-negotiation: No
Advertised link modes: Not reported
Advertised pause frame use: No
Advertised auto-negotiation: No
Speed: 10000Mb/s
Duplex: Full
Port: MII
PHYAD: 8
Transceiver: external
Auto-negotiation: on
Link detected: yes

What I do find then is that dhclient eth1 doesn’t ever finish and if you cancel the operation; eth1 inet isn’t configured

eth1 Link encap:Ethernet HWaddr 00:51:82:11:22:01
inet6 addr: fe80::251:82ff:fe11:2201/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:14 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:2048
RX bytes:0 (0.0 B) TX bytes:2700 (2.7 KB)
Interrupt:122 Memory:f4000000-f408ffff

[xorinoxParticipant]

I captured the packets during the dhclient eth1 call. It looks a lot different compared to dealing with eth2, it received an IP from the same dhcp server within seconds.

“1”,”0.000000″,”0.0.0.0″,”255.255.255.255″,”DHCP”,”342″,”DHCP Discover – Transaction ID 0x2a43b635″
“2”,”3.227463″,”0.0.0.0″,”255.255.255.255″,”DHCP”,”342″,”DHCP Discover – Transaction ID 0x2a43b635″
“3”,”9.930143″,”0.0.0.0″,”255.255.255.255″,”DHCP”,”342″,”DHCP Discover – Transaction ID 0x2a43b635″
“4”,”19.028635″,”0.0.0.0″,”255.255.255.255″,”DHCP”,”342″,”DHCP Discover – Transaction ID 0x2a43b635″
“5”,”33.800305″,”0.0.0.0″,”255.255.255.255″,”DHCP”,”342″,”DHCP Discover – Transaction ID 0x2a43b635″
“6”,”47.784562″,”0.0.0.0″,”255.255.255.255″,”DHCP”,”342″,”DHCP Discover – Transaction ID 0x2a43b635″
“7”,”60.478413″,”0.0.0.0″,”255.255.255.255″,”DHCP”,”342″,”DHCP Discover – Transaction ID 0x2a43b635″

[xorinoxParticipant]

Linux macchiatobin-node-1 4.4.52-armada-17.10.3-g6adee55 #6 SMP PREEMPT Tue Feb

[JohnBscvParticipant]

Are you able to confirm that you actually got a link on the other end, and not only that the interface is reporting link?
My interfaces report that they have a link all the time, despite not having one. I suspect it might be the way they’re physically wired for us to be able to do SFP/Copper

[xorinoxParticipant]

My Switch XS716E is reporting that the cables are ok. But you are correct, on the switch I do not see any LED blinking indicating that a link was established.

This is really annoying! How long have been trying to make this work?

[JohnBscvParticipant]

Yep, that’s the behaviour I see here

I’ve had a lot of other projects to take care of, but on and off since I got the board, which was in February

[JohnBscvParticipant]

Edit: My bad! The November date was for a different board, since beginning of February for this one

[xorinoxParticipant]

I have called the US office, and was asked to open a support case, as the US office is only doing sales, which I just did. I have also forwarded the link to this conversation to them. I keep you posted.

[xorinoxParticipant]

Answer from support, we cannot really help you unless you buy support package 5 hours for $450..

[teknoraverParticipant]

Hi,

FYI, I’m using a vanilla 5.1-rc6 kernel and the SFP+ ports works great.
With a small kernel patch I can do 9.5 gbit per port:

root@macchiatobin:~# dmesg |grep mvpp2
[    5.480502] mvpp2 f2000000.ethernet eth0: Using firmware node mac address 00:51:82:11:22:00
[    5.496545] mvpp2 f4000000.ethernet eth1: Using firmware node mac address 00:51:82:11:22:01
[    5.498129] mvpp2 f4000000.ethernet eth2: Using firmware node mac address 00:51:82:11:22:02
[    5.499741] mvpp2 f4000000.ethernet eth3: Using firmware node mac address 00:51:82:11:22:03
[    6.316291] mvpp2 f4000000.ethernet eth2: PHY [f212a200.mdio-mii:00] driver [Marvell 88E1510]
[    6.327412] mvpp2 f4000000.ethernet eth2: configuring for phy/sgmii link mode
[   11.510594] mvpp2 f4000000.ethernet eth2: Link is Up - 1Gbps/Full - flow control rx/tx
[   18.866179] mvpp2 f4000000.ethernet eth2: Link is Up - 1Gbps/Full - flow control off
[   72.111082] mvpp2 f2000000.ethernet eth0: PHY [f212a600.mdio-mii:00] driver [mv88x3310]
[   72.132102] mvpp2 f2000000.ethernet eth0: configuring for phy/10gbase-kr link mode
[   72.143157] mvpp2 f4000000.ethernet eth1: PHY [f212a600.mdio-mii:08] driver [mv88x3310]
[   72.164114] mvpp2 f4000000.ethernet eth1: configuring for phy/10gbase-kr link mode
[   73.190188] mvpp2 f4000000.ethernet eth1: Link is Up - 10Gbps/Full - flow control off
[   73.198094] mvpp2 f2000000.ethernet eth0: Link is Up - 10Gbps/Full - flow control off

root@macchiatobin:~# uname -a
Linux macchiatobin 5.1.0-rc6 #27 SMP Tue Apr 30 15:58:34 CEST 2019 aarch64 aarch64 aarch64 GNU/Linux

root@macchiatobin:~# iperf3 -s
warning: this system does not seem to support IPv6 - trying IPv4
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
Accepted connection from 192.168.0.1, port 46818
[  5] local 192.168.0.2 port 5201 connected to 192.168.0.1 port 46820
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec   822 MBytes  6.89 Gbits/sec                  
[  5]   1.00-2.00   sec   860 MBytes  7.21 Gbits/sec                  
[  5]   2.00-3.00   sec   856 MBytes  7.18 Gbits/sec                  
[  5]   3.00-4.00   sec   854 MBytes  7.16 Gbits/sec                  
[  5]   4.00-5.00   sec   854 MBytes  7.16 Gbits/sec                  
[  5]   5.00-6.00   sec   853 MBytes  7.16 Gbits/sec                  
[  5]   6.00-7.00   sec   783 MBytes  6.57 Gbits/sec                  
[  5]   7.00-8.00   sec   744 MBytes  6.24 Gbits/sec                  
[  5]   8.00-9.00   sec   762 MBytes  6.40 Gbits/sec                  
[  5]   9.00-10.00  sec   762 MBytes  6.39 Gbits/sec                  
[  5]  10.00-11.00  sec   747 MBytes  6.26 Gbits/sec                  
[  5]  11.00-12.00  sec   749 MBytes  6.28 Gbits/sec                  
[  5]  12.00-13.00  sec   738 MBytes  6.19 Gbits/sec                  
[  5]  13.00-14.00  sec   751 MBytes  6.30 Gbits/sec                  
[  5]  14.00-15.00  sec  1.05 GBytes  8.99 Gbits/sec                  
[  5]  15.00-16.00  sec  1.10 GBytes  9.41 Gbits/sec                  
[  5]  16.00-17.00  sec  1.10 GBytes  9.41 Gbits/sec                  
[  5]  17.00-18.00  sec  1.10 GBytes  9.41 Gbits/sec                  
[  5]  18.00-19.00  sec  1.10 GBytes  9.41 Gbits/sec                  
[  5]  19.00-20.00  sec  1.10 GBytes  9.41 Gbits/sec                  
[  5]  20.00-20.04  sec  43.8 MBytes  9.41 Gbits/sec                  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-20.04  sec  17.4 GBytes  7.48 Gbits/sec                  receiver
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
iperf3: interrupt - the server has terminated

Cheers,

[7GrokParticipant]

Can you provide details on the kernel patch? What distro are you using the 5.1 kernel?

—
Thanks

[Author]

Posts