[OmniOS-discuss] slowness in aggr - bonded 10G

[Lee Damon]

I have four file servers all of which have at least two (some three)
aggregates made of two 10GB lines.  Three of the four servers are getting
pretty good throughput as reported by iperf:

(hvfs1 has iperf -s, other hosts iperf -c)
[ ID] Interval           Transfer     Bandwidth
[  5]   0.00-10.00  sec  8.04 GBytes  6.91 Gbits/sec                  sender
[  5]   0.00-10.00  sec  8.04 GBytes  6.91 Gbits/sec
receiver

but one, fs2, is getting roughly half that speed when it is running iperf
-c:

[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-10.00  sec  3.81 GBytes  3.27 Gbits/sec                  sender
[  4]   0.00-10.00  sec  3.81 GBytes  3.27 Gbits/sec
receiver
(with occasional bumps to 4.05 Gb/s)

Even stranger, when I run iperf -c on fs2 and iperf -s on hvfs1 I get a
better (though still not as good) result bouncing between 4.6 Gb/s and 5.2
Gb/s.

It doesn't matter if I test via aggr0 or aggr_nas0, I get roughly the same
numbers. It also doesn't make any difference if aggr0 is dual-i40e or
dual-ixgbe devices.

Tests were done with hvfs1 acting as the iperf -s server via aggr_front0
(the equivalent to aggr0 on the other hosts) or aggr_nas0 as appropriate.
All hosts are directly connected to the same Juniper switch. (aggr info is
postpended at the end of this email.)

One theory I had was that fs2 wasn't actually using the full throughput of
the bond so I physically removed one of the two cables. Sure enough, the
bandwidth reported by iperf remained around 3.2Gb/s. I tested this with
both aggr0 and aggr_nas0 with the same result. There are times where it
gets closer to the 6Gb of the other hosts so it is clearly getting more
than just one link's worth but not often.

Unlike the other servers, fs2 is running 151026. The rest are all on
151022. I didn't see any references to aggr in the release notes for 151024
or 151026 but I'm not going to claim exhaustive search.

Aggregates were created with the device equivalent of

   dladm create-aggr -L active -l i40e0 -l i40e1 aggr0

My question is, what blatantly obvious thing did I miss in creating the
aggregates on fs2?

thanks,
nomad


aggr info:

fs1 (omnios-r151022-5e982daae6):
LINK            POLICY   ADDRPOLICY           LACPACTIVITY  LACPTIMER
 FLAGS
aggr0           L4       auto                 passive       short
 -----  (not used for this test)
aggr_nas0       L4       auto                 active        short
 -----
aggr_front0     L4       auto                 active        short
 -----

LINK        CLASS     MTU    STATE    BRIDGE     OVER
e1000g1     phys      1500   up       --         --
e1000g0     phys      1500   up       --         --
aggr0       aggr      1500   up       --         e1000g0,e1000g1
ixgbe0      phys      1500   up       --         --
ixgbe1      phys      1500   up       --         --
aggr_nas0   aggr      1500   up       --         ixgbe0,ixgbe1
ixgbe2      phys      1500   up       --         --
ixgbe3      phys      1500   up       --         --
aggr_front0 aggr      1500   up       --         ixgbe2,ixgbe3


fs2 (omnios-r151026-51c7d6fd75):
LINK            POLICY   ADDRPOLICY           LACPACTIVITY  LACPTIMER
 FLAGS
aggr_nas0       L4       auto                 active        short
 -----
aggr_net10      L4       auto                 active        short
 -----  (not used for this test)
aggr0           L4       auto                 active        short
 -----

LINK        CLASS     MTU    STATE    BRIDGE     OVER
i40e2       phys      1500   up       --         --
igb0        phys      1500   up       --         --
i40e0       phys      1500   up       --         --
i40e3       phys      1500   up       --         --
igb1        phys      1500   up       --         --
i40e1       phys      1500   up       --         --
ixgbe0      phys      1500   unknown  --         --
ixgbe1      phys      1500   unknown  --         --
aggr_nas0   aggr      1500   up       --         i40e2,i40e3
aggr_net10  aggr      1500   up       --         igb0,igb1
aggr0       aggr      1500   up       --         i40e0,i40e1


hvfs1 (omnios-r151022-5e982daae6):
LINK            POLICY   ADDRPOLICY           LACPACTIVITY  LACPTIMER
 FLAGS
aggr0           L4       auto                 active        short
 -----  (not used for this test)
aggr_front0     L4       auto                 active        short
 -----
aggr_nas0       L4       auto                 active        short
 -----

LINK        CLASS     MTU    STATE    BRIDGE     OVER
ixgbe0      phys      1500   up       --         --
igb0        phys      1500   up       --         --
igb1        phys      1500   up       --         --
igb2        phys      1500   unknown  --         --
oce0        phys      1500   up       --         --
igb3        phys      1500   unknown  --         --
ixgbe1      phys      1500   up       --         --
oce1        phys      1500   up       --         --
aggr0       aggr      1500   up       --         igb0,igb1
aggr_front0 aggr      1500   up       --         ixgbe0,ixgbe1
aggr_nas0   aggr      1500   up       --         oce0,oce1


hvfs2 (omnios-r151022-89f6242508):
LINK            POLICY   ADDRPOLICY           LACPACTIVITY  LACPTIMER
 FLAGS
aggr_nas0       L4       auto                 active        short
 -----
aggr0           L4       auto                 active        short
 -----

LINK        CLASS     MTU    STATE    BRIDGE     OVER
ixgbe2      phys      1500   up       --         --
ixgbe0      phys      1500   up       --         --
ixgbe3      phys      1500   up       --         --
ixgbe1      phys      1500   up       --         --
aggr_nas0   aggr      1500   up       --         ixgbe0,ixgbe1
aggr0       aggr      1500   up       --         ixgbe3,ixgbe2
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://omniosce.org/ml-archive/attachments/20180917/a03e8cef/attachment-0001.html>