[OmniOS-discuss] [discuss] Re: [networking] rge_intr troubles

Sun Oct 2 17:20:11 UTC 2016

Yuck.  This driver does the loan up thing … which is ugly and pointless on
“modern” gigE hardware, bcopy is faster.  Especially as this driver is
probably only ever used with 1500 byte frames (it cannot do conventional
jumbo frames, but can do frames up to 7k… which is just bizarre!)  It looks
like this was copy-paste developed from nge, which explains some of the
sadness here.

But as a first test, you can try calling rge_receive().  The simplest way I
can see to do that is to OR in the value of RGE_NO_RXDESC_INT in the check
at 1495. (Btw the ordering of the checks at 1488 and 1495 are suboptimal,
as the rx interrupt should be the *hot* code path.

One thing you might also try doing is changing the value in rge.h for
RGE_RECV_COPY_SIZE from 256 to something much larger — 8000 will be larger
than the largest possible rge frame.  I have a theory that part of the
problem you are encountering may be due to being out of buffers to loan up,
and the screwy handling for that case.

If I were doing maintenance on this driver, I would gut all the rx loan up
code, and just bcopy.  On CPUs produced in the past decade or so, its
cheaper to bcopy than to do the loan up.  Especially if you rip out all the
checks you have to do to make the decision about which to do.  (And the
driver becomes much leaner and simpler.)

On Sun, Oct 2, 2016 at 9:59 AM, Garrett D'Amore <garrett at damore.org> wrote:

> Let me take a look.
>
> On Sun, Oct 2, 2016 at 5:23 AM, Saso Kiselkov <skiselkov.ml at gmail.com>
> wrote:
>
>> Well what I know so far is that after a fixed number of packets
>> (probably after filling up the RX ring), we get a storm of
>> RX_FIFO_OVERFLOW_INT | NO_RXDESC_INT interrupts, probably because the
>> adapter is telling us that it filled up the RX ring and we didn't let it
>> know that we dequeued the old packets. Unfortunately, for the life of
>> me, I can't figure out how we're supposed to let it know that. I've been
>> staring at the drivers (both our and FreeBSD's) for hours and to me it's
>> all just a jumble of "DMA sync this, write reg that".
>>
>> --
>> Saso
>>
>> On 10/2/16 3:40 AM, Garrett D'Amore wrote:
>> > probably we should do something.  like reap the descriptors. i am afk
>> but the usual strategy is to treat these kinds of interrupts just like
>> normal rx. after that you should ack the interrupt of course.
>> >
>> > Sent from my iPhone
>> >
>> >> On Oct 1, 2016, at 6:31 PM, Saso Kiselkov <skiselkov.ml at gmail.com>
>> wrote:
>> >>
>> >>> On 10/2/16 12:23 AM, Robert Mustacchi wrote:
>> >>>> On 10/1/16 15:15 , Saso Kiselkov wrote:
>> >>>>> On 10/1/16 11:45 PM, Dale Ghent wrote:
>> >>>>>
>> >>>>>> On Oct 1, 2016, at 3:36 PM, Saso Kiselkov <skiselkov.ml at gmail.com>
>> wrote:
>> >>>>>>
>> >>>>>> So I'm playing around with a box that has an on-board Realtek NIC
>> and
>> >>>>>> periodically, about once every 2-5 minutes, the network just goes
>> out to
>> >>>>>> lunch and stops responding to ping or attempts to send anything
>> from
>> >>>>>> the box. I noticed that while doing so, the box is getting floored
>> by
>> >>>>>> interrupts from the NIC, so I see tons of rge_intr activity and
>> one CPU
>> >>>>>> core receiving about 160000 interrupts per second (other cores are
>> idle).
>> >>>>>
>> >>>>> One core getting all the interrupts is expected, as both these
>> chips and the driver do not support RSS.
>> >>>>>
>> >>>>> The key thing here is to see what rge_intr() is actually doing. It
>> has 2 outcomes: It identifies the interrupt type, processes it, then
>> returns to the DDI that it was claimed. IF it doesn't identify the
>> interrupt, rge_intr() returns and reports unclaimed to the DDI.
>> >>>>>
>> >>>>> Knowing this info would be a good first step in figuring out what's
>> going on.
>> >>>>
>> >>>> Gah, I'm an idiot, it's actually a bitmask of two things:
>> >>>>
>> >>>> RX_FIFO_OVERFLOW_INT | NO_RXDESC_INT
>> >>>>
>> >>>> Apparently, we don't give it enough rx descriptors. Trying to now
>> figure
>> >>>> out where to change that...
>> >>>
>> >>> There'll always be cases where we don't have enough rx descriptors for
>> >>> devices. Presumably we shouldn't actually care about receiving that
>> >>> interrupt. Do you happen to have a specification for the device handy?
>> >>>
>> >>> Given that we're not doing anything with the NO_RXDESC_INT, we
>> probably
>> >>> should just mask it on the device if possible.
>> >>
>> >> Just as a general FYI, I'm dealing with 8168G version of the MAC.
>> >> FreeBSD does have a driver that supports it, but since the driver there
>> >> appears home-grown (similar to ours), trying to transplant it would be
>> a
>> >> major undertaking. I'll try to identify the major differences between
>> >> the versions we support and the 8168G, but of course, this being
>> >> hardware, they are many and few of them make any logical sense.
>> >>
>> >> --
>> >> Saso
>> >>
>> >
>> >
>>
>>
>> -------------------------------------------
>> illumos-networking
>> Archives: https://www.listbox.com/member/archive/182193/=now
>> RSS Feed: https://www.listbox.com/member/archive/rss/182193/22035935-
>> aad458cf
>> Modify Your Subscription: https://www.listbox.com/member
>> /?member_id=22035935&id_secret=22035935-4809d4da
>> Powered by Listbox: http://www.listbox.com
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://omniosce.org/ml-archive/attachments/20161002/19440062/attachment.html>