Re: Interrupt Handlers and Multiple CPUs

Stefan Saroiu wrote:
> This is the context.
> 
> I am writing a packet capture application on a PIV 2GHz single proc
> machine with a SK-9843 1Gbps network card. Under 6-700 Mbps of traffic,
> 60-70% of the CPU is spent inside of the interrupt handler. This
> is the interrupt coming from the NIC (pci/if_sk.c).
> 
> Because so many CPU cycles are spent on these interrupts, the packet
> capture application gets starved over time.

This is called "receiver livelock".  Read the DEC Western Research
Laboratories paper by Jaff Mogul on this from 1991, then read the
Rice University papers on the from 1196 and 1998 by Peter Druschel
and Mohit Aron.


> Now the question:
> ================
> 
> If I drop a second CPU in the box, will FreeBSD be smart to schedule the
> interrupts on one CPU and the packet capture code on the other? What
> FreeBSD version do I need (I have 4.5)?
> 
> The SMP docs on the FreeBSD site seem to point to an affirmative answer,
> but I'd like to get more opinions before I blow more money on a second
> CPU.

[ I assume that this is for a commercial "1 Gigbit!" capture tool... ]

The interrupts will be scheduled on the CPU that has the vector.

In the common case, this will be the same CPU that's running the
capture application, because it's "in the kernel" waiting for I/O
from the card, and thereore has the kernel reentrancy lock, which
implicitly gives it the vector for the interrupts.

Ideally, FreeBSD would have two other modes of operating:

1)	Direct vector of interrupts, in which interrupts from
	a specific card are directed to a particular CPU, AND

2)	Virtual wire mode, in which interrupts are directed
	to a "virtual" interrupt line, and whatever CPU wants
	to handle them can.

The main barrier to both of these approaches is that only one
CPU is allowed into the kernel code at a time in FreeBSD 4.x,
and the entry barrier points are "system call", "fault handler",
and "interrupt handler".

Druschel and Aron have implemented a fix for the problem you
are seeing, called LRP ("Lazy Receiver Processing").  This has
been available for FreeBSD since the 2.x days, and you can
download code for FreeBSD 3.x and 4.x.  The 4.x code should work
for you.  It contains accounting constructis for resource
accounting, to ensure scheduling of user space processes, thus
avoiding the live lock.  This implementation is called ResCon
("Resource Containers").

The license on this code prohibits commercial use without
contacting the Rice University Technology Transfer office,
but it should be enough to let you see what you are missing,
and whether or not you want to license.


In general, I would say that throwing hardware at the problem
will not work, and any company that thinks that this is a
workable solution, regardless of what their platform is, will
find themselves disappointed.

-- Terry

To Unsubscribe: send mail to [e-mail]
with "unsubscribe freebsd-smp" in the body of the message