Re: Giant instrumentation

On Thu, 21 Mar 2002, John Baldwin wrote:

> Kelly Yancey implemented most of the syscall Giant instrumentation stuff
> I outlined in an earlier e-mail.  I have since mostly finished it and
> changed a few things.  Most of the changes were design changes so that
> the instrumentation stuff was as separated from the rest of the kernel
> (and thus easily condionitionally compiled out with minimal impact) as
> possible.  Also, it now provides better support for handling other ABIs. 
> It's not complete yet (it only has instrumentation for the native ABI at
> the moment) but I wanted to post this patch for feedback.  I have tested
> it some and it is being tested some more on alpha and x86 at the moment. 
> The patch can be found at: 
> 
> http://www.FreeBSD.org/~jhb/patches/giantvars.patch

Some general thoughts--

(1) One thing I particularly like about placing the instrumentation
    outside of the system call code itself, since it means avoiding
    restructuring the system call code to handle consitional lock release
    at every return point.  For things like proc-locking, it may not
    substantially change the code flow, but for some of the more
    complicated VFS calls with many return points, that may be useful. 
    Adding more components to an "unwind" tends to complicate system call
    code a great deal.  One important thing we'd like to do for these
    "complicated" calls is avoid changing the code flow when we add
    instrumentation, since that will potentially introduce bugs associated
    with control flow changes.  Also, when we later remove the
    instrumentation, we end up potentially wanting to restore the old
    structure, introducing new bugs or changing code flow.  Where
    possible, it would be nice if the pre-instrumention code flow,
    instrumented code flow, and post-instrumentation flow were visually
    (and actually) the same.  That is, adding instrumentation should be
    non-intrusive, which this certainly accomplishes. 

(2) Question: how easily does this framework extend to trap handling, such
    as VM traps where we might similarly want to be able to instrument
    entry to traps that might or might not want Giant?  The current stuff
    is pretty system-call specific in my reading, at least, in where it
    gets the information.  Similarly, could this framework be adapted for
    use with interrupt/driver entry points?

(3) For the td_ucred changes, it seems to me that introducing
    instrumentation may be gratuitous, regardless of the instrumentation
    approach taken.  If anything, the *setting* of td_ucred is something
    that should be instrumented as a use of the proc lock.  One hesitates
    to say it, but "it should just work".  We've (in the
    alfred/siego sense) actually already successfully pushed down locks
    for some relatively major subsystems with very little problem.  If
    anything, experience so far has suggested that the most important
    thing is having effective locking assertions is the most powerful
    debugging tool, be it via various mtx_assert(), etc, things, or the
    ability to use Witness to enforce more general lock order and locking
    relationship constraints.

(4) One down-side to this approach is that it's not possible to push the
    instrumentation point down below the system call level.  For example,
    suppose we're confident that the first level code in the various
    generic file descriptor-based calls is correct, but we want to
    continue to instrument Giant over the VFS but not over pipes.  In such
    a scenario, you could imagine something like the following:

	system call (giant instrumentation: proc, filedesc)
	struct file indirection (giant instrumentation: filedesc, file)
	file_ops pipe (giant instrumentation: file, pipes)
	pipe implementation (giant instrumentation: pipes)

    If we had a mechanism permitting us to identify that file and pipe
    locking were suspect, then Giant would be grabbed when we hit the
    struct file indirection code, and it would presumably be retained
    until we walked back up the stack to there.

    That said, I'm not convinced this sort of spl-like mechanism is
    actually desirable.  If the goal of instrumentation is to provide an
    "escape hatch" during the integration of a particular phase of
    locking, simply locking at the system call level is sufficient to
    protect the average developer from suffering.  Anything else is icing. 

(5) Another advantage of this model is that it centralizes the
    instrumentation information.  I can easily look at the list of
    instrumented entry points in a single place, and walk through to drop
    in a new instrumentation type where necessary.  I can walk the list of
    socket calls and mark them as IPv4-instrumented, if I'm pushing down
    Giant on the IPv4 stack without having to walk a large number of
    files.  Likewise, I can easily strip an instrumentation type from the
    table in one place.  This does, however, point again to the
    desirability of having similar tables for drivers, interrupt bits,
    etc.  Likewise, if I've added MAC hooks for VFS entry points, I can
    easily add a new MAC Giant instrumentation for all the relevant system
    calls.  Paying the development cost up-front for a centralized
    instrumentation mechanism reduces the later cost of changing the
    instrumentation as we move forward with fine-grained locking, and
    easily allows developers to modify instrumentation in a local tree
    without generating diffs involving hundreds of files, especially diffs
    that may be mixed in with the locking changes themselves.

Robert N M Watson             FreeBSD Core Team, TrustedBSD Project
[e-mail]      NAI Labs, Safeport Network Services



To Unsubscribe: send mail to [e-mail]
with "unsubscribe freebsd-smp" in the body of the message