After playing with this a bit today I found that a large chunk of the
difference seems to be related to the caching of various structures
related to process fork()ing, such as thread and kernel stacks, vmspace
structures, process structures, and so forth.
I made a few commits to make some of these parameters tunable, then
tuned them up to absurd levels in /boot/loader.conf (requiring a reboot):
lwkt.cache_threads=4096
vm.cache_vmspaces=4096
And the results were as expected. A much flatter, curve.
I have to say though that this is NOT a good idea in general. FreeBSD
just should not be trying to cache thousands of threads worth of
structures and associated data... it should be throwing that away to free
up memory for other purposes, and probably contributes to KVM fragmentation
as well. Even though it makes the fork benchmark look better it is
a bad idea for a production system.
-Matt