OK, so the buffer allocation problem with ZFS is fixed, but now I

This is a cryptographically signed message in MIME format.

--------------ms030907030401050008070005
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: quoted-printable

 From this morning.....

     3 users    Load  2.08  2.40  2.33                  Mar 16 10:41

Mem:KB    REAL            VIRTUAL                       VN PAGER SWAP PAG=
ER
         Tot   Share      Tot    Share    Free           in out     in   =
out
Act 2341416   20052  9279140    56272  436064 count                 44   =
 32
All 3144676   25544 10111364   255624 pages                112    43
Proc: Interrupts
   r   p   d   s   w   Csw  Trp  Sys  Int  Sof  Flt     43 ioflt 29689 to=
tal
   3  40     200  45   80k  40k 186k  17k  438  14k   1348 cow 11 uart0 4=

                                                      1890 zfod 2085=20
uhci0 16
  8.0%Sys   2.3%Intr  5.6%User  0.0%Nice 84.1%Idle      48 ozfod      =20
pcm0 17
|    |    |    |    |    |    |    |    |    | 2%ozfod       ehci0 uhci
=3D=3D=3D=3D+>>> daefr       uhci1 21
                                         55 dtbuf     2123 prcfr 520=20
uhci3 ehci
Namei     Name-cache   Dir-cache    485946 desvn     8196 totfr 1045=20
arcmsr0 30
    Calls    hits   %    hits   %    164802 numvn       22 react 1085=20
cpu0:timer
    14254   14202 100                121388 frevn          pdwak 77 mps0 =
256
                                                   1663632 pdpgs 7020=20
em0:rx 0
Disks   da0   da1   da2   da3   da4   da5   da6        49 intrn 6864=20
em0:tx 0
KB/t   8.20 63.89 11.91 29.66 34.00 17.36  0.00   4829120 wire       =20
em0:link
tps      77  1497     9    19    15     9     0   2117724 act 86 em1:rx 0=

MB/s   0.61 93.43  0.11  0.54  0.51  0.15  0.00  17078072 inact 87 em1:tx=
 0
%busy    26    26     1     3     2     1     0    431968 cache      =20
em1:link
                                                      7832 free       =20
ahci0:ch0
                                                   1694896 buf        =20
ahci0:ch2
655 cpu1:timer
898 cpu11:time
627 cpu2:timer
784 cpu10:time
938 cpu5:timer
1054 cpu13:time
636 cpu4:timer
476 cpu12:time
579 cpu3:timer
702 cpu8:timer
646 cpu6:timer
573 cpu9:timer
670 cpu7:timer
1056 cpu14:time
515 cpu15:time

This is a rather busy (read: extreme demands on the system) time during=20
which I have managed to provoke some really awful behavior, including=20
filesystem stalls.  The system in question has both ufs and zfs=20
filesystems (but won't for much longer) along with running both SMB=20
service (samba) and Postgres.

Of note is that nasty "inact" page count.  It has driven the adaptive=20
ARC code patch (which is on this box) to trim the ARC cache down to the=20
minimum, where it remains pinned.

My reading of the "inact" page count is that pages shouldn't stay in=20
that state on an indefinite basis - - they should either be reactivated=20
(if they're re-used) or invalidated and moved to the "cache" bucket=20
where the VM code can free them.

Buuuuut.... neither is happening over the space of several hours and a=20
look at the RSS of the working processes doesn't show anything=20
interesting -- or different than normal activity in that regard.  17=20
_*gigabytes*_ of inactive pages (out of 24 GB of RAM in total) and=20
they're not being reclaimed?

Time for me to dig into the vm code?

FreeBSD 10.0-STABLE #13 r263037M: Fri Mar 14 14:58:11 CDT 2014=20
[e-mail]:/usr/obj/usr/src/sys/KSD-SMP

--=20
-- Karl
[e-mail]


--------------ms030907030401050008070005
Content-Type: application/pkcs7-signature; name="smime.p7s"
Content-Transfer-Encoding: base64
Content-Disposition: attachment; filename="smime.p7s"
Content-Description: S/MIME Cryptographic Signature

MIAGCSqGSIb3DQEHAqCAMIACAQExCzAJBgUrDgMCGgUAMIAGCSqGSIb3DQEHAQAAoIIFTzCC
BUswggQzoAMCAQICAQgwDQYJKoZIhvcNAQEFBQAwgZ0xCzAJBgNVBAYTAlVTMRAwDgYDVQQI
EwdGbG9yaWRhMRIwEAYDVQQHEwlOaWNldmlsbGUxGTAXBgNVBAoTEEN1ZGEgU3lzdGVtcyBM
TEMxHDAaBgNVBAMTE0N1ZGEgU3lzdGVtcyBMTEMgQ0ExLzAtBgkqhkiG9w0BCQEWIGN1c3Rv
bWVyLXNlcnZpY2VAY3VkYXN5c3RlbXMubmV0MB4XDTEzMDgyNDE5MDM0NFoXDTE4MDgyMzE5
MDM0NFowWzELMAkGA1UEBhMCVVMxEDAOBgNVBAgTB0Zsb3JpZGExFzAVBgNVBAMTDkthcmwg
RGVubmluZ2VyMSEwHwYJKoZIhvcNAQkBFhJrYXJsQGRlbm5pbmdlci5uZXQwggIiMA0GCSqG
SIb3DQEBAQUAA4ICDwAwggIKAoICAQC5n2KBrBmG22nVntVdvgKCB9UcnapNThrW1L+dq6th
d9l4mj+qYMUpJ+8I0rTbY1dn21IXQBoBQmy8t1doKwmTdQ59F0FwZEPt/fGbRgBKVt3Quf6W
6n7kRk9MG6gdD7V9vPpFV41e+5MWYtqGWY3ScDP8SyYLjL/Xgr+5KFKkDfuubK8DeNqdLniV
jHo/vqmIgO+6NgzPGPgmbutzFQXlxUqjiNAAKzF2+Tkddi+WKABrcc/EqnBb0X8GdqcIamO5
SyVmuM+7Zdns7D9pcV16zMMQ8LfNFQCDvbCuuQKMDg2F22x5ekYXpwjqTyfjcHBkWC8vFNoY
5aFMdyiN/Kkz0/kduP2ekYOgkRqcShfLEcG9SQ4LQZgqjMpTjSOGzBr3tOvVn5LkSJSHW2Z8
Q0dxSkvFG2/lsOWFbwQeeZSaBi5vRZCYCOf5tRd1+E93FyQfpt4vsrXshIAk7IK7f0qXvxP4
GDli5PKIEubD2Bn+gp3vB/DkfKySh5NBHVB+OPCoXRUWBkQxme65wBO02OZZt0k8Iq0i4Rci
WV6z+lQHqDKtaVGgMsHn6PoeYhjf5Al5SP+U3imTjF2aCca1iDB5JOccX04MNljvifXgcbJN
nkMgrzmm1ZgJ1PLur/ADWPlnz45quOhHg1TfUCLfI/DzgG7Z6u+oy4siQuFr9QT0MQIDAQAB
o4HWMIHTMAkGA1UdEwQCMAAwEQYJYIZIAYb4QgEBBAQDAgWgMAsGA1UdDwQEAwIF4DAsBglg
hkgBhvhCAQ0EHxYdT3BlblNTTCBHZW5lcmF0ZWQgQ2VydGlmaWNhdGUwHQYDVR0OBBYEFHw4
+LnuALyLA5Cgy7T5ZAX1WzKPMB8GA1UdIwQYMBaAFF3U3hpBZq40HB5VM7B44/gmXiI0MDgG
CWCGSAGG+EIBAwQrFilodHRwczovL2N1ZGFzeXN0ZW1zLm5ldDoxMTQ0My9yZXZva2VkLmNy
bDANBgkqhkiG9w0BAQUFAAOCAQEAZ0L4tQbBd0hd4wuw/YVqEBDDXJ54q2AoqQAmsOlnoxLO
31ehM/LvrTIP4yK2u1VmXtUumQ4Ao15JFM+xmwqtEGsh70RRrfVBAGd7KOZ3GB39FP2TgN/c
L5fJKVxOqvEnW6cL9QtvUlcM3hXg8kDv60OB+LIcSE/P3/s+0tEpWPjxm3LHVE7JmPbZIcJ1
YMoZvHh0NSjY5D0HZlwtbDO7pDz9sZf1QEOgjH828fhtborkaHaUI46pmrMjiBnY6ujXMcWD
pxtikki0zY22nrxfTs5xDWGxyrc/cmucjxClJF6+OYVUSaZhiiHfa9Pr+41okLgsRB0AmNwE
f6ItY3TI8DGCBQowggUGAgEBMIGjMIGdMQswCQYDVQQGEwJVUzEQMA4GA1UECBMHRmxvcmlk
YTESMBAGA1UEBxMJTmljZXZpbGxlMRkwFwYDVQQKExBDdWRhIFN5c3RlbXMgTExDMRwwGgYD
VQQDExNDdWRhIFN5c3RlbXMgTExDIENBMS8wLQYJKoZIhvcNAQkBFiBjdXN0b21lci1zZXJ2
aWNlQGN1ZGFzeXN0ZW1zLm5ldAIBCDAJBgUrDgMCGgUAoIICOzAYBgkqhkiG9w0BCQMxCwYJ
KoZIhvcNAQcBMBwGCSqGSIb3DQEJBTEPFw0xNDAzMTYxNTU2MjhaMCMGCSqGSIb3DQEJBDEW
BBS1GLDmVkM5De90TrlBijuI3XtVzDBsBgkqhkiG9w0BCQ8xXzBdMAsGCWCGSAFlAwQBKjAL
BglghkgBZQMEAQIwCgYIKoZIhvcNAwcwDgYIKoZIhvcNAwICAgCAMA0GCCqGSIb3DQMCAgFA
MAcGBSsOAwIHMA0GCCqGSIb3DQMCAgEoMIG0BgkrBgEEAYI3EAQxgaYwgaMwgZ0xCzAJBgNV
BAYTAlVTMRAwDgYDVQQIEwdGbG9yaWRhMRIwEAYDVQQHEwlOaWNldmlsbGUxGTAXBgNVBAoT
EEN1ZGEgU3lzdGVtcyBMTEMxHDAaBgNVBAMTE0N1ZGEgU3lzdGVtcyBMTEMgQ0ExLzAtBgkq
hkiG9w0BCQEWIGN1c3RvbWVyLXNlcnZpY2VAY3VkYXN5c3RlbXMubmV0AgEIMIG2BgsqhkiG
9w0BCRACCzGBpqCBozCBnTELMAkGA1UEBhMCVVMxEDAOBgNVBAgTB0Zsb3JpZGExEjAQBgNV
BAcTCU5pY2V2aWxsZTEZMBcGA1UEChMQQ3VkYSBTeXN0ZW1zIExMQzEcMBoGA1UEAxMTQ3Vk
YSBTeXN0ZW1zIExMQyBDQTEvMC0GCSqGSIb3DQEJARYgY3VzdG9tZXItc2VydmljZUBjdWRh
c3lzdGVtcy5uZXQCAQgwDQYJKoZIhvcNAQEBBQAEggIAd5yyl8dNfMiH03AiZ0t2MurhEyrk
dBikvzH8O1zuOxUYg1DxK4pqk2Ijf9u6VWUzwv/Qpkd4KmNaTjSuyCCrIgoZXGaBfmuC0H7W
+EMjUIaVc+ZSxH6tEpCtaePQQG/fp//drdFtF1NQacbqLurnclAd5uaFT/fqYHM4PwnfbZha
ySmZu38Qr8ftKTK9FF8xEKl4qT99tvddQgdRe/Q4Yl0B53W/1PC9g/QZ5Y0yOyqQV366h2o2
/pAaWJVfTvSq4iJZEiCGKbIL01lbU+uLNClKFWDAjj3Fm+1T7hTT8PirnxaOHW1eCLM4PqZV
Nd5HWu71BolnnJTbPzLMFoV2DeT9oLdXU2ItaKWlFENxhkAJ0Tl3iocKKjMc04/2rwSt3D13
QkRuDYbiRqeWSaysxdcacg3oWx0QlblfnvsPd7l7wnrP5t5fBzwwMaRW+Vjlw/xUah8VGCQW
77H9aJss38G6JDywes0ZTGZeFJqlBCpMoEl3KAT7jN5yLs2JQmM5AXzhgpe4DWasXlWN/LIQ
m6O46UFfthRucvLRXIGd5dFJnrhEnZKCbsmOqLDrhIa2LODYQvQr17JN9wDqEEh490IJxTHo
DR6Hn1DrYJwcDwqCYigxWWoI98KNlWObROpgvSg+5yrpQKve4jICs1P225OI4eiB7r4YXwJx
05//kaoAAAAAAAA=
--------------ms030907030401050008070005--