performance of jailed processes

Can anyone explain why jailed processes seem to perform much worse
than non-jailed processes in recent -CURRENT?

Specifically, running a query against a remote MySQL server from
inside a jail takes an order of magnitude more time than from outside
the jail.  Tcpdump shows that the TCP packets carrying the result are
evenly spaced, so this is not a matter of the server timing out on a
DNS lookup or anything like that.

Running a configure script also takes much longer inside the jail than
outisde, and again, progress is even (though slow), so it is clearly
not a matter of DNS timing out.

There is no NFS or NIS in the equation either.  Parts of the file
space inside the jail is a nullfs mount, but we've also tried without
nullfs.

The system currently uses SCHED_ULE, but we had similar trouble with
SCHED_4BSD on 5.1-RELEASE before we went -CURRENT.

The machine currently has ~2600 processes running in ~400 jails.  Is
it conceivable that be scalability issues, perhaps in the credentials
code, could cause vastly increased syscall overhead for jailed
processes?

DES
--=20
Dag-Erling Sm=C3=B8rgrav - [e-mail]
_______________________________________________
[e-mail] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[e-mail]"

> -------------------------------------------------------------------------- <

發信人: [e-mail] (Julian Elischer), 看板: FB_current
標  題: Re: performance of jailed processes
發信站: NCTU CSIE FreeBSD Server (Tue Mar 30 18:03:46 2004)
轉信站: ptt!FreeBSD.csie.NCTU!not-for-mail

Dag-Erling Sm繪rgrav wrote:
> Can anyone explain why jailed processes seem to perform much worse
> than non-jailed processes in recent -CURRENT?
> 
> Specifically, running a query against a remote MySQL server from
> inside a jail takes an order of magnitude more time than from outside
> the jail.  Tcpdump shows that the TCP packets carrying the result are
> evenly spaced, so this is not a matter of the server timing out on a
> DNS lookup or anything like that.
> 
> Running a configure script also takes much longer inside the jail than
> outisde, and again, progress is even (though slow), so it is clearly
> not a matter of DNS timing out.
> 
> There is no NFS or NIS in the equation either.  Parts of the file
> space inside the jail is a nullfs mount, but we've also tried without
> nullfs.
> 
> The system currently uses SCHED_ULE, but we had similar trouble with
> SCHED_4BSD on 5.1-RELEASE before we went -CURRENT.
> 
> The machine currently has ~2600 processes running in ~400 jails.  Is
> it conceivable that be scalability issues, perhaps in the credentials
> code, could cause vastly increased syscall overhead for jailed

that suggests 400 addresses, which suggests that lots of linked lists are being 
traversed  for received packets..

> processes?
> 
> DES


-- 
+------------------------------------+       ______ _  __
|   __--_|\  Julian Elischer         |       \     U \/ / hard at work in
|  /       \ [e-mail]     +------>x   USA    \ a very strange
| (   OZ    )                                \___   ___ | country !
+- X_.---._/    presently in San Francisco       \_/   \\
           v

_______________________________________________
[e-mail] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[e-mail]"

> -------------------------------------------------------------------------- <

發信人: [e-mail] (Robert Watson), 看板: FB_current
標  題: Re: performance of jailed processes
發信站: NCTU CSIE FreeBSD Server (Tue Mar 30 18:39:34 2004)
轉信站: ptt!FreeBSD.csie.NCTU!not-for-mail

On Tue, 30 Mar 2004, Dag-Erling Sm=F8rgrav wrote:

> Can anyone explain why jailed processes seem to perform much worse than
> non-jailed processes in recent -CURRENT?=20
>=20
> Specifically, running a query against a remote MySQL server from inside
> a jail takes an order of magnitude more time than from outside the jail.=
=20
> Tcpdump shows that the TCP packets carrying the result are evenly
> spaced, so this is not a matter of the server timing out on a DNS lookup
> or anything like that.=20
>=20
> Running a configure script also takes much longer inside the jail than
> outisde, and again, progress is even (though slow), so it is clearly not
> a matter of DNS timing out.=20
>=20
> There is no NFS or NIS in the equation either.  Parts of the file space
> inside the jail is a nullfs mount, but we've also tried without nullfs.=
=20
>=20
> The system currently uses SCHED_ULE, but we had similar trouble with
> SCHED_4BSD on 5.1-RELEASE before we went -CURRENT.=20
>=20
> The machine currently has ~2600 processes running in ~400 jails.  Is it
> conceivable that be scalability issues, perhaps in the credentials code,
> could cause vastly increased syscall overhead for jailed processes?=20

This is bizarre, although it sounds like you've covered a lot of the
interesting potential causes already.  The short answer is, no: the
overhead of jail on all system call paths is very, very low, especially
given that the most critical fields in the prison structure are immutable
(IP address), and so shouldn't impact performance much at all.  And, in
fact, since we're using thread-local credential references, the entire
thing should be quite cheap once the jail is created.

If I had to guess at causes, some of which you've looked at, I'd chase the
following:

- DNS -- I know you mentioned it, but I'd check anyway.  Especially if
  resolv.conf has bad DNS servers in it in the jails, etc.  You might try
  writing a trivial gethostbyname() test app and timing it in and out of
  the jail.  Also look at the reverse lookup done by the MySQL server.
  The impact of the source IP address might be particularly interesting.

- Compare the impact of jail() vs chroot() for configure performance.  The
  jail code barely if at all impacts vnode operations and lookup relative
  to chroot(), but the impact of chroot() might well be interesting to
  look at.=20

- Nullfs should slow down file system I/O some, especially lookups, and
  the more nullfs mounts there are, the more impact it should have (by
  dramatically increasing the number of vnodes in use).  However, if
  you've already done the side-by-side comparison...  Again, chroot() vs
  jail() would be interesting.

- It would be interesting to know if applications outside the jail bound
  to various IP addresses see performance differences depending on the IP
  used.  We have hashed IP address lookup, but there are some operations
  in the stack that require walking the list of addresses, etc.  If the
  non-jailed software always uses the first address because they're all in
  the same subnet, that might conceivably make a difference.  Taking jail
  out of the picture in some basic micro-benchmarks might help here also.=
=20

Can you identify any micro-benchmarks rather than macro-benchmarks that
reflect a significant difference?

Robert N M Watson             FreeBSD Core Team, TrustedBSD Projects
[e-mail]      Senior Research Scientist, McAfee Research


_______________________________________________
[e-mail] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[e-mail]"

> -------------------------------------------------------------------------- <

發信人: [e-mail] (Dag-Erling Sm鷨grav), 看板: FB_current
標  題: Re: performance of jailed processes
發信站: NCTU CSIE FreeBSD Server (Tue Mar 30 19:05:42 2004)
轉信站: ptt!FreeBSD.csie.NCTU!not-for-mail

Robert Watson <[e-mail]> writes:
> - DNS -- I know you mentioned it, but I'd check anyway.  Especially if
>   resolv.conf has bad DNS servers in it in the jails, etc.  You might try
>   writing a trivial gethostbyname() test app and timing it in and out of
>   the jail.  Also look at the reverse lookup done by the MySQL server.
>   The impact of the source IP address might be particularly interesting.

Packet traces already show that there is no delay between query and
reply, the reply just takes a long time to transmit.

> - It would be interesting to know if applications outside the jail bound
>   to various IP addresses see performance differences depending on the IP
>   used.  We have hashed IP address lookup, but there are some operations
>   in the stack that require walking the list of addresses, etc.  If the
>   non-jailed software always uses the first address because they're all in
>   the same subnet, that might conceivably make a difference.  Taking jail
>   out of the picture in some basic micro-benchmarks might help here also.=
=20

Non-jailed software always uses the first IP address, which is in its
own subnet.  The jails draw from a pool of ~1000 IP addresses on the
same interface, but in a different subnet.  The jail I've been testing
in is about a quarter of the way down the list.

> Can you identify any micro-benchmarks rather than macro-benchmarks that
> reflect a significant difference?

haven't had much luck with that...  fetch, for instance, doesn't seem
to suffer, but with mysql the difference is dramatic:

(outside jail)
1 row in set (0.01 sec)

(inside jail)
1 row in set (13.20 sec)

note that 13 seconds is far too short for a DNS issue, and that the
time reported is measured *after* login (i.e. after any DNS lookup)

DES
--=20
Dag-Erling Sm=F8rgrav - [e-mail]
_______________________________________________
[e-mail] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[e-mail]"

> -------------------------------------------------------------------------- <

發信人: [e-mail] (Pawel Jakub Dawidek), 看板: FB_current
標  題: Re: performance of jailed processes
發信站: NCTU CSIE FreeBSD Server (Tue Mar 30 19:48:42 2004)
轉信站: ptt!FreeBSD.csie.NCTU!not-for-mail


--znFrcFTOc9PDN8kJ
Content-Type: text/plain; charset=iso-8859-2
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

On Tue, Mar 30, 2004 at 09:09:35PM +0200, Dag-Erling Sm?rgrav wrote:
+> > Can you identify any micro-benchmarks rather than macro-benchmarks that
+> > reflect a significant difference?
+>=20
+> haven't had much luck with that...  fetch, for instance, doesn't seem
+> to suffer, but with mysql the difference is dramatic:
+>=20
+> (outside jail)
+> 1 row in set (0.01 sec)
+>=20
+> (inside jail)
+> 1 row in set (13.20 sec)
+>=20
+> note that 13 seconds is far too short for a DNS issue, and that the
+> time reported is measured *after* login (i.e. after any DNS lookup)

I'm wondering if this piece of code is responsible for this delay:
(sys/netinet/in_pcb.c:551)

	if (laddr.s_addr =3D=3D INADDR_ANY && jailed(socred)) {
		bzero(&sa, sizeof(sa));
		sa.sin_addr.s_addr =3D htonl(prison_getip(socred));
		sa.sin_len =3D sizeof(sa);
		sa.sin_family =3D AF_INET;
		error =3D in_pcbbind_setup(inp, (struct sockaddr *)&sa,
		    &laddr.s_addr, &lport, cred);
		if (error)
			return (error);
	}

Maybe you can test it by putting a printf() after this if() and check
if it is printed with fetch and with mysql.

--=20
Pawel Jakub Dawidek                       http://www.FreeBSD.org
[e-mail]                           http://garage.freebsd.pl
FreeBSD committer                         Am I Evil? Yes, I Am!

--znFrcFTOc9PDN8kJ
Content-Type: application/pgp-signature
Content-Disposition: inline

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (FreeBSD)

iD8DBQFAadDkForvXbEpPzQRAocwAJ9CRyst2AoSwS+BznjRo4tG6l6uYQCcCZNq
7yQ1/Djqc79vmixvVZJrbIw=
=oZyv
-----END PGP SIGNATURE-----

--znFrcFTOc9PDN8kJ--

> -------------------------------------------------------------------------- <

發信人: [e-mail] (Eirik Oeverby), 看板: FB_current
標  題: Re: performance of jailed processes
發信站: NCTU CSIE FreeBSD Server (Tue Mar 30 20:02:56 2004)
轉信站: ptt!FreeBSD.csie.NCTU!not-for-mail

Hi there,

and I thought I was using jails on a 'big' scale... 400 jails on one 
single box, that's pretty amazing! What kind of jails are these, i.e. 
what are they used for? Encapsulating single processes/tasks only, or 
more complex things too? And what hardware are you on, CPU and memory-wise?

/Eirik

Dag-Erling Sm鷨grav wrote:
> Robert Watson <[e-mail]> writes:
> 
>>- DNS -- I know you mentioned it, but I'd check anyway.  Especially if
>>  resolv.conf has bad DNS servers in it in the jails, etc.  You might try
>>  writing a trivial gethostbyname() test app and timing it in and out of
>>  the jail.  Also look at the reverse lookup done by the MySQL server.
>>  The impact of the source IP address might be particularly interesting.
> 
> 
> Packet traces already show that there is no delay between query and
> reply, the reply just takes a long time to transmit.
> 
> 
>>- It would be interesting to know if applications outside the jail bound
>>  to various IP addresses see performance differences depending on the IP
>>  used.  We have hashed IP address lookup, but there are some operations
>>  in the stack that require walking the list of addresses, etc.  If the
>>  non-jailed software always uses the first address because they're all in
>>  the same subnet, that might conceivably make a difference.  Taking jail
>>  out of the picture in some basic micro-benchmarks might help here also. 
> 
> 
> Non-jailed software always uses the first IP address, which is in its
> own subnet.  The jails draw from a pool of ~1000 IP addresses on the
> same interface, but in a different subnet.  The jail I've been testing
> in is about a quarter of the way down the list.
> 
> 
>>Can you identify any micro-benchmarks rather than macro-benchmarks that
>>reflect a significant difference?
> 
> 
> haven't had much luck with that...  fetch, for instance, doesn't seem
> to suffer, but with mysql the difference is dramatic:
> 
> (outside jail)
> 1 row in set (0.01 sec)
> 
> (inside jail)
> 1 row in set (13.20 sec)
> 
> note that 13 seconds is far too short for a DNS issue, and that the
> time reported is measured *after* login (i.e. after any DNS lookup)
> 
> DES
_______________________________________________
[e-mail] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[e-mail]"

> -------------------------------------------------------------------------- <

發信人: [e-mail] (Robert Watson), 看板: FB_current
標  題: Re: performance of jailed processes
發信站: NCTU CSIE FreeBSD Server (Tue Mar 30 20:15:07 2004)
轉信站: ptt!FreeBSD.csie.NCTU!not-for-mail

On Tue, 30 Mar 2004, Dag-Erling Sm=F8rgrav wrote:

> Robert Watson <[e-mail]> writes:
> > - DNS -- I know you mentioned it, but I'd check anyway.  Especially if
> >   resolv.conf has bad DNS servers in it in the jails, etc.  You might t=
ry
> >   writing a trivial gethostbyname() test app and timing it in and out o=
f
> >   the jail.  Also look at the reverse lookup done by the MySQL server.
> >   The impact of the source IP address might be particularly interesting=
=2E
>=20
> Packet traces already show that there is no delay between query and
> reply, the reply just takes a long time to transmit.=20

Somewhat more painful suggestion, but could you generate ktraces against a
mysql client doing the query inside and out of jail, then using whatever
flag sets relative timestamps on kdump, diff the two and see where the
substantial differences begin?

> > - It would be interesting to know if applications outside the jail boun=
d
> >   to various IP addresses see performance differences depending on the =
IP
> >   used.  We have hashed IP address lookup, but there are some operation=
s
> >   in the stack that require walking the list of addresses, etc.  If the
> >   non-jailed software always uses the first address because they're all=
 in
> >   the same subnet, that might conceivably make a difference.  Taking ja=
il
> >   out of the picture in some basic micro-benchmarks might help here als=
o.=20
>=20
> Non-jailed software always uses the first IP address, which is in its
> own subnet.  The jails draw from a pool of ~1000 IP addresses on the
> same interface, but in a different subnet.  The jail I've been testing
> in is about a quarter of the way down the list.=20
>=20
> > Can you identify any micro-benchmarks rather than macro-benchmarks that
> > reflect a significant difference?
>=20
> haven't had much luck with that...  fetch, for instance, doesn't seem
> to suffer, but with mysql the difference is dramatic:
>=20
> (outside jail)
> 1 row in set (0.01 sec)
>=20
> (inside jail)
> 1 row in set (13.20 sec)
>=20
> note that 13 seconds is far too short for a DNS issue, and that the time
> reported is measured *after* login (i.e. after any DNS lookup)=20

13 seconds is too long for most of the potential things I have in mind...

Robert N M Watson             FreeBSD Core Team, TrustedBSD Projects
[e-mail]      Senior Research Scientist, McAfee Research


_______________________________________________
[e-mail] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[e-mail]"

> -------------------------------------------------------------------------- <

發信人: [e-mail] (Pawel Jakub Dawidek), 看板: FB_current
標  題: Re: performance of jailed processes
發信站: NCTU CSIE FreeBSD Server (Tue Mar 30 20:15:07 2004)
轉信站: ptt!FreeBSD.csie.NCTU!not-for-mail


--kGqy73/z6kAbhglh
Content-Type: text/plain; charset=iso-8859-2
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

On Tue, Mar 30, 2004 at 09:09:35PM +0200, Dag-Erling Sm?rgrav wrote:
+> > Can you identify any micro-benchmarks rather than macro-benchmarks that
+> > reflect a significant difference?
+>=20
+> haven't had much luck with that...  fetch, for instance, doesn't seem
+> to suffer, but with mysql the difference is dramatic:
+>=20
+> (outside jail)
+> 1 row in set (0.01 sec)
+>=20
+> (inside jail)
+> 1 row in set (13.20 sec)
+>=20
+> note that 13 seconds is far too short for a DNS issue, and that the
+> time reported is measured *after* login (i.e. after any DNS lookup)

Could you produce some ktraces inside and outside jail?

--=20
Pawel Jakub Dawidek                       http://www.FreeBSD.org
[e-mail]                           http://garage.freebsd.pl
FreeBSD committer                         Am I Evil? Yes, I Am!

--kGqy73/z6kAbhglh
Content-Type: application/pgp-signature
Content-Disposition: inline

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (FreeBSD)

iD8DBQFAadsvForvXbEpPzQRAvL4AJ0WynbIjMl4bj0leCMv2Eyjoii7CgCfdhUi
jXS0C0HvtJVrQ4Ku20OPNa8=
=kSgu
-----END PGP SIGNATURE-----

--kGqy73/z6kAbhglh--

> -------------------------------------------------------------------------- <

發信人: [e-mail] (Dag-Erling Sm鷨grav), 看板: FB_current
標  題: Re: performance of jailed processes
發信站: NCTU CSIE FreeBSD Server (Tue Mar 30 20:15:07 2004)
轉信站: ptt!FreeBSD.csie.NCTU!not-for-mail

Eirik Oeverby <[e-mail]> writes:
> and I thought I was using jails on a 'big' scale... 400 jails on one
> single box, that's pretty amazing! What kind of jails are these,
> i.e. what are they used for? Encapsulating single processes/tasks
> only, or more complex things too? And what hardware are you on, CPU
> and memory-wise?

Virtual hosting.  Dual Athlon MP 3400+, 2 GB dual-channel PC3200 RAM.

DES
--=20
Dag-Erling Sm=F8rgrav - [e-mail]
_______________________________________________
[e-mail] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[e-mail]"

> -------------------------------------------------------------------------- <

發信人: [e-mail] (Dag-Erling Sm鷨grav), 看板: FB_current
標  題: Re: performance of jailed processes
發信站: NCTU CSIE FreeBSD Server (Tue Mar 30 20:15:07 2004)
轉信站: ptt!FreeBSD.csie.NCTU!not-for-mail

Robert Watson <[e-mail]> writes:
> Somewhat more painful suggestion, but could you generate ktraces against a
> mysql client doing the query inside and out of jail, then using whatever
> flag sets relative timestamps on kdump, diff the two and see where the
> substantial differences begin?

I'll give it a shot tomorrow.

> 13 seconds is too long for most of the potential things I have in mind...

although the query only returns one row, it's a pretty big row, so 13
seconds could be explained by per-syscall or per-packet overhead.

DES
--=20
Dag-Erling Sm=F8rgrav - [e-mail]
_______________________________________________
[e-mail] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[e-mail]"

> -------------------------------------------------------------------------- <

發信人: [e-mail] (Dag-Erling Sm鷨grav), 看板: FB_current
標  題: Re: performance of jailed processes
發信站: NCTU CSIE FreeBSD Server (Tue Mar 30 20:15:07 2004)
轉信站: ptt!FreeBSD.csie.NCTU!not-for-mail

Pawel Jakub Dawidek <[e-mail]> writes:
> I'm wondering if this piece of code is responsible for this delay:
> (sys/netinet/in_pcb.c:551)

that code is in the connect path, so it's not relevant.

DES
--=20
Dag-Erling Sm=F8rgrav - [e-mail]
_______________________________________________
[e-mail] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[e-mail]"

> -------------------------------------------------------------------------- <

發信人: [e-mail] (Robert Watson), 看板: FB_current
標  題: Re: performance of jailed processes
發信站: NCTU CSIE FreeBSD Server (Tue Mar 30 20:49:05 2004)
轉信站: ptt!FreeBSD.csie.NCTU!not-for-mail


On Tue, 30 Mar 2004, Dag-Erling Sm=F8rgrav wrote:

> Robert Watson <[e-mail]> writes:
> > Somewhat more painful suggestion, but could you generate ktraces agains=
t a
> > mysql client doing the query inside and out of jail, then using whateve=
r
> > flag sets relative timestamps on kdump, diff the two and see where the
> > substantial differences begin?
>=20
> I'll give it a shot tomorrow.

I've had quite a bit of luck resolving mysql problems in jail using this
approach, fwiw, during some confusion relating to UNIX domain sockets at
an ISP I provide some help to.

> > 13 seconds is too long for most of the potential things I have in mind.=
=2E.
>=20
> although the query only returns one row, it's a pretty big row, so 13
> seconds could be explained by per-syscall or per-packet overhead.=20

Theory goes that there should be no per-read/write system call change in
behavior for TCP with jail.  Jail impacts bind/connect, and potentially
each I/O on UDP for an unbound socket using sendto.=20

Robert N M Watson             FreeBSD Core Team, TrustedBSD Projects
[e-mail]      Senior Research Scientist, McAfee Research


_______________________________________________
[e-mail] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[e-mail]"

> -------------------------------------------------------------------------- <

發信人: [e-mail] (Dag-Erling Sm鷨grav), 看板: FB_current
標  題: Re: performance of jailed processes
發信站: NCTU CSIE FreeBSD Server (Tue Mar 30 21:02:34 2004)
轉信站: ptt!FreeBSD.csie.NCTU!not-for-mail

Robert Watson <[e-mail]> writes:
> On Tue, 30 Mar 2004, Dag-Erling Sm=F8rgrav wrote:
> > although the query only returns one row, it's a pretty big row, so 13
> > seconds could be explained by per-syscall or per-packet overhead.
> Theory goes that there should be no per-read/write system call change in
> behavior for TCP with jail.  Jail impacts bind/connect, and potentially
> each I/O on UDP for an unbound socket using sendto.

root@outside /# /usr/bin/time -- sh -c 'echo "my sql query;" | mysql -comma=
nd -line -arguments >/dev/null'
        0.06 real         0.00 user         0.03 sys
root@outside /# jexec 55 zsh
root@inside /# /usr/bin/time -- sh -c 'echo "my sql query;" | mysql -comman=
d -line -arguments >/dev/null'
       13.65 real         0.01 user         0.04 sys

so it's definitely not CPU overhead - more likely a scheduling problem.

DES
--=20
Dag-Erling Sm=F8rgrav - [e-mail]
_______________________________________________
[e-mail] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[e-mail]"

> -------------------------------------------------------------------------- <

發信人: [e-mail] (Dag-Erling Sm鷨grav), 看板: FB_current
標  題: Re: performance of jailed processes
發信站: NCTU CSIE FreeBSD Server (Tue Mar 30 21:39:57 2004)
轉信站: ptt!FreeBSD.csie.NCTU!not-for-mail

Robert Watson <[e-mail]> writes:
> Somewhat more painful suggestion, but could you generate ktraces against a
> mysql client doing the query inside and out of jail, then using whatever
> flag sets relative timestamps on kdump, diff the two and see where the
> substantial differences begin?

OK, ktrace shows some interesting differences.  I used ktrace -tcsw to
get syscalls, signals and context switches.

Here's mysql outside the jail reading the row returned from the query
(from kdump -tcs -R):

 12698 mysql    0.000219 CALL  read(0x3,0x806b000,0x82af)
 12698 mysql    0.000117 RET   read 27981/0x6d4d
 12698 mysql    0.000029 CALL  read(0x3,0x8071d4d,0x1562)
 12698 mysql    0.000189 RET   read 2896/0xb50
 12698 mysql    0.000005 CALL  read(0x3,0x807289d,0xa12)
 12698 mysql    0.000394 RET   read 2578/0xa12

here's mysql inside the jail doing the same thing:

 12703 mysql    0.000264 CALL  read(0x3,0x8070000,0x82af)
 12703 mysql    0.000043 RET   read 469/0x1d5
 12703 mysql    0.000005 CALL  read(0x3,0x80701d5,0x80da)
 12703 mysql    1.208382 RET   read 4344/0x10f8
 12703 mysql    0.001126 CALL  read(0x3,0x80712cd,0x6fe2)
 12703 mysql    0.000027 RET   read 1448/0x5a8
 12703 mysql    0.000032 CALL  read(0x3,0x8071875,0x6a3a)
 12703 mysql    0.000183 RET   read 1448/0x5a8
 12703 mysql    0.000029 CALL  read(0x3,0x8071e1d,0x6492)
 12703 mysql    1.197240 RET   read 1448/0x5a8
 12703 mysql    0.000529 CALL  read(0x3,0x80723c5,0x5eea)
 12703 mysql    1.199249 RET   read 2896/0xb50
 12703 mysql    0.000535 CALL  read(0x3,0x8072f15,0x539a)
 12703 mysql    1.199303 RET   read 2896/0xb50
 12703 mysql    0.000701 CALL  read(0x3,0x8073a65,0x484a)
 12703 mysql    0.002045 RET   read 2896/0xb50
 12703 mysql    0.000028 CALL  read(0x3,0x80745b5,0x3cfa)
 12703 mysql    1.197229 RET   read 1448/0x5a8
 12703 mysql    0.000261 CALL  read(0x3,0x8074b5d,0x3752)
 12703 mysql    1.200116 RET   read 2896/0xb50
 12703 mysql    0.001101 CALL  read(0x3,0x80756ad,0x2c02)
 12703 mysql    1.199051 RET   read 2896/0xb50
 12703 mysql    0.000307 CALL  read(0x3,0x80761fd,0x20b2)
 12703 mysql    1.200152 RET   read 2896/0xb50
 12703 mysql    0.000735 CALL  read(0x3,0x8076d4d,0x1562)
 12703 mysql    1.198556 RET   read 2896/0xb50
 12703 mysql    0.000135 CALL  read(0x3,0x807789d,0xa12)
 12703 mysql    0.001835 RET   read 2578/0xa12
 12703 mysql    0.000208 CALL  break(0x8083000)
 12703 mysql    0.000009 RET   break 0

let's look at an inside trace *with* context switches and see where
the longest delays (>=3D 0.1 s) are:

# kdump -R | egrep -B1 ' (0\.[1-9]|1\.[0-9])'
 12703 mysql    0.000003 CSW  stop kernel
 12703 mysql    1.193959 CSW  resume kernel
--
 12703 mysql    0.000086 CSW  stop kernel
 12703 mysql    1.208101 CSW  resume kernel
--
 12703 mysql    0.000197 CSW  stop kernel
 12703 mysql    1.196910 CSW  resume kernel
--
 12703 mysql    0.000005 CSW  stop kernel
 12703 mysql    1.199010 CSW  resume kernel
--
 12703 mysql    0.000216 CSW  stop kernel
 12703 mysql    1.199035 CSW  resume kernel
--
 12703 mysql    0.000006 CSW  stop kernel
 12703 mysql    1.197189 CSW  resume kernel
--
 12703 mysql    0.000007 CSW  stop kernel
 12703 mysql    1.199705 CSW  resume kernel
--
 12703 mysql    0.000747 CSW  stop kernel
 12703 mysql    1.198046 CSW  resume kernel
--
 12703 mysql    0.000206 CSW  stop kernel
 12703 mysql    1.199760 CSW  resume kernel
--
 12703 mysql    0.000250 CSW  stop kernel
 12703 mysql    1.198188 CSW  resume kernel

oops, they're all in context switches...

DES
--=20
Dag-Erling Sm=F8rgrav - [e-mail]
_______________________________________________
[e-mail] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[e-mail]"

> -------------------------------------------------------------------------- <

發信人: [e-mail] (Pawel Jakub Dawidek), 看板: FB_current
標  題: Re: performance of jailed processes
發信站: NCTU CSIE FreeBSD Server (Wed Mar 31 02:02:13 2004)
轉信站: ptt!FreeBSD.csie.NCTU!not-for-mail


--GI1PA4Sghwo1XriW
Content-Type: text/plain; charset=iso-8859-2
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

On Tue, Mar 30, 2004 at 11:17:39PM +0200, Dag-Erling Sm?rgrav wrote:
+> Robert Watson <[e-mail]> writes:
+> > On Tue, 30 Mar 2004, Dag-Erling Sm?rgrav wrote:
+> > > although the query only returns one row, it's a pretty big row, so 13
+> > > seconds could be explained by per-syscall or per-packet overhead.
+> > Theory goes that there should be no per-read/write system call change =
in
+> > behavior for TCP with jail.  Jail impacts bind/connect, and potentially
+> > each I/O on UDP for an unbound socket using sendto.
+>=20
+> root@outside /# /usr/bin/time -- sh -c 'echo "my sql query;" | mysql -co=
mmand -line -arguments >/dev/null'
+>         0.06 real         0.00 user         0.03 sys
+> root@outside /# jexec 55 zsh
+> root@inside /# /usr/bin/time -- sh -c 'echo "my sql query;" | mysql -com=
mand -line -arguments >/dev/null'
+>        13.65 real         0.01 user         0.04 sys
+>=20
+> so it's definitely not CPU overhead - more likely a scheduling problem.

Could you try chroot to / ? By doing:

	# jail / test <YOUR_IP> `which zsh`
	# <your test>

?

--=20
Pawel Jakub Dawidek                       http://www.FreeBSD.org
[e-mail]                           http://garage.freebsd.pl
FreeBSD committer                         Am I Evil? Yes, I Am!

--GI1PA4Sghwo1XriW
Content-Type: application/pgp-signature
Content-Disposition: inline

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (FreeBSD)

iD8DBQFAafLOForvXbEpPzQRAiZ9AKD2q8J/u3RmLjI7JkDxTCE2QZzhqwCgptXF
D7std9sFCBCo/VOVVMk/7fc=
=16Wn
-----END PGP SIGNATURE-----

--GI1PA4Sghwo1XriW--

> -------------------------------------------------------------------------- <

發信人: [e-mail] (Dag-Erling Sm鷨grav), 看板: FB_current
標  題: Re: performance of jailed processes
發信站: NCTU CSIE FreeBSD Server (Wed Mar 31 02:02:13 2004)
轉信站: ptt!FreeBSD.csie.NCTU!not-for-mail

Pawel Jakub Dawidek <[e-mail]> writes:
> Could you try chroot to / ? By doing:
>
> 	# jail / test <YOUR_IP> `which zsh`
> 	# <your test>
>
> ?

hmm, interesting, that worked fine...

DES
--=20
Dag-Erling Sm=F8rgrav - [e-mail]
_______________________________________________
[e-mail] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[e-mail]"

> -------------------------------------------------------------------------- <

發信人: [e-mail] (Robert Watson), 看板: FB_current
標  題: Re: performance of jailed processes
發信站: NCTU CSIE FreeBSD Server (Wed Mar 31 02:02:13 2004)
轉信站: ptt!FreeBSD.csie.NCTU!not-for-mail


On Wed, 31 Mar 2004, Dag-Erling Sm=F8rgrav wrote:

> Pawel Jakub Dawidek <[e-mail]> writes:
> > Could you try chroot to / ? By doing:
> >
> > =09# jail / test <YOUR_IP> `which zsh`
> > =09# <your test>
> >
> > ?
>=20
> hmm, interesting, that worked fine...

Suggests perhaps one of the following:

Maybe /etc is configured differently -- perhaps libmap.conf or the like?

Maybe the software build for the base system or application is
substantially different.

If you plug 'YOUR_IP' with variations, does it change things?

Robert N M Watson             FreeBSD Core Team, TrustedBSD Projects
[e-mail]      Senior Research Scientist, McAfee Research

_______________________________________________
[e-mail] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[e-mail]"

> -------------------------------------------------------------------------- <

發信人: [e-mail] (Dag-Erling Sm鷨grav), 看板: FB_current
標  題: Re: performance of jailed processes
發信站: NCTU CSIE FreeBSD Server (Wed Mar 31 02:02:13 2004)
轉信站: ptt!FreeBSD.csie.NCTU!not-for-mail

Robert Watson <[e-mail]> writes:
> If you plug 'YOUR_IP' with variations, does it change things?

Aha!  using the primary IP address works fine, but using one of the
jail IPs does not.

(before anyone asks, I've already spent an entire day verifying that
this is not a firewall issue)

DES
--=20
Dag-Erling Sm=F8rgrav - [e-mail]
_______________________________________________
[e-mail] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[e-mail]"

> -------------------------------------------------------------------------- <

發信人: [e-mail] (Julian Elischer), 看板: FB_current
標  題: Re: performance of jailed processes
發信站: NCTU CSIE FreeBSD Server (Wed Mar 31 02:02:13 2004)
轉信站: ptt!FreeBSD.csie.NCTU!not-for-mail



On Wed, 31 Mar 2004, Dag-Erling [iso-8859-1] Sm=F8rgrav wrote:

> Robert Watson <[e-mail]> writes:
> > If you plug 'YOUR_IP' with variations, does it change things?
>=20
> Aha!  using the primary IP address works fine, but using one of the
> jail IPs does not.

ANY of them?
try the first added and the last added. do they have different delays?


_______________________________________________
[e-mail] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[e-mail]"

> -------------------------------------------------------------------------- <

發信人: [e-mail] (Dag-Erling Sm鷨grav), 看板: FB_current
標  題: Re: performance of jailed processes
發信站: NCTU CSIE FreeBSD Server (Wed Mar 31 02:02:13 2004)
轉信站: ptt!FreeBSD.csie.NCTU!not-for-mail

Julian Elischer <[e-mail]> writes:
> On Wed, 31 Mar 2004, Dag-Erling [iso-8859-1] Sm=F8rgrav wrote:
> > Robert Watson <[e-mail]> writes:
> > > If you plug 'YOUR_IP' with variations, does it change things?
> > Aha!  using the primary IP address works fine, but using one of the
> > jail IPs does not.
> ANY of them?
> try the first added and the last added. do they have different delays?

No noticeable difference.  I get more variation from one run to
another (about 10%) then I get from changing the IP address.

DES
--=20
Dag-Erling Sm=F8rgrav - [e-mail]
_______________________________________________
[e-mail] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[e-mail]"

> -------------------------------------------------------------------------- <

發信人: [e-mail] (Robert Watson), 看板: FB_current
標  題: Re: performance of jailed processes
發信站: NCTU CSIE FreeBSD Server (Wed Mar 31 02:02:13 2004)
轉信站: ptt!FreeBSD.csie.NCTU!not-for-mail


On Wed, 31 Mar 2004, Dag-Erling Sm=F8rgrav wrote:

> Robert Watson <[e-mail]> writes:
> > If you plug 'YOUR_IP' with variations, does it change things?
>=20
> Aha!  using the primary IP address works fine, but using one of the jail
> IPs does not.
>=20
> (before anyone asks, I've already spent an entire day verifying that
> this is not a firewall issue)=20

I'd be very interested in knowing if changing your application to bind
alternative IP addresses rather than using jail to force the binding to an
alternative address changes the performance results.  I.e., are we looking
at a problem with additional aliases and not a problem with jail at all...=
=20

Another thing I'd be interested to know is: if you instrument the
application to get the socket buffer size using getsockopt() and
SO_{RCV,SND}BUF in the in-jail and out-of-jail cases.

Robert N M Watson             FreeBSD Core Team, TrustedBSD Projects
[e-mail]      Senior Research Scientist, McAfee Research


_______________________________________________
[e-mail] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[e-mail]"

> -------------------------------------------------------------------------- <

發信人: [e-mail] (Dag-Erling Sm鷨grav), 看板: FB_current
標  題: Re: performance of jailed processes
發信站: NCTU CSIE FreeBSD Server (Wed Mar 31 02:02:13 2004)
轉信站: ptt!FreeBSD.csie.NCTU!not-for-mail

Robert Watson <[e-mail]> writes:
> I'd be very interested in knowing if changing your application to bind
> alternative IP addresses rather than using jail to force the binding to an
> alternative address changes the performance results.  I.e., are we looking
> at a problem with additional aliases and not a problem with jail at all...

I reproduced the problem with scp, then threw in -oBindAddress=3Dfoo.
It seems you're on to something; running it outside any jail but bound
to one of the aliases gave the same symptoms as running it from inside
a jail.

DES
--=20
Dag-Erling Sm=F8rgrav - [e-mail]
_______________________________________________
[e-mail] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[e-mail]"

> -------------------------------------------------------------------------- <

發信人: [e-mail] (Robert Watson), 看板: FB_current
標  題: Re: performance of jailed processes
發信站: NCTU CSIE FreeBSD Server (Wed Mar 31 02:02:13 2004)
轉信站: ptt!FreeBSD.csie.NCTU!not-for-mail


On Wed, 31 Mar 2004, Dag-Erling Sm=F8rgrav wrote:

> Robert Watson <[e-mail]> writes:
> > I'd be very interested in knowing if changing your application to bind
> > alternative IP addresses rather than using jail to force the binding to=
 an
> > alternative address changes the performance results.  I.e., are we look=
ing
> > at a problem with additional aliases and not a problem with jail at all=
=2E..
>=20
> I reproduced the problem with scp, then threw in -oBindAddress=3Dfoo.  It
> seems you're on to something; running it outside any jail but bound to
> one of the aliases gave the same symptoms as running it from inside a
> jail.=20

Are your aliases configured on lo0, or on the ethernet interface?  Could
we see some excerpted ifconfig output for your interface (perhaps only ten
-- first five, last five of the 2000+ IP addresses :-).=20

Robert N M Watson             FreeBSD Core Team, TrustedBSD Projects
[e-mail]      Senior Research Scientist, McAfee Research


_______________________________________________
[e-mail] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[e-mail]"

> -------------------------------------------------------------------------- <

發信人: [e-mail] (Dag-Erling Sm鷨grav), 看板: FB_current
標  題: Re: performance of jailed processes
發信站: NCTU CSIE FreeBSD Server (Wed Mar 31 02:02:13 2004)
轉信站: ptt!FreeBSD.csie.NCTU!not-for-mail

Robert Watson <[e-mail]> writes:
> Are your aliases configured on lo0, or on the ethernet interface?

They're on the NIC.  I tried moving one of them to lo0 (made sure
forwarding was enabled) and saw essentially no difference.  Average
ping time went slightly down, but well within standard deviation, so
it may have been a fluke.  Mysql performance was neither worse nor
better.

> Could
> we see some excerpted ifconfig output for your interface (perhaps only ten
> -- first five, last five of the 2000+ IP addresses :-).=20

There are only ~1000:

fxp0: flags=3D8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
        inet 1.2.80.2 netmask 0xffffffe0 broadcast 1.2.80.31
        inet 1.2.84.2 netmask 0xfffffe00 broadcast 1.2.85.255
        inet 1.2.84.3 netmask 0xffffffff broadcast 1.2.84.3
        inet 1.2.84.4 netmask 0xffffffff broadcast 1.2.84.4
        inet 1.2.84.5 netmask 0xffffffff broadcast 1.2.84.5
[...]
        inet 1.2.87.250 netmask 0xffffffff broadcast 1.2.87.250
        inet 1.2.87.251 netmask 0xffffffff broadcast 1.2.87.251
        inet 1.2.87.252 netmask 0xffffffff broadcast 1.2.87.252
        inet 1.2.87.253 netmask 0xffffffff broadcast 1.2.87.253
        inet 1.2.87.254 netmask 0xffffffff broadcast 1.2.87.254
        ether 00:12:34:56:78:9a
        media: Ethernet autoselect (100baseTX <full-duplex>)
        status: active

DES
--=20
Dag-Erling Sm=F8rgrav - [e-mail]
_______________________________________________
[e-mail] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[e-mail]"

> -------------------------------------------------------------------------- <

發信人: [e-mail] ("Marc G. Fournier"), 看板: FB_current
標  題: Re: performance of jailed processes
發信站: NCTU CSIE FreeBSD Server (Wed Mar 31 02:02:13 2004)
轉信站: ptt!FreeBSD.csie.NCTU!not-for-mail


You are running 1000 jail's on one machine??  Now I'm impressed ... what
is the hardware?

On Wed, 31 Mar 2004, Dag-Erling [iso-8859-1] Sm=F8rgrav wrote:

> Robert Watson <[e-mail]> writes:
> > Are your aliases configured on lo0, or on the ethernet interface?
>
> They're on the NIC.  I tried moving one of them to lo0 (made sure
> forwarding was enabled) and saw essentially no difference.  Average
> ping time went slightly down, but well within standard deviation, so
> it may have been a fluke.  Mysql performance was neither worse nor
> better.
>
> > Could
> > we see some excerpted ifconfig output for your interface (perhaps only =
ten
> > -- first five, last five of the 2000+ IP addresses :-).
>
> There are only ~1000:
>
> fxp0: flags=3D8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
>         inet 1.2.80.2 netmask 0xffffffe0 broadcast 1.2.80.31
>         inet 1.2.84.2 netmask 0xfffffe00 broadcast 1.2.85.255
>         inet 1.2.84.3 netmask 0xffffffff broadcast 1.2.84.3
>         inet 1.2.84.4 netmask 0xffffffff broadcast 1.2.84.4
>         inet 1.2.84.5 netmask 0xffffffff broadcast 1.2.84.5
> [...]
>         inet 1.2.87.250 netmask 0xffffffff broadcast 1.2.87.250
>         inet 1.2.87.251 netmask 0xffffffff broadcast 1.2.87.251
>         inet 1.2.87.252 netmask 0xffffffff broadcast 1.2.87.252
>         inet 1.2.87.253 netmask 0xffffffff broadcast 1.2.87.253
>         inet 1.2.87.254 netmask 0xffffffff broadcast 1.2.87.254
>         ether 00:12:34:56:78:9a
>         media: Ethernet autoselect (100baseTX <full-duplex>)
>         status: active
>
> DES
> --
> Dag-Erling Sm=F8rgrav - [e-mail]
> _______________________________________________
> [e-mail] mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-current
> To unsubscribe, send any mail to "[e-mail]=
"
>

----
Marc G. Fournier           Hub.Org Networking Services (http://www.hub.org)
Email: [e-mail]           Yahoo!: yscrappy              ICQ: 7615664
_______________________________________________
[e-mail] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[e-mail]"

> -------------------------------------------------------------------------- <

發信人: [e-mail] (Dag-Erling Sm鷨grav), 看板: FB_current
標  題: Re: performance of jailed processes
發信站: NCTU CSIE FreeBSD Server (Wed Mar 31 02:35:10 2004)
轉信站: ptt!FreeBSD.csie.NCTU!not-for-mail

"Marc G. Fournier" <[e-mail]> writes:
> You are running 1000 jail's on one machine??  Now I'm impressed ... what
> is the hardware?

"reading the rest of the thread" is apparently a lost art...

DES
--=20
Dag-Erling Sm=F8rgrav - [e-mail]
_______________________________________________
[e-mail] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[e-mail]"

> -------------------------------------------------------------------------- <

發信人: [e-mail] ("Marc G. Fournier"), 看板: FB_current
標  題: Re: performance of jailed processes
發信站: NCTU CSIE FreeBSD Server (Wed Mar 31 04:14:03 2004)
轉信站: ptt!FreeBSD.csie.NCTU!not-for-mail

On Wed, 31 Mar 2004, Dag-Erling [iso-8859-1] Sm=F8rgrav wrote:

> "Marc G. Fournier" <[e-mail]> writes:
> > You are running 1000 jail's on one machine??  Now I'm impressed ... wha=
t
> > is the hardware?
>
> "reading the rest of the thread" is apparently a lost art...

sorry, jump'd in half way :(  it wasn't until the 1000 IPs that it peaked
my interest ...

----
Marc G. Fournier           Hub.Org Networking Services (http://www.hub.org)
Email: [e-mail]           Yahoo!: yscrappy              ICQ: 7615664
_______________________________________________
[e-mail] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[e-mail]"