I have some questions about what needs tuned on a high traffic syslog box.
I seem to be dropping quite a few syslog packets.
This is a syslog server for a high usage Firewall btw.
Nic is a Compaq tl0
4.8-P13
netstat -s -p udp | grep buf
19,762,079 dropped due to full socket buffers
uptime
5:28PM up 7 days, 18:30, 2 users, load averages: 0.21, 0.23, 0.23
I though maybe syslogd was the problem, but running nc on the syslog port and
sending output to /dev/null still shows the buffer problem.
i've tried uping net.inet.udp.recvspace
if this gets too high i will no longer be able to send udp packets
and will get a socket buff full err.
net.local.dgram.recvspace This didn't do much.
i tried moving kern.ipc.maxsockbuf in by doubling each time
This didn't help
kern.ipc.maxsockbuf: 1048576 <- This is what it currently is set to.
if someone could point me in the right direction that would be great :).
here is some info on the box in question.
btw all these command were run while the system
was doing about 1500 pps (as per netstat -inb 1)
kern.maxfilesperproc: 8272
kern.openfiles: 86
btw syslogd runs at %20 cpu from top
systat -vm 1
shows disk mostly idle (1-5% usage).
this box has 6 9 gig drives in raid5 also.
Which i think show up as one drive.
/dev/idad0s2a on / (ufs, local)
/dev/idad0s2f on /tmp (ufs, local)
/dev/idad0s2e on /usr (ufs, local, soft-updates)
/dev/idad0s2g on /var (ufs, local, soft-updates)
ps -axwwj | grep syslogd
root 84 1 84 c500e740 0 Rs ?? 1601:25.44 /usr/sbin/syslogd -n
ps -axwwu | grep syslogd
root 84 18.6 0.1 972 620 ?? Rs 26Oct03 1601:30.54 /usr/sbin/syslogd -n
ifconfig tl0
tl0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
ether 00:08:c7:9f:78:1e
media: Ethernet 100baseTX <full-duplex>
status: active
netstat -inb 1
This can peak at around 2100 pps.
low is about 600 pps.
packets errs bytes packets errs bytes colls
1568 0 226804 6 0 0 0
1274 0 200785 1 0 178 0
netstat -in
Name Mtu Network Address Ipkts Ierrs Opkts Oerrs Coll
tl0 1500 <Link#1> 00:08:c7:9f:78:1e 713151669 0 83482 0 0
netstat -s -p udp
udp:
711282523 datagrams received
0 with incomplete header
0 with bad data length field
0 with bad checksum
1 with no checksum
306 dropped due to no socket
0 broadcast/multicast datagrams dropped due to no socket
19783694 dropped due to full socket buffers
0 not for hashed pcb
691498523 delivered
20954 datagrams output
netstat -m
66/336/81408 mbufs in use (current/peak/max):
66 mbufs allocated to data
64/220/20352 mbuf clusters in use (current/peak/max)
524 Kbytes allocated to network (0% of mb_map in use)
0 requests for memory denied
0 requests for memory delayed
0 calls to protocol drain routines
I was using ipf, but now its disabled (no rules, and ipf -D).
top line of.. top
CPU states: 9.9% user, 0.0% nice, 9.3% system, 3.3% interrupt, 77.5% idle
Mem: 12M Active, 461M Inact, 64M Wired, 25M Cache, 67M Buf, 1076K Free
Swap: 768M Total, 112K Used, 768M Free
dmesg.boot
btw its a dual 400
Copyright (c) 1992-2003 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
The Regents of the University of California. All rights reserved.
FreeBSD 4.8-RELEASE-p13 #2: Sun Oct 26 22:47:48 CST 2003
root@ME.MYDOMAIN.com:/usr/obj/usr/src/sys/SYSLOG
Timecounter "i8254" frequency 1193182 Hz
Timecounter "TSC" frequency 399072197 Hz
CPU: Pentium II/Pentium II Xeon/Celeron (399.07-MHz 686-class CPU)
Origin = "GenuineIntel" Id = 0x652 Stepping = 2
Features=0x183fbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CM
OV,PAT,PSE36,MMX,FXSR>
real memory = 603979776 (589824K bytes)
avail memory = 583192576 (569524K bytes)
Preloaded elf kernel "kernel" at 0xc0368000.
Pentium Pro MTRR support enabled
md0: Malloc disk
npx0: <math processor> on motherboard
npx0: INT 16 interface
pcib0: <Intel 82443BX host to PCI bridge (AGP disabled)> on motherboard
pci0: <PCI bus> on pcib0
pci0: <Cirrus Logic GD5446 SVGA controller> at 11.0
pcib1: <DEC 21150 PCI-PCI bridge> at device 13.0 on pci0
pci1: <PCI bus> on pcib1
tl0: <Compaq Netelligent 10/100 Proliant> port 0x2c00-0x2c0f mem 0xc6efcdf0-0xc6
efcdff irq 5 at device 7.0 on pci1
tl0: Ethernet address: 00:08:c7:9f:78:1e
miibus0: <MII bus> on tl0
nsphy0: <DP83840 10/100 media interface> on miibus0
nsphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
tlphy0: <ThunderLAN 10baseT media interface> on miibus0
tlphy0: 10base2/BNC, 10base5/AUI
sym0: <875> port 0x2000-0x20ff mem 0xc6eff000-0xc6efffff,0xc6efcf00-0xc6efcfff i
rq 9 at device 9.0 on pci1
sym0: No NVRAM, ID 7, Fast-20, SE, parity checking
sym1: <875> port 0x2400-0x24ff mem 0xc6efe000-0xc6efefff,0xc6efce00-0xc6efceff i
rq 10 at device 9.1 on pci1
sym1: No NVRAM, ID 7, Fast-20, SE, parity checking
pci1: <unknown card> (vendor=0x10b8, dev=0x0005) at 10.0 irq 15
pci0: <unknown card> (vendor=0x0e11, dev=0xa0f0) at 14.0
pcib2: <IBM 82351 PCI-PCI bridge> at device 15.0 on pci0
pci2: <PCI bus> on pcib2
ida0: <Compaq SMART-2/P array controller> port 0x3000-0x30ff mem 0xb8000000-0xbf
ffffff,0xc6ffff00-0xc6ffffff irq 11 at device 0.0 on pci2
ida0: drives=1 firm_rev=3.08
idad0: <Compaq Logical Drive> on ida0
idad0: 34707MB (71081760 sectors), blocksize=512
isab0: <Intel 82371AB PCI to ISA bridge> at device 20.0 on pci0
isa0: <ISA bus> on isab0
atapci0: <Intel PIIX4 ATA33 controller> port 0xf100-0xf10f at device 20.1 on pci
0
ata0: at 0x1f0 irq 14 on atapci0
ata1: at 0x170 irq 15 on atapci0
pci0: <Intel 82371AB/EB (PIIX4) USB controller> at 20.2 irq 0
chip1: <Intel 82371AB Power management controller> at device 20.3 on pci0
orm0: <Option ROMs> at iomem 0xc0000-0xc7fff,0xc8000-0xcbfff,0xe8000-0xedfff,0xe
e000-0xeffff on isa0
fdc0: <NEC 72065B or clone> at port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on isa0
fdc0: FIFO enabled, 8 bytes threshold
fd0: <1440-KB 3.5" drive> on fdc0 drive 0
atkbdc0: <Keyboard controller (i8042)> at port 0x60,0x64 on isa0
atkbd0: <AT Keyboard> flags 0x1 irq 1 on atkbdc0
kbd0 at atkbd0
psm0: <PS/2 Mouse> irq 12 on atkbdc0
psm0: model IntelliMouse Explorer, device ID 4
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
sio0 at port 0x3f8-0x3ff irq 4 flags 0x10 on isa0
sio0: type 16550A
sio1 at port 0x2f8-0x2ff irq 3 on isa0
sio1: type 16550A
ppc0: parallel port not found.
IP Filter: v3.4.31 initialized. Default = pass all, Logging = enabled
acd0: CDROM <CD-ROM CDU701-Q> at ata0-master PIO4
Waiting 15 seconds for SCSI devices to settle
Mounting root from ufs:/dev/idad0s2a
KERN CONFIG file
machine i386
cpu I686_CPU
ident SYSLOG
options INET #InterNETworking
options INET6 #IPv6 communications protocols
options FFS #Berkeley Fast Filesystem
options FFS_ROOT #FFS usable as root device [keep this!]
options SOFTUPDATES #Enable FFS soft updates support
options MFS #Memory Filesystem
options MD_ROOT #MD is a potential root device
options NFS #Network Filesystem
options NFS_ROOT #NFS usable as root device, NFS required
options MSDOSFS #MSDOS Filesystem
options CD9660 #ISO 9660 Filesystem
options CD9660_ROOT #CD-ROM usable as root, CD9660 required
options PROCFS #Process filesystem
options COMPAT_43 #Compatible with BSD 4.3 [KEEP THIS!]
options SCSI_DELAY=15000 #Delay (in ms) before probing SCSI
options UCONSOLE #Allow users to grab the console
options USERCONFIG #boot -c editor
options VISUAL_USERCONFIG #visual boot -c editor
options KTRACE #ktrace(1) support
options SYSVSHM #SYSV-style shared memory
options SYSVMSG #SYSV-style message queues
options SYSVSEM #SYSV-style semaphores
options P1003_1B #Posix P1003_1B real-time extensions
options _KPOSIX_PRIORITY_SCHEDULING
options ICMP_BANDLIM #Rate limit bad replies
options KBD_INSTALL_CDEV # install a CDEV entry in /dev
options IPFILTER #ipfilter support
options IPFILTER_LOG #ipfilter logging
options SC_NORM_ATTR="(FG_GREEN|BG_BLACK)"
options SC_NORM_REV_ATTR="(FG_YELLOW|BG_GREEN)"
options SC_KERNEL_CONS_ATTR="(FG_RED|BG_BLACK)"
options SC_KERNEL_CONS_REV_ATTR="(FG_BLACK|BG_RED)"
options UFS_DIRHASH
options INCLUDE_CONFIG_FILE
options NMBUFS=81408
options NMBCLUSTERS=20352
device isa
device pci
device fdc0 at isa? port IO_FD1 irq 6 drq 2
device fd0 at fdc0 drive 0
device fd1 at fdc0 drive 1
device ata0 at isa? port IO_WD1 irq 14
device ata1 at isa? port IO_WD2 irq 15
device ata
device atadisk # ATA disk drives
device atapicd # ATAPI CDROM drives
device atapifd # ATAPI floppy drives
device atapist # ATAPI tape drives
options ATA_STATIC_ID #Static device numbering
device sym # NCR/Symbios Logic (newer chipsets)
device scbus # SCSI bus (required)
device da # Direct Access (disks)
device pass # Passthrough device (direct SCSI access)
device ida # Compaq Smart RAID
device atkbdc0 at isa? port IO_KBD
device atkbd0 at atkbdc? irq 1 flags 0x1
device psm0 at atkbdc? irq 12
device vga0 at isa?
pseudo-device splash
device sc0 at isa? flags 0x100
device npx0 at nexus? port IO_NPX irq 13
device sio0 at isa? port IO_COM1 flags 0x10 irq 4
device sio1 at isa? port IO_COM2 irq 3
device sio2 at isa? disable port IO_COM3 irq 5
device sio3 at isa? disable port IO_COM4 irq 9
device ppc0 at isa? irq 7
device ppbus # Parallel port bus (required)
device lpt # Printer
device plip # TCP/IP over parallel
device ppi # Parallel port interface device
device miibus # MII bus support
device fxp # Intel EtherExpress PRO/100B (82557, 82558)
device tl # Texas Instruments ThunderLAN
pseudo-device loop # Network loopback
pseudo-device ether # Ethernet support
pseudo-device pty # Pseudo-ttys (telnet etc)
pseudo-device md # Memory "disks"
pseudo-device bpf #Berkeley packet filter
same random stuff from /etc/sysctl.conf
net.inet.udp.recvspace=84160
net.inet.tcp.blackhole=1
net.inet.udp.blackhole=1
net.inet.icmp.log_redirect=1
net.inet.tcp.log_in_vain=1
Is this too much info btw?
I just wanted to make sure i didn't get a, not enough info
reply, sorry if this was too much.
_______________________________________________
freebsd-stable@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org"
> -------------------------------------------------------------------------- <
發信人: ilya@samara.net (Ilya Varlashkin), 看板: FB_stable
標 題: Re: (long) high traffic syslog server.
發信站: NCTU CSIE FreeBSD Server (Thu Nov 6 08:49:13 2003)
轉信站: ptt!FreeBSD.csie.NCTU!not-for-mail
On Mon, Nov 03, 2003 at 06:05:33PM -0600, John wrote:
> I though maybe syslogd was the problem, but running nc on the syslog port and
> sending output to /dev/null still shows the buffer problem.
This looks like the system isn't processing interrupts from network card
fast enough. Try running 'systat -vm' and see if '%Intr' is high.
I'm not familiar with particular network card you're using, but try to check
if with some Intel 100Mbps card you get better performance.
If your syslog traffic load is always high, it doesn't matter how large buffers
you make - at some point they will be filled up if system can't cope with
such amount of traffic.
Kind regards,
Ilya Varlashkin
_______________________________________________
freebsd-stable@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org"
> -------------------------------------------------------------------------- <
發信人: rb@gid.co.uk (Bob Bishop), 看板: FB_stable
標 題: Re: (long) high traffic syslog server.
發信站: NCTU CSIE FreeBSD Server (Thu Nov 6 08:49:13 2003)
轉信站: ptt!FreeBSD.csie.NCTU!not-for-mail
Hi,
At 00:05 4/11/03, John wrote:
>[...]
>btw its a dual 400
>[etc]
Just for clarification: it seems from the data you posted that you aren't
running an SMP kernel, right? If CPU is your problem, that might make a
difference :-)
--
Bob Bishop +44 (0)118 977 4017
rb@gid.co.uk fax +44 (0)118 989 4254
_______________________________________________
freebsd-stable@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org"
> -------------------------------------------------------------------------- <
發信人: strgout@unixjunkie.com (John), 看板: FB_stable
標 題: Re: (long) high traffic syslog server.
發信站: NCTU CSIE FreeBSD Server (Thu Nov 6 08:49:13 2003)
轉信站: ptt!FreeBSD.csie.NCTU!not-for-mail
On Tue, Nov 04, 2003 at 08:56:57AM +0000, Bob Bishop wrote:
> Hi,
>
> At 00:05 4/11/03, John wrote:
> >[...]
> >btw its a dual 400
> >[etc]
>
> Just for clarification: it seems from the data you posted that you aren't
> running an SMP kernel, right? If CPU is your problem, that might make a
> difference :-)
>
>
> --
> Bob Bishop +44 (0)118 977 4017
> rb@gid.co.uk fax +44 (0)118 989 4254
sorry, i disabled SMP kernel to see if it was a SMP kernel issue.
Which it doesn't seem like it was as it acts the same UNI or SMP kernel.
Ilya Varlashkin <ilya@samara.net>
22.0%Sys 8.3%Intr 13.6%User 0.0%Nice 56.1%Idl <-
That is at a high point.
i'll also see if i can dig up a fxp nic and link0 it.
on the doesn't matter how large buffers comment.
Part of the question is Which buffers should i be looking at?
_______________________________________________
freebsd-stable@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org"