February 2009 - syslog-ng - lists.balabit.hu

Configuration question, logging to db not working.
by Liam Kirsher 01 Mar '09

01 Mar '09

Hi -- I am /almost/ there, logging to Postgres database. However, I've discovered a puzzling and problematic behavior.This is probably just some simple misunderstanding on my part, since this is my first foray into syslog-ng. I am logging to two different db tables. Which table I log to is determined by a regexp filter. The value is either root.ut_access or root.geocode. I can get either one to work, but not both at the same time. If I comment out the log entry for the geocode, then ut_access works. However, if both log entries exist, only the gecocode_access_log table gets a new row. Nothing is logged to the ut_access_log table! (Both messages are logged to d_obsidian destination file, however.) I've attached my config file. Any tips would be greatly appreciated!!! Liam ---------------------------------- @version:3.0 # syslog-ng configuration file. options { flush_lines (0); time_reopen (10); log_fifo_size (1000); long_hostnames (off); use_dns (no); use_fqdn (no); create_dirs (yes); keep_hostname (yes); }; source s_sys { file ("/proc/kmsg" program_override("kernel: ")); unix-stream ("/dev/log"); internal(); udp(ip(0.0.0.0) port(514)); tcp(ip(0.0.0.0) port(5000) max-connections(1000)); }; destination d_cons { file("/dev/console"); }; destination d_mesg { file("/var/log/messages"); }; destination d_auth { file("/var/log/secure"); }; destination d_mail { file("/var/log/maillog" flush_lines(10)); }; destination d_spol { file("/var/log/spooler"); }; destination d_boot { file("/var/log/boot.log"); }; destination d_cron { file("/var/log/cron"); }; destination d_mlal { usertty("*"); }; destination d_obsidian { file("/var/log/django/$PROGRAM/$R_YEAR$R_MONTH$R_DAY.log"); }; destination d_ut_access { sql( type(pgsql) host("localhost") username("postgres") password("xxxxxx") database("prodlogs") table("ut_access_log") columns("datetime", "query_time", "host", "program", "pid", "request_id", "level", "ip", "phone_id", "phone_type", "software_version", "client_version", "query_string", "art_id", "session_id", "lat", "lng") values("$R_ISODATE", "${UT.QTIME}", "$HOST", "$PROGRAM", "$PID", "${UT.REQUEST_ID}", "$LEVEL", "${UT.IP}", "${UT.PHONE_ID}", "${UT.PHONE_TYPE}", "${UT.SOFTWARE_VERSION}", "${UT.CLIENT_VERSION}", "${UT.QUERY_STRING}", "${UT.ART_ID}", "${UT.SESSION_ID}", "${UT.LAT}", "${UT.LNG}") indexes("datetime", "host", "program", "ip", "phone_id") ); }; destination d_geocode { sql( type(pgsql) host("localhost") username("postgres") password("xxxxxx") database("prodlogs") table("geocode_access_log") columns("datetime", "querytime", "host", "program", "pid", "request_id", "level", "ip", "name", "place", "lat", "lng") values("$R_ISODATE", "${GEO.QTIME}", "$HOST", "$PROGRAM", "$PID", "${GEO.REQUEST_ID}", "$LEVEL", "${GEO.IP}", "${GEO.NAME}", "${GEO.PLACE}", "${GEO.LAT}", "${GEO.LNG}") indexes("datetime", "host", "program", "pid", "ip", "name", "place") ); }; parser p_ut_access { csv-parser( columns("UT.QTIME", "UT.IP", "UT.REQUEST_ID", "UT.CATEGORY", "UT.MYLEVEL", "UT.PHONE_ID", "UT.PHONE_TYPE", "UT.SOFTWARE_VERSION", "UT.CLIENT_VERSION", "UT.QUERY_STRING", "UT.ART_ID", "UT.SESSION_ID", "UT.LAT", "UT.LNG") delimiters(",") quote-pairs('""') flags(escape-double-char, strip-whitespace) ); }; parser p_geocode { csv-parser( columns("GEO.QTIME", "GEO.IP", "GEO.REQUEST_ID", "GEO.CATEGORY", "GEO.MYLEVEL", "GEO.NAME", "GEO.PLACE", "GEO.LAT", "GEO.LNG") delimiters(",") quote-pairs('""') flags(escape-double-char, strip-whitespace) ); }; #filter f_filter1 { facility(kern); }; filter f_filter2 { level(info..emerg) and not facility(mail,authpriv,cron); }; filter f_filter3 { facility(authpriv); }; filter f_filter4 { facility(mail); }; filter f_filter5 { level(emerg); }; filter f_filter6 { facility(uucp) or (facility(news) and level(crit..emerg)); }; filter f_filter7 { facility(local7); }; filter f_filter8 { facility(cron); }; filter f_obsidian { program("^obsidian$") and level(info); }; filter f_ut_access { filter(f_obsidian) and message("root\.ut_access"); }; filter f_geocode { filter(f_obsidian) and message("root\.geocode"); }; #log { source(s_sys); filter(f_filter1); destination(d_cons); }; log { source(s_sys); filter(f_filter2); destination(d_mesg); }; log { source(s_sys); filter(f_filter3); destination(d_auth); }; log { source(s_sys); filter(f_filter4); destination(d_mail); }; log { source(s_sys); filter(f_filter5); destination(d_mlal); }; log { source(s_sys); filter(f_filter6); destination(d_spol); }; log { source(s_sys); filter(f_filter7); destination(d_boot); }; log { source(s_sys); filter(f_filter8); destination(d_cron); }; log { source(s_sys); filter(f_ut_access); parser(p_ut_access); destination(d_ut_access); }; ### With this log entry commented out, logging to d_ut_access works. But if I uncomment it, nothing is logged to d_ut_access! ### What am I missing? #log { #source(s_sys); #filter(f_geocode); #parser(p_geocode); #destination(d_geocode); #}; log { source(s_sys); filter(f_obsidian); destination(d_obsidian); }; -- Liam Kirsher PGP: http://liam.numenet.com/pgp/

2 7

creating new facility
by gatfi sami 28 Feb '09

28 Feb '09

hi i am a student i installed 1.6.12-76.2 i am asked to create a new costumised facility ( newfacility ) in bothe syslog-ng cient and server for exemple so Errorlog syslog:newfacility ( in /etc/apache2/mod_log.conf.config) is it possible another thing please if not i use local 1 (Errorlog syslog:local1) if i send mysql error log also in local1 can i seperate them in different destination ( file )on my syslogserver. thkx

1 0

[Bug 35] New: syslog-ng crashes with a certain .conf
by bugzilla＠bugzilla.balabit.com 28 Feb '09

28 Feb '09

https://bugzilla.balabit.com/show_bug.cgi?id=35 Summary: syslog-ng crashes with a certain .conf Product: syslog-ng Version: 3.0.x Platform: PC OS/Version: Linux Status: NEW Severity: normal Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: tek(a)serverop.de Type of the Report: bug Estimated Hours: 0.0 Created an attachment (id=13) --> (https://bugzilla.balabit.com/attachment.cgi?id=13) example syslog-ng.conf that produces the segfault If syslog-ng is started ./syslog-ng -f /etc/syslog-ng/syslog-ng.conf --pidfile=/tmp/foo.pid then it will segfault: Feb 28 00:13:50 basra kernel: syslog-ng[2974]: segfault at bf418ffc ip b7ca9cc3 sp bf419000 error 6 in libc-2.8.so[b7c01000+130000] If the 31th line is uncommented, syslog-ng starts just fine. # ./syslog-ng -V syslog-ng 3.0.1 Revision: ssh+git://bazsi@git.balabit//var/scm/git/syslog-ng/syslog-ng-ose--mainline--3.0#master#93a342dae3a2b0cb15811d0c34ea7f58b3fba14e Compile-Date: Feb 27 2009 23:56:41 Enable-Threads: off Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-Sun-STREAMS: off Enable-Sun-Door: off Enable-IPv6: on Enable-Spoof-Source: on Enable-TCP-Wrapper: on Enable-SSL: on Enable-SQL: off Enable-Linux-Caps: on Enable-Pcre: on -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 1

Re: [syslog-ng] File descriptor leaks with 3.0.1
by chris packham 28 Feb '09

28 Feb '09

Ping. Any interest in this? Should I submit something into bugzilla? Write a test case? Send the patch as a separate email? On Tue, 2009-02-17 at 09:58 +1300, chris packham wrote: > On Mon, 2009-02-16 at 15:52 +1300, chris packham wrote: > > Also seems to be present in the latest version checked out from the git repo but I'll do some more digging. > > > > I've confirmed that this does indeed happen on the latest version from > the git repo. I've attached a patch that addresses this. Looks like the > 'include' implementation assumes that yy_switch_to_buffer closes the old > file (which flex 2.5.33 doesn't, I haven't tried other versions). > > ______________________________________________________________________________ > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng > Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng > FAQ: http://www.campin.net/syslog-ng/faq.html >

2 1

[patch] only stat dns_cache_hosts file once in a second
by ILLES, Marton 28 Feb '09

28 Feb '09

Hi, I was doing today some performance tests where use_dns(yes) and persistent dns cache (dns_cache_hosts) were configured and I got awful results. When I disabled dns_cache_hosts the message rate doubled. After a quick check I realized that for every message a dns cache lookup was performed which stat()-ed my dns_cache_hosts file. However it does not make much sense to check the file for modification so often, I guess once a second is enough most of the cases. In my test configuration I received msgs over syslog() source and sent to an empty destination. The server was a 2way quad-core 1.9GHz Opteron box running 32bit syslog-ng and messsages were generated with loggen. >From an average 80,000-90,000 msg/s using the fix I could do 220,000-230,000 msg/s. Interesting that on my laptop (dual-core 2.4GHz Intel 64bit) the performance gain was smaller from 90,000 to 100,000 msg/s. I am still wondering about this big difference, specially because my laptop has higher frequency CPU... Anyhow the patch could be found in my git repo, Bazsi please apply. git://git.balabit.hu/marci/syslog-ng-3.0.git M -- Key fingerprint = F78C 25CA 5F88 6FAF EA21 779D 3279 9F9E 1155 670D

2 1

Re: [syslog-ng] syslog-ng Digest, Vol 46, Issue 31
by 牛田　善信 28 Feb '09

28 Feb '09

Dear Bazsi-san, Thank you for the reply. >what kind of binaries are you using? PE/OSE binaries supplied by >BalaBit, or you are using binaries from another source? I am using what downloaded from rpmsearch. It is following URL. http://rpm.pbone.net/index.php3/stat/4/idpl/10055325/com/syslog-ng-2.1.1-5.… And, I used syslog-ng-2.1.3-1 supplied by balabit. I have done the question with indefinite purpose of me. My final purpose is to get rid of CPU use rate 100%. Could you direct the best possible method that I should do? Best Regards, Yoshinobu

2 1

Re: [syslog-ng] File descriptor leaks with 3.0.1
by chris packham 28 Feb '09

28 Feb '09

On Mon, 2009-02-16 at 15:52 +1300, chris packham wrote: > Also seems to be present in the latest version checked out from the git repo but I'll do some more digging. > I've confirmed that this does indeed happen on the latest version from the git repo. I've attached a patch that addresses this. Looks like the 'include' implementation assumes that yy_switch_to_buffer closes the old file (which flex 2.5.33 doesn't, I haven't tried other versions).

2 1

high memory usage (swapping)
by Otto Berger 27 Feb '09

27 Feb '09

Hi, today i got a weird cenario: i rebooted our syslog-ng server and forgot to start the syslog-ng daemon. So far so (not) good. One of our webservers which sends his logs via UDP to the syslog-server became suddenly very slow. It is reproducable: whenever i stop the central syslog-server the sender-servers syslog-ng-process gets a very high memory usage (4GB!) within seconds... The log-traffic on this server is about 30MB/day... UDP is a connection-less protocol isnt? ;) The configuration of the sender-server: ----- options { sync (0); time_reopen (10); log_fifo_size (1000); long_hostnames (off); use_dns (no); use_fqdn (no); create_dirs (no); keep_hostname (yes); }; source s_sys { file ("/proc/kmsg" log_prefix("kernel: ")); unix-stream ("/dev/log"); internal(); # udp(ip(0.0.0.0) port(514)); }; destination a_udp { udp( "example.com" port(514) ); }; #log { source(s_sys); destination (a_udp); }; ----- sender: syslog-ng 2.1.3 reciever: syslog-ng 2.1.2 someone got an idea on this issue? Otto

2 2

Load balancing using $SEC
by Vincent Panel 27 Feb '09

27 Feb '09

Hi, I'm trying to do some kind of load-balancing using syslog-ng. I'm willing to write to 2 different files named "syslog1" and "syslog2". The first one would get all messages received by syslog whose timestamp have a second ending with 0, 1, 2, 3 or 4. Syslog 2 would get all messages whose second ends with 5, 6, 7, 8 or 9. For instance, if I get those messages : Feb 27 00:00:03 blabla1 Feb 27 00:00:04 blabla2 Feb 27 00:00:05 blabla3 Feb 27 00:00:06 blabla4 The first two messages would end up in syslog1 and the last 2 messages would end up in syslog2 However, I was not able to do this because : * filtering is not possible on any date-related macro (I don't know why, but only MSG and MSGHDR macros seem populated at that point in time) * I can't apply a rewrite rule on $SEC * I can't apply any function to the $SEC macro in the destination statement, or I don't know how to do it Do you think it's still possible to achieve what I want with syslog-ng ? Vincent

1 0

creating new custumised facility
by gatfi sami 26 Feb '09

26 Feb '09

Hi i am getting started with syslog can i add a new facility different from ( mail ... local0 to 7) what i did til now is sending every thing (src) to a syslog server via udp #########" client syslog > filter myfil1 { facility(local1); };# let only local1 = error log apache > > destination loghost { > udp("192.168.1.66" port(514)); > }; > log { > source(src); > > filter(myfil1); > destination(loghost); > }; > ###Server syslog source src { > # > # include internal syslog-ng messages > # note: the internal() soure is required! > # > # internal(); > > # > # the default log socket for local logging: > # > # unix-dgram("/dev/log"); > > # > # uncomment to process log messages from network: > # > #udp(ip("0.0.0.0") port(514)); > udp(port(514)); > > > }; > source src { > # > # include internal syslog-ng messages > # note: the internal() soure is required! > # > # internal(); > > # > # the default log socket for local logging: > # > # unix-dgram("/dev/log"); > > # > # uncomment to process log messages from network: > # > #udp(ip("0.0.0.0") port(514)); > udp(port(514)); > > > }; > and sent Apache erreor to syslog :local1 /mod_log_config.conf Errorlog syslog:local1 ************** what i tried and didnot work is adding /var/log/apache2/errorlog as sorce because of permession denial and using unix-stream unix-dgram pipe ( i created file with) mkififo between apache and syslog nothing so sorry if i am bothering but my questions are 1 can i add a new facility like local1 and collect on it apapche error log if no is Errorlog syslog: local1 the *best* way with a filter on local1 2 does local1in my case contain only appache error log 3 if there is another way be my gest i am hering Sorry am not good in eglish

1 0