- syslog-ng - lists.balabit.hu

syslog-ng OSE 3.3 Administrator Guide released
by Fekete Róbert 05 Aug '11

05 Aug '11

Hi, Even though OSE 3.3 is still in a beta state, the documentation has been finally updated with the new features of syslog-ng OSE 3.3, so it contains a basic description for the mongo database support and for running syslog-ng in multihreaded mode, and a bunch of error corrections and clarifications. For a list of changes, see http://www.balabit.com/sites/default/files/documents/syslog-ng-ose-3.3-guid… The guide is available at the BalaBit Documentation Page at http://www.balabit.com/support/documentation?&product_line=46&product_type=… , in PDF, HTML, and single-page HTML format. As always, comments, criticism, error reports, and feedback is welcome here on the list or at the documentation(a)balabit.com e-mail address. Kind Regards, Robert Fekete

1 0

Parsing Question
by Brandon Phelps 05 Aug '11

05 Aug '11

Hello All, Could anyone explain how I would parse a message that looks like this: Jul 29 08:58:38 192.168.1.1 id=firewall sn=0017C5158708 time="2011-07-29 08:58:38" fw=100.1.1.1 pri=6 c=262144 m=98 msg="Connection Opened" n=0 src=192.168.2.100:123:X0 dst=74.1.2.3:X1 proto=udp/ntp I am logging to mysql and would like to extract the 'src' and 'dst' fields from the above message so that I can insert them into indexed fields in my database. Currently my destination looks like this: destination d_mysql { sql( type(mysql) host("localhost") username("myusername") password("mypassword") database("syslog") table("logs") columns("host", "facility", "priority", "level", "tag", "datetime", "program", "msg") values("$HOST_FROM", "$FACILITY", "$PRIORITY", "$LEVEL", "$TAG", "$YEAR-$MONTH-$DAY $HOUR:$MIN:$SEC", "$PROGRAM", "$MSG") indexes("host", "facility", "priority", "datetime", "program") ); }; This table (logs) also has source_ip and destination_ip fields which are currently unused since I don't know how to extract that from the message. For the above example, I would want those fields to contact '192.168.2.100' and '74.1.2.3' respectively. Is my only option in this case to write a perl script or something that watches a named pipe and have syslog-ng log to the named pipe instead, while my perl script does the actual parsing? Or can I do what I want with syslog-ng alone? Any help would be greatly appreciated. Thanks, Brandon

4 5

MongoDB and syslog-ng 3.3 on Centos 5.5
by Viacheslav Biriukov 03 Aug '11

03 Aug '11

Hello. Can you help me with syslog-ng 3.3 on the CentoOS 5.5 with MongoDB 1.6 I install syslog-ng version 3.3 where mongodb support exists. But when I add to the config mongodb destination I get an error: Starting syslog-ng: Error parsing destination, destination plugin mongodb > not found in /etc/syslog-ng/syslog-ng.conf at line 70, column 5: > mongodb( > ^^^^^^^ syslog-ng.conf @version: 3.3 > # > # configuration file for syslog-ng, customized for remote logging > # > source s_internal { internal(); }; > destination d_syslognglog { file("/var/log/syslog-ng.log"); }; > log { source(s_internal); destination(d_syslognglog); }; > # Local sources, filters and destinations are commented out > # If you want to replace sysklogd simply uncomment the following > # parts and disable sysklogd > # > # Local sources > # > source s_local { > unix-dgram("/dev/log"); > file("/proc/kmsg" program_override("kernel")); > }; > # > # Local filters > # > filter f_messages { level(info..emerg); }; > filter f_secure { facility(authpriv); }; > filter f_mail { facility(mail); }; > filter f_cron { facility(cron); }; > filter f_emerg { level(emerg); }; > filter f_spooler { level(crit..emerg) and facility(uucp, news); }; > filter f_local7 { facility(local7); }; > # > # Local destinations > # > destination d_messages { file("/var/log/messages"); }; > destination d_secure { file("/var/log/secure"); }; > destination d_maillog { file("/var/log/maillog"); }; > destination d_cron { file("/var/log/cron"); }; > destination d_console { usertty("root"); }; > destination d_spooler { file("/var/log/spooler"); }; > destination d_bootlog { file("/var/log/boot.log"); }; > # > # Local logs - order DOES matter ! > # > log { source(s_local); filter(f_emerg); destination(d_console); }; > log { source(s_local); filter(f_secure); destination(d_secure); > flags(final); }; > log { source(s_local); filter(f_mail); destination(d_maillog); > flags(final); }; > log { source(s_local); filter(f_cron); destination(d_cron); flags(final); > }; > log { source(s_local); filter(f_spooler); destination(d_spooler); }; > log { source(s_local); filter(f_local7); destination(d_bootlog); }; > log { source(s_local); filter(f_messages); destination(d_messages); }; > > # Remote logging > source s_remote { > tcp(ip(0.0.0.0) port(514)); > udp(ip(0.0.0.0) port(514)); > }; > destination d_separatedbyhosts { > file("/var/log/syslog-ng/$HOST/messages" owner("root") group("root") > perm(0640) dir_perm(0750) create_dirs(yes)); > }; > log { source(s_remote); destination(d_separatedbyhosts); }; > > > #MongoDB > destination d_mongodb { > mongodb( > host("127.0.0.1") > port(27017) > database("syslog-ng") > collection("logs") > keys("date", "facility", "level", "host", "program", "pid", > "message") > values("${R_YEAR}-${R_MONTH}-${R_DAY} > ${R_HOUR}:${R_MIN}:${R_SEC}", "$FACILITY", "$LEVEL", "$HOST","$PROGRAM", > "$PID", "$MSGONLY") > ); > }; > log { source(s_local); destination(d_mongodb); }; > In the modules.conf I try to add @afmongodb but this isn't change the situation. What's wrong? -- Viacheslav Biriukov BR http://biriukov.com

2 1

[Bug 128] New: template() broken in latest 3.3 git
by bugzilla＠bugzilla.balabit.com 01 Aug '11

01 Aug '11

https://bugzilla.balabit.com/show_bug.cgi?id=128 Summary: template() broken in latest 3.3 git Product: syslog-ng Version: unspecified Platform: PC OS/Version: Linux Status: NEW Severity: normal Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: algernon(a)balabit.hu Type of the Report: --- Estimated Hours: 0.0 Hi! While updating my 3.3 installation from an ancient version to latest git head, I noticed that template functions stopped working. If I have any kind of template in the config, I get a segmentation fault: ,---- | Program received signal SIGSEGV, Segmentation fault. | log_template_reset_compiled (self=0x0) at templates.c:661 | 661 while (self->compiled_template) | (gdb) bt | #0 log_template_reset_compiled (self=0x0) at templates.c:661 | #1 0x00007ffff7b9f302 in log_template_compile (self=0x0, template=0x63d7e0 "foo: bar", error=0x7fffffffb5d8) at templates.c:788 | #2 0x00007ffff5e96761 in affile_parse (lexer=<value optimized out>, instance=<value optimized out>, arg=<value optimized out>) at affile-grammar.y:592 | #3 0x00007ffff7b9bd72 in cfg_parser_parse (self=0x7ffff609ce30, cfg=<value optimized out>, yylloc=<value optimized out>, arg=<value optimized out>) at cfg-parser.h:83 | #4 plugin_parse_config (self=0x7ffff609ce30, cfg=<value optimized out>, yylloc=<value optimized out>, arg=<value optimized out>) at plugin.c:124 | #5 0x00007ffff7ba89f7 in main_parse (lexer=<value optimized out>, dummy=<value optimized out>, arg=<value optimized out>) at cfg-grammar.y:592 | #6 0x00007ffff7b7ee93 in cfg_parser_parse (self=0x611720, lexer=0x6188a0, parser=0x7ffff7dd92a0, result=<value optimized out>, arg=<value optimized out>) at cfg-parser.h:83 | #7 cfg_run_parser (self=0x611720, lexer=0x6188a0, parser=0x7ffff7dd92a0, result=<value optimized out>, arg=<value optimized out>) at cfg.c:370 | #8 0x00007ffff7b7efa6 in cfg_read_config (self=0x611720, fname=0x7ffff7bb7068 "/home/algernon/tmp/sng/install/3.3/etc/syslog-ng.conf", syntax_only=<value optimized out>, preprocess_into=0x0) at cfg.c:392 | #9 0x00007ffff7b970c1 in main_loop_init () at mainloop.c:611 | #10 0x0000000000401865 in main (argc=1, argv=0x7fffffffdc78) at main.c:242 | (gdb) `---- The config that triggers this is as follows: ,---- | @version: 3.3 | @include "scl.conf" | | source s_local { internal(); }; | destination d_local { file("/tmp/template-test" template("foo: bar")); }; | log { source(s_local); destination(d_local); }; `---- After bisecting the problem, I ended up with the following commit: ,---- | 2443b07b90c822414a628aae6a9ca513357b52fc is the first bad commit | commit 2443b07b90c822414a628aae6a9ca513357b52fc | Author: Balazs Scheidler <bazsi(a)balabit.hu> | Date: Sat Jun 11 14:09:01 2011 +0200 | | template: make template compilation thread safe | | Instead of doing template compilation on-demand, require the application | code to do it explicitly, at initialization, in order to avoid having to | acquire a lock during runtime. | | This patch contains an incompatible API change, and requires updating | all call-sites. | | The change makes it also possible to change template string of a constructed | LogTemplate class multiple times with a call to log_template_compile(). | | Signed-off-by: Balazs Scheidler <bazsi(a)balabit.hu> | | :040000 040000 a40fc910d215dac2184aa02ae1498c6748e3e64a dea1d2a7fc02d1d0ec8e095f2f62499ce42b895a M lib | :040000 040000 1641cf3152897406874716d4f090b5cc795292ab f66d2e5c482c3ff91b9d4219812766d66067c7a1 M modules | :040000 040000 2c2a37fd3a4f9c5bddc81813605c20ece033ce2f dff570d331b8e7ada6c1202761066878b9bb3910 M tests `---- I'm not quite sure why this happens yet, I'll keep digging. I'll post a patch if and when I figure out what exactly broke. Meanwhile, this bug report will serve as a reminder for me to keep looking. -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

4 11

TCP / max-connections & flow-control
by Paul Muther 01 Aug '11

01 Aug '11

Greetings all, I have not been able to find confirmation on this but I wanted to find details on TCP connections in 3.2 open-source. When a client establishes a TCP connection to a host log server, does it reuse that connection for all subsequent log transmissions? Does it have a threshold for destroying the connection and creating a new one if it does? Thanks much, PM

3 2

Check connection to server
by Josu Lazkano 01 Aug '11

01 Aug '11

Hello again. I had my syslog server configured and running. All servers has public IP but they are on the same LAN. Now I want to get all logs from a remote server. I configured the remote server same as LAN servers but there is no logs from WAN. How could I check if it is sending data to my syslog server? Maybe I must configure filters to do that? Thanks and best regards.

2 1

Inability to filter/log hostnames
by Norman Elton 01 Aug '11

01 Aug '11

I'm running syslog-ng 3.2.4 from RedHat's RPM. Unfortunately, I can't seem to log the hostname as specified in the incoming UDP packet. We don't do DNS resolution; rather, just want to log what the sending host is passing along. No relays in the mix, but we have keep_hostname() enabled. My global options: flush_lines(10); flush_timeout(750); time_reopen (10); log_fifo_size (1000); keep_hostname (yes); When I log $HOSTNAME or $HOST, I just get the sender's IP address. Similarly, filters based on these macros don't work properly. This all seemed to work on prior versions of syslog-ng (2.something). Any ideas? Thanks! Norman

2 1

3.3.0beta1 leaking memeory (Re: syslog-ng 3.3.0beta & ESX crashes)
by Hendrik Visage 01 Aug '11

01 Aug '11

Hi there, I've mailed Gergely the Valgrind output to analyze 3.3.0beta1 memory leak(s), so if anybody else are interested in it, please contact me for them. I've since compiled 3.2.4+noescape patch (since only 3.3.0+ have the default option for SQL insert I need/use for the autoincrementing ID field in systemevents), and using the same config file (other than @version 3.2 and the noescape('default') for my postgresql insertions) and the system is stable with no descernable memory growth compared to syslog-ng 3.3.0beta1's explosion in memory size. Hendrik On Mon, Jul 18, 2011 at 3:02 PM, Gergely Nagy <algernon(a)balabit.hu> wrote: > Hendrik Visage <hvjunk(a)gmail.com> writes: > >> Hi Gergely, >> >> Looks like I have a syslog-ng 3.3beta1 (fetched from GITHEAD) that is >> having a serious memory leak (and I'm not yet pushing all our logs to >> it ;( >> >> Any advice in how to help you track this problem? > > Hrm. Well, valgrind would be best, but that slows things down to almost > a grounding halt. (But in case you want to try that: > G_SLICE=always-malloc valgrind --leak-check=full -v --show-reachable-yes > --log-file=valgrind.log syslog-ng -F would be the command to run ;) > > Other than that, a config could be useful, so I can try and reproduce > the problem (or at least identify possible areas where the leak might be > coming from, and go from there), and see if I can fix it. > > -- > |8] > > ______________________________________________________________________________ > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng > Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng > FAQ: http://www.balabit.com/wiki/syslog-ng-faq > >

3 6

AUTO: David Barry is out of the office through 8/9/2011 (returning 08/09/2011)
by dbarry＠princesscruises.com 30 Jul '11

30 Jul '11

I am out of the office until 08/09/2011. I will respond to your message when I return. For general security issues, you may contact Russell Swift or Christy Belknap. Or, you may contact my manager, Paul Johnson. Note: This is an automated response to your message "syslog-ng Digest, Vol 75, Issue 38" sent on 7/30/2011 3:00:01. This is the only notification you will receive while this person is away.

1 0

AUTO: James B Horwath/TheGuardian is out of the office. (returning 08/06/2011)
by James B Horwath 30 Jul '11

30 Jul '11

I am out of the office until 08/06/2011. I am out of the office until Monday,August 8, 2011 with limited access to e-mail. If there are Nitro issues please contact Dante Rodino at 212-919-2819 or Sunil Gupta at 610-807-6710. Any other issues please contact Christopher Clarke manager of Security Operations Thanks, Jim Note: This is an automated response to your message "syslog-ng Digest, Vol 75, Issue 38" sent on 7/30/11 6:00:01. This is the only notification you will receive while this person is away. ----------------------------------------- This message, and any attachments to it, may contain information that is privileged, confidential, and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are notified that any use, dissemination, distribution, copying, or communication of this message is strictly prohibited. If you have received this message in error, please notify the sender immediately by return e-mail and delete the message and any attachments. Thank you.

1 0

Rapid filling of /var/log/syslog-ng.log with "--MARK--" messages?
by mike＠lague.org 25 Jul '11

25 Jul '11

On a system running syslog-ng 3.0.5, we observed the following: * * * Jul 21 11:53:37.724586 sys89 -- MARK -- Jul 21 11:53:37.724623 sys89 -- MARK -- Jul 21 11:53:37.724660 sys89 -- MARK -- Jul 21 11:53:37.724696 sys89 -- MARK -- Jul 21 11:53:37.724733 sys89 -- MARK -- Jul 21 11:53:37.724770 sys89 -- MARK -- Jul 21 11:53:37.724807 sys89 -- MARK -- Jul 21 11:53:37.724843 sys89 -- MARK -- Jul 21 11:53:37.724880 sys89 -- MARK ? The log was written to at this frequency, with these "MARK" messages, until the partition filled up. It appears that the first of these messages was logged very close in time to when the system date was set back by a couple of minutes. However, I was unable to reproduce this result on a different system (by setting the date back in time.) However, is it possible that this problem is the same as: SYSLOG-NG] [BUG 110] NEW: ADVANCING THE SYSTEM DATE AND TIME BEYOND 2038 CAUSES FLOOD OF MESSAGES (i.e., https://lists.balabit.hu/pipermail/syslog-ng/2011-January/015754.html) It appears that this is fixed in 3.2.3; it is fixed in other versions? In versions where this is not fixed, is it possible to avoid this problem by turning off the "mark" feature, i.e., by putting mark_freq(0); in the options clause of the configuration? Thank you, --Mike Lague

1 0

Re: [syslog-ng] log file size limit
by Hendrik Visage 25 Jul '11

25 Jul '11

Better to make use of monthly, weekly, daily, hourly type logs in syslog-ng, and the to use cront to clean up as needed ;) On Mon, Jul 25, 2011 at 5:50 PM, William Tambellini <William.Tambellini(a)galaxysemi.com> wrote: > Dear Syslog-ng team, > > We are currently evaluating syslog-ng in order to satisfy some requests > from our customers regarding logs from our software. > > One of their requests is to limit log file size using some kind of rotation > algo : if size > 6Mo then rotate to another file. > > We have found some options in file destination like fifo buffer control but > nothing about files rotation. > > Is there anything for that kind of feature ? > > Regards > > WT. > > > > ---------------------- > > William Tambellini > > Lead Technical Organizer > > www.galaxysemi.com > > > > “Sometimes I think the surest sign that intelligent life exists elsewhere in > the universe is that none of it has tried to contact us.” > > Bill Waterson (Calvin & Hobbes) > > > > > > ______________________________________________________________________________ > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng > Documentation: > http://www.balabit.com/support/documentation/?product=syslog-ng > FAQ: http://www.balabit.com/wiki/syslog-ng-faq > > >

1 0

Re: [syslog-ng] log file size limit
by Patrick H. 25 Jul '11

25 Jul '11

Syslog-ng doesnt perform file rotation. This would the the job of something like 'logrotate <https://fedorahosted.org/logrotate/>'. Sent: Mon Jul 25 2011 09:50:42 GMT-0600 (MST) From: William Tambellini <William.Tambellini(a)galaxysemi.com> To: syslog-ng(a)lists.balabit.hu Subject: [syslog-ng] log file size limit > > Dear Syslog-ng team, > > We are currently evaluating syslog-ng in order to satisfy some > requests from our customers regarding logs from our software. > > One of their requests is to limit log file size using some kind of > rotation algo : if size > 6Mo then rotate to another file. > > We have found some options in file destination like fifo buffer > control but nothing about files rotation. > > Is there anything for that kind of feature ? > > Regards > > WT. > > ---------------------- > > William Tambellini > > Lead Technical Organizer > > www.galaxysemi.com > > "Sometimes I think the surest sign that intelligent life exists > elsewhere in the universe is that none of it has tried to contact us." > > Bill Waterson (Calvin & Hobbes) > > > ______________________________________________________________________________ > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng > Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng > FAQ: http://www.balabit.com/wiki/syslog-ng-faq >

1 0

log file size limit
by William Tambellini 25 Jul '11

25 Jul '11

Dear Syslog-ng team, We are currently evaluating syslog-ng in order to satisfy some requests from our customers regarding logs from our software. One of their requests is to limit log file size using some kind of rotation algo : if size > 6Mo then rotate to another file. We have found some options in file destination like fifo buffer control but nothing about files rotation. Is there anything for that kind of feature ? Regards WT. ---------------------- William Tambellini Lead Technical Organizer www.galaxysemi.com "Sometimes I think the surest sign that intelligent life exists elsewhere in the universe is that none of it has tried to contact us." Bill Waterson (Calvin & Hobbes)

1 0

can syslog-ng extract part of the hostname and make directory on the fly?
by fedora fedora 22 Jul '11

22 Jul '11

I have few hundreds of hosts, and say hosta has dns name groupa-hosta hostb has dns name groupbhostb hostc has dns name groupc.hostc ..etc..etc (note the first 6 characters defines the group) When a log comes in, can syslog-ng extracts the first 6 character from dns name, and drop the log file to the corresponding group? (So log from hostb will go under /log/groupb/groupahostb ) I cannot predefine the groups though, since i don't know how many groups will be there, the only thing i know is the first 6 character of the hostname will be the group name thanks

3 2

Anyone Going to Cisco Live/Networkers in Las Vegas next week?
by Clayton Dukes 22 Jul '11

22 Jul '11

Hey everyone, If any of you are attending the Cisco Live conference in Las Vegas next week and are interested in learning more about building scalable syslog management solutions, please feel free to sign up for my session: BRKNMS-2031. I will be covering step-by-step instructions on how to set up a system using syslog-ng. If you can't make the session, there is also a Lunch/Table topic you can sign up for just to pick my brain (what little there is, heh). See you there! ______________________________________________________________ Clayton Dukes ______________________________________________________________

2 5

Severity/Pri/Level
by Paul Muther 22 Jul '11

22 Jul '11

Hello all, Just joined the list. I was wondering if there was a method to rewrite the severity of an event based on the text of the entry? I know I can use regex to read the message but from what I have seen I can't set the severity level of an event. In reading posts it would appear it was a feature under consideration in the past. Thanks much, PM

3 4

3.3b1: "EOF occurred while idle"
by Nix 22 Jul '11

22 Jul '11

I'm seeing a lot of these: May 29 18:59:48 fold err: POLLERR occurred while idle; fd='25' [...] May 29 19:04:13 fold err: EOF occurred while idle; fd='25' (It also just coredumped, but this led to an autoreboot and the log was not preserved across the reboot :( ) -- NULL && (void)

2 7

cannot filter by program name
by Luis Pugoy 21 Jul '11

21 Jul '11

Hello. I am having trouble filtering by program name and was wondering if anyone could help. I am using syslog-ng v3.1.2 on Ubuntu 10.04 machines. I installed syslog-ng by installing the provided deb package. In Machine 1, the relevant lines in syslog-ng.conf are below: source s_tag_a { file('/home/ubuntu/a.tag' flags(no-parse) program_override('atag')); }; destination d_tag_a { tcp('machine2' port(40000)); }; log { source(s_tag_a); destination(d_tag_a); }; source s_tag_b { file('/home/ubuntu/b.tag' flags(no-parse) program_override('btag')); }; destination d_tag_b { tcp('machine2' port(40000)); }; log { source(s_tag_b); destination(d_tag_b); }; In Machine 2, the relevant lines in syslog-ng.conf are below: source s_tag { tcp(ip(0.0.0.0) port(40000)); }; destination d_tag_a { file('/home/ubuntu/a.tag'); }; filter f_tag_a { program('atag'); }; log { source(s_tag); destination(d_tag_a); filter(f_tag_a); }; destination d_tag_b { file('/home/ubuntu/b.tag'); }; filter f_tag_b { program('btag'); }; log { source(s_tag); destination(d_tag_b); filter(f_tag_b); }; Now when I execute the following commands in Machine 1: # echo 'atag' >> a.tag # echo 'btag' >> b.tag In Machine 2, the files a.tag and b.tag both contain the given lines. # cat a.tag atag btag # cat b.tag atag btag Is there something wrong in my configuration? Thank you.

1 1

Lot of log lost
by maxime.denier＠orange-ftgroup.com 20 Jul '11

20 Jul '11

Hello, I have recently installed syslog-ng OSE 3.1 as log collector and I face a problem. A great number of logs arrive on the server, but a little part of them arrive in the destination files, but all the destination files have some logs on it. I have enabled the verbose mode and I see this: Jul 20 07:52:04 sparte1 syslog-ng[2557]: Initializing destination file writer; template='/var/logs/${NSM.DEVICE:-Unknown_device}/${NSM.RECEIVED_TIME.YE AR}${NSM.RECEIVED_TIME.MONTH}${NSM.RECEIVED_TIME.DAY}2400.csv', filename='/var/logs/zidane2/201107202400.csv' Jul 20 07:52:31 sparte1 syslog-ng[2557]: Reaping unused destination files; template='/var/logs/${NSM.DEVICE:-Unknown_device}/${NSM.RECEIVED_TIME.YE AR}${NSM.RECEIVED_TIME.MONTH}${NSM.RECEIVED_TIME.DAY}2400.csv' Jul 20 07:53:01 sparte1 syslog-ng[2557]: Reaping unused destination files; template='/var/logs/${NSM.DEVICE:-Unknown_device}/${NSM.RECEIVED_TIME.YE AR}${NSM.RECEIVED_TIME.MONTH}${NSM.RECEIVED_TIME.DAY}2400.csv' Jul 20 07:53:01 sparte1 syslog-ng[2557]: Destination timed out, reaping; template='/var/logs/${NSM.DEVICE:-Unknown_device}/${NSM.RECEIVED_TIME.YE AR}${NSM.RECEIVED_TIME.MONTH}${NSM.RECEIVED_TIME.DAY}2400.csv', filename='/var/logs/peony2/201107202400.csv' Jul 20 07:53:01 sparte1 syslog-ng[2557]: Closing log transport fd; fd='31' Jul 20 07:53:01 sparte1 syslog-ng[2557]: Destination timed out, reaping; template='/var/logs/${NSM.DEVICE:-Unknown_device}/${NSM.RECEIVED_TIME.YE AR}${NSM.RECEIVED_TIME.MONTH}${NSM.RECEIVED_TIME.DAY}2400.csv', filename='/var/logs/decca2/201107202400.csv' Jul 20 07:53:01 sparte1 syslog-ng[2557]: Closing log transport fd; fd='19' Jul 20 07:53:16 sparte1 syslog-ng[2557]: Initializing destination file writer; template='/var/logs/${NSM.DEVICE:-Unknown_device}/${NSM.RECEIVED_TIME.YE AR}${NSM.RECEIVED_TIME.MONTH}${NSM.RECEIVED_TIME.DAY}2400.csv', filename='/var/logs/hyenne2/201107202400.csv' Jul 20 07:53:17 sparte1 syslog-ng[2557]: Initializing destination file writer; template='/var/logs/${NSM.DEVICE:-Unknown_device}/${NSM.RECEIVED_TIME.YE AR}${NSM.RECEIVED_TIME.MONTH}${NSM.RECEIVED_TIME.DAY}2400.csv', filename='/var/logs/olive2/201107202400.csv' Jul 20 07:53:31 sparte1 syslog-ng[2557]: Reaping unused destination files; template='/var/logs/${NSM.DEVICE:-Unknown_device}/${NSM.RECEIVED_TIME.YE AR}${NSM.RECEIVED_TIME.MONTH}${NSM.RECEIVED_TIME.DAY}2400.csv' Jul 20 07:53:31 sparte1 syslog-ng[2557]: Destination timed out, reaping; template='/var/logs/${NSM.DEVICE:-Unknown_device}/${NSM.RECEIVED_TIME.YE AR}${NSM.RECEIVED_TIME.MONTH}${NSM.RECEIVED_TIME.DAY}2400.csv', filename='/var/logs/zidane2/201107202400.csv' Jul 20 07:53:31 sparte1 syslog-ng[2557]: Closing log transport fd; fd='24' I haven't found information about the root cause of these timed out. This seems to be a writing problem. Before using syslog-ng, Logs were processed by a application owned by the firewall publisher on the same type of hardware without this great number of log lost. If any body have already faced this problem and have a solution, it would be wonderful. Regards, Maxime Denier ******************************************************************************** IMPORTANT.Les informations contenues dans ce message electronique y compris les fichiers attaches sont strictement confidentielles et peuvent etre protegees par la loi. Ce message electronique est destine exclusivement au(x) destinataire(s) mentionne(s) ci-dessus. Si vous avez recu ce message par erreur ou s il ne vous est pas destine, veuillez immediatement le signaler a l expediteur et effacer ce message et tous les fichiers eventuellement attaches. Toute lecture, exploitation ou transmission des informations contenues dans ce message est interdite. Tout message electronique est susceptible d alteration. A ce titre, le Groupe France Telecom decline toute responsabilite notamment s il a ete altere, deforme ou falsifie. De meme, il appartient au destinataire de s assurer de l absence de tout virus. IMPORTANT.This e-mail message and any attachments are strictly confidential and may be protected by law. This message is intended only for the named recipient(s) above. If you have received this message in error, or are not the named recipient(s), please immediately notify the sender and delete this e-mail message. Any unauthorized view, usage or disclosure ofthis message is prohibited. Since e-mail messages may not be reliable, France Telecom Group shall not be liable for any message if modified, changed or falsified. Additionally the recipient should ensure they are actually virus free. ********************************************************************************

3 2

max-connections(1000)
by Josu Lazkano 20 Jul '11

20 Jul '11

Hello list, this morning I had some problems with my syslog system I am adding lots of server to send TCP logs to the syslog-ng server. But there are not on the server. After readingon the web, I notice that there is a max-connections parameter on the TCP. So I fix to 50. With this there are more logs, then I added 100 and there more. So I fix it to 1000: source s_all { internal(); unix-stream("/dev/log"); file("/proc/kmsg" log_prefix("kernel: ")); tcp(max-connections(1000)); }; Is this a good idea? How can I check if this is a good maximum connection value? Thanks for all your help and best regards.

4 5

Securing syslog-ng
by Kārlis Repsons 20 Jul '11

20 Jul '11

All, I've been sorting system information with syslog-ng just fine, but it happened in a LAN environment. Now I plan starting off with a public IPv6 address and that raises some concerns. What would you advice to check to be sure? How should remote logging be set up so that some mutual (or at least client) cryptographic authentication happens? Thanks...

4 3

syslogd as well as syslog-ng running simultaneously
by Rakesh Rajasekharan 20 Jul '11

20 Jul '11

Hi, I have compiled syslog-ng 3.0.7 successfully on an AIX machine. But I have noticed that I now have both traditional syslog syslogd as well as syslog-ng daemons running simultaneously. root 237722 139372 0 Mar 24 - 2:34 /opt/soe/local/syslog_ng/sbin/syslog-ng -F -f /opt/soe/local/syslog_ng/etc/syslog-ng.conf -p /var/run/syslog-ng.pid root 307444 139372 0 Apr 04 - 0:03 /usr/sbin/syslogd Is this fine or ther is something that has to be done to fix it. Please let me know. Thanks, Rakesh

3 2

AIX Syslog Messages
by Ricardo Oliveira 20 Jul '11

20 Jul '11

Hi, I'm having some problems properly storing messages received from AIX servers. The format which they come in is like this: "Jul 5 19:30:59 Message forwarded from server2: su: from root to ..." According to a thread on this mailing list (https://lists.balabit.hu/pipermail/syslog-ng/2006-October/009372.html) and if I understood correctly, this should be OK, and I should get the expected behaviour of replacing this with the form: "Jul 5 19:30:59 server2 su: from root to ..." However, what I get in the log is: "Jul 5 19:30:59 192.168.1.1 su: from root to ..." Where the 192.168.1.1 is the IP of the machine I got the message from and not the name of the server (server2 in this case). The issue here is that these messages belong to several machines which are sending their syslog messages to a NIM server which in turn forwards them to our syslog server, so the IP we end up with is not the machine's IP, but rather the NIM server IP, which is not what we need. I tried parsing the message on arrival, but it doesn't work, I suppose it's because syslog-ng processes it before the parsers kick in. Is there a way to do this? TIA, Ricardo.

3 4

Third-party module collection updates
by Gergely Nagy 20 Jul '11

20 Jul '11

Hi! A repository for third-party and experimental modules was announced in May[0], and I'd like to give a quick update about what's been brewing there since then: * The SMTP destination has been updated a few times for newer versions of syslog-ng 3.3. As of this writing, it's up to date with current 3.3. * The MongoDB destination was copied there too, as its easier to maintain one copy, instead of a branch for 3.3 and 3.4 each. It's also been updated to use recent versions of the libmongo-client library, so it can take advantage of features added in the past few months, such as the ability to use Replica Sets and safe mode (see my earlier RFC[1] about these - what's mentioned there is implemented now). The host() and port() parameters are now gone, and no backward compatibility exists yet. That will be fixed at a later time. * On the modules/rss branch, there is an initial RSS destination by Viktor Tusa <tusa(a)balabit.hu>. It has a few serious issues, like not escaping the log messages, and outputting an invalid format, but those will be corrected Soon. There's not much one can use it for just yet, but it's going to be a neat toy, nevertheless! * The log visualisation helper[2] is included in tools/visualisation/. By the way, a new demo video is also available at http://vimeo.com/25700679, which shows ~3 months of logs in 50 minutes. Quite a bit longer than the previous video, and made with different gource settings, to be a bit less messy. For the time being, the module collection is primarily developed against syslog-ng 3.3. The modules have a high chance of working with 3.4 aswell, but that's not tested, and only minimal effort is made to be compatible with both. I'll try to figure out an easy way to properly support both versions in the future, but at the moment, the focus is on 3.3. 0: http://article.gmane.org/gmane.comp.syslog-ng/11403 1: http://article.gmane.org/gmane.comp.syslog-ng/11557 2: http://article.gmane.org/gmane.comp.syslog-ng/11569 -- |8]

2 1