March 2014 - syslog-ng - lists.balabit.hu

pattern-db parser
by Justin B 31 Mar '14

31 Mar '14

Hello I have the Syslog-ng edition that came in built with Solaris Operating system and I did not find the pattern-db.xml or the default directory where it exists. Can I still create manually and use the Pattern-db parser?? I would want to use this on the operating system logs. Appreciate your response on this -- Kaladhar

2 1

Timestamp format
by Ou, Jimmy 28 Mar '14

28 Mar '14

Hello, I am having trouble using the timestamp formatter. The document says I can use ts_format() as rfc3164, bsd, rfc3339, or iso. I've tried multiple values and have gotten the same result. I've tried putting the ts_format(rfc3339) in the global options and in the destination's syslog() and have gotten the same result. My test logs show the following no matter what ts_format I set: <113>1 2014-11-25T11:00:00+00:00 10.1.1.1 RedBox - I want it to look like the following: <113>1 2014-11-25T11:00:00.000Z 10.1.1.1 RedBox - Am I missing something? Thanks, Jimmy Ou

2 1

[Bug 274] New: 3.5.4.1 memory leak
by bugzilla＠bugzilla.balabit.com 27 Mar '14

27 Mar '14

https://bugzilla.balabit.com/show_bug.cgi?id=274 Summary: 3.5.4.1 memory leak Product: syslog-ng Version: 3.5.x Platform: PC OS/Version: Linux Status: NEW Severity: normal Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: erempel(a)uvic.ca Type of the Report: --- Estimated Hours: 0.0 in our production environment our syslog process consumed 8GB of RAM while not queueing any messages. We take 10 minute syslog-ng-ctl stats snapshots and 15 minute process (ps -ef) style snapshots. There is definitely a memory leak there somewhere. We run multiple pattern databases, with multiple json output streams along side regular file destinations. What would you like from me to track it down? - valgrind - core dump - recompile with debuggin? If you provide the flags, I'll provide the data :-) -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 3

How to assign specific value in a pattern
by C. L. Martinez 27 Mar '14

27 Mar '14

Hi all, I need to assign a specific value to only one pattern inside a ruleset with multiple patterns. For example, I have defined the following patterns for one ruleset: <ruleset name="kernel" id='1002'> <pattern>kernel</pattern> <rules> <rule provider="By me" class='1002' id='1002'> <patterns> <pattern>pattern1</pattern> <pattern>pattern2</pattern> <pattern>pattern3</pattern> </patterns> </rule> </rules> </ruleset> The I need to do the following: <ruleset name="kernel" id='1002'> <pattern>kernel</pattern> <rules> <rule provider="By me" class='1002' id='1002'> <patterns> <pattern>pattern1</pattern> <pattern>pattern2</pattern> <values> <value name="mytestvalue">Downloaded file</value> </values> <pattern>pattern3</pattern> </patterns> </rule> </rules> </ruleset> As you can see I need to assign "Downloaded file" value to "mytestvalue" but only for pattern2 and not the others ... At this time, value "mytestvalue" is assigned to all patterns ... Is it possible to do this??

2 4

syslog-ng Potential Memory Leak
by Tamayo, Andres 26 Mar '14

26 Mar '14

Hello, I have been testing syslog-ng under different scenarios and I noticed that when syslog-ng is trying to connect to an external syslog-ng server, it exhibits what could be called a memory leak. Let me explain how I am setting up my test. I have a syslog-ng client running collecting logs from a file (source) and sending them to an external syslog-ng server over TCP (destination). If the syslog-ng server is online, the syslog-ng client daemon process virtual memory does not increase. Now, when I kill the syslog-ng server, the syslog-ng client daemon process starts to request more and more virtual memory. Here are the details: I started both the syslog-ng client and server and started to send logs from the client to the server. At this point, the syslog-ng client daemon maintained its VmSize equal to 7144 Kbytes. This condition was stable for 4 hours while the server was online. Then, after 4 hours, I killed the syslog-ng server. As soon as I did that, the syslog-ng client daemon virtual memory size started to increate every few seconds. Started at 7144 Kbytest and after 4 hours VmSize was equal to 7752 Kbytes. So it grew 608 Kbytes over the next 4 hours. Finally, after the 4 hours I just mentioned, I restarted the syslog-ng server. All the logs that had not been sent for 4 hours were transmitted to the syslog-ng server in a few seconds. At this point, the syslog-ng client daemon virtual memory stopped growing and stayed stable for 4 hours. I think that this behavior may be related to the logs generated every time the client tries to reconnect with the server. By the way, the client time_reopen configuration was set to 15 seconds. I am worried that over a long long period of time, if the syslog-ng server is not online, the syslog-ng client daemon may be killed by the OS for using too much memory. Please let me know if you have any ideas how to fix this problem, thank you.

3 4

Re: [syslog-ng] odd behavior writing "binary" logfiles - followup data
by jrhendri＠roadrunner.com 25 Mar '14

25 Mar '14

and in the self-inflicted wounds category... I did some additional digging on the box and discovered what certainly *appears* to be the cause: [a0142566@vxpip-eeisl001 ~]$ grep host /etc/nsswitch.conf #hosts: db files nisplus nis dns hosts: files dns [a0142566@vxpip-eeisl001 ~]$ [a0142566@vxpip-eeisl001 ~]$ diff /etc/hosts /etc/hosts.03162014 2c2 < 10.177.74.55 vxpip-eeisl001 --- > 10.177.74.55 . vxpip-eeisl001 [a0142566@vxpip-eeisl001 ~]$ I fixed this and then re-ran some tests using the attached syslog-ng.conf.DEBUG and the following invocation: sudo strace /usr/local/sbin/syslog-ng -vde -f /usr/local/etc/syslog-ng.conf.DEBUG > /tmp/syslog-ng-DEBUG-strace 2>&1 and things look MUCH better. [a0142566@vxpip-eeisl001 etc]$ file /var/log/debug* /var/log/debug.messages.vxpip-eeisl001.auth.info: ASCII text /var/log/debug.messages.vxpip-eeisl001.authpriv.notice: ASCII text /var/log/debug.messages.vxpip-eeisl001.cron.info: ASCII text /var/log/debug.network_messages.vddp13e-0d6b677: ASCII text /var/log/debug.template_messages: ASCII text [a0142566@vxpip-eeisl001 etc]$ Note: I also found another system that has the same config error: [a0142566@vxkip-eeisl001 ~]$ cat /etc/hosts 127.0.0.1 localhost.localdomain localhost 10.181.73.32 . vxkip-eeisl001 and ran a quick test through a resolver perl script that does (basically) this: my $iaddr = inet_aton($thingy); if (my $name = gethostbyaddr($iaddr,AF_INET)) { print "$thingy,\t$name\n"; } [a0142566@vxkip-eeisl001 ~]$ echo 10.181.73.32 | ~/bin/resolve.pl 10.181.73.32, . Sorry again for bothering the list (I really should have caught that!!... And yes – I will be bringing this to the OS team’s attention :-) Although it does raise a point that (maybe) a default value like "please_fix_my_hostname" could be used in case this happens elsewhere? Just a thought. Thank you again!! Jim ---- Jim Hendrick <jrhendri(a)roadrunner.com> wrote: > I will take some time today and re-run this with various debugging flags. > > (just realized I should have... ) > > On 03/15/2014 09:37 PM, jrhendri(a)roadrunner.com wrote: > > Odd - I do not see any gethost calls at all in the strace > > > > this is from "sudo strace /usr/local/sbin/syslog-ng -f /usr/local/etc/syslog-ng.conf > syslog-strace 2>&1" > > > > > > I include the config.status and config.log also to see if that may help. > > > > Jim > > > > > > ---- Balazs Scheidler <bazsi77(a)gmail.com> wrote: > >> During startup syslog-ng queries the hostname using gethostname. Can you > >> strace syslog-ng and look for this call to see what it returns? > >> > >> We are also resolving this using the dns which is also interesting. > >> > >> Thanks > >> On Mar 13, 2014 7:08 PM, <jrhendri(a)roadrunner.com> wrote: > >> > >>> OK - after much poking around (including testing this on a different host > >>> with this morning's latest source) here's what I found > >>> > >>> I believe that syslog-ng is (for some reason) unable to come up with a > >>> correct hostname for the local system. > >>> I would really like to know why this is happening > >>> I would also really like to know if any of the source options I tried > >>> should have worked (I was kind of desperate with some of the combinations...) > >>> > >>> That said - here's my summary: > >>> > >>> Examining the "binary" files, It was always writing 256 bytes of "0" where > >>> ${HOST} should have been. > >>> > >>> It *never* happened with a network source (tried with Kiwi sysloggen using > >>> RCF and non RCF messages) > >>> It *always* happened with any local source including all the ones > >>> commented out (I tested each separately) > >>> > >>> source s_local { > >>> unix-dgram("/dev/log"); # standard Linux log source (this is the default > >>> place for the syslog() function to send logs to) > >>> #!# unix-stream("/dev/log"); # standard Linux log source (this is the > >>> default place for the syslog() function to send logs to) > >>> #!# system(); > >>> #!# internal(); > >>> }; > >>> > >>> Once I identified this behavior, I tried a number of options / > >>> combinations to work around this including (each separately) > >>> source s_local { > >>> system(); > >>> #!# system( keep_hostname(no) host_override("localhost") ); > >>> #!# system( keep_hostname(yes) host_override("localhost") ); > >>> #!# system( flags(no-parse) host_override("localhost") ); > >>> #!# system( flags(no-hostname) host_override("localhost") ); > >>> #!# system( flags(no-parse) ); > >>> #!# system( flags(no-hostname) ); > >>> #!# internal( ); > >>> }; > >>> > >>> And nothing worked. > >>> > >>> Finally I kinda punted... I am now doing something really kludgy as a work > >>> around: > >>> source s_local { > >>> system(); > >>> internal( ); > >>> }; > >>> > >>> source s_network { > >>> udp(); > >>> }; > >>> > >>> destination d_local_template { > >>> > >>> file("/data/syslog-ng/$YEAR/$MONTH/$DAY/localhost/localhost.$FACILITY.$PRIORITY.$DATE" > >>> template("${ISODATE} localhost ${PROGRAM} ${MESSAGE}\n") ); > >>> }; > >>> > >>> > >>> destination d_network { > >>> > >>> file("/data/syslog-ng/$YEAR/$MONTH/$DAY/$HOST_FROM/$HOST_FROM.$FACILITY.$PRIORITY.$DATE"); > >>> }; > >>> > >>> log { > >>> source(s_local); > >>> destination(d_local_template); > >>> }; > >>> > >>> log { > >>> source(s_network); > >>> destination(d_network); > >>> }; > >>> > >>> > >>> ---- jrhendri(a)roadrunner.com wrote: > >>>> I have a really odd problem that I hope someone can assist with. > >>>> > >>>> My install is inserting a number of zeroes into the logfile between the > >>> date and the rest of the message. > >>>> Here are some details: > >>>> > >>>> System is RHEL: > >>>> > >>>> $ uname -a > >>>> Linux vxpip-eeisl001 2.6.32-358.el6.x86_64 #1 SMP Tue Jan 29 11:47:41 > >>> EST 2013 x86_64 x86_64 x86_64 GNU/Linux > >>>> $ > >>>> > >>>> > >>>> Syslog-ng was built on the box yesterday with these sources: > >>>> eventlog-0.2.12+20120504+1700 > >>>> syslog-ng-3.5.3 > >>>> > >>>> > >>>> $ /usr/local/sbin/syslog-ng --version > >>>> syslog-ng 3.5.3 > >>>> Installer-Version: 3.5.3 > >>>> Revision: ssh+git://algernon@git.balabit > >>> /var/scm/git/syslog-ng/syslog-ng-ose--mainline--3.5#master#ccb05a22408ba4c837d998b2538854d994f845a5 > >>>> Compile-Date: Mar 12 2014 11:37:32 > >>>> Available-Modules: > >>> afmongodb,afstomp,syslogformat,affile,basicfuncs,csvparser,confgen,system-source,afamqp,linux-kmsg-format,afprog,afuser,afsocket,dbparser,cryptofuncs,afsocket-notls > >>>> Enable-Debug: off > >>>> Enable-GProf: off > >>>> Enable-Memtrace: off > >>>> Enable-IPv6: on > >>>> Enable-Spoof-Source: off > >>>> Enable-TCP-Wrapper: off > >>>> Enable-Linux-Caps: off > >>>> Enable-Pcre: off > >>>> > >>>> > >>>> > >>>> This is the default config file that came with it: > >>>> > >>>> > >>>> > >>> ############################################################################# > >>>> # Default syslog-ng.conf file which collects all local logs into a > >>>> # single file called /var/log/messages. > >>>> # > >>>> > >>>> @version: 3.5 > >>>> @include "scl.conf" > >>>> > >>>> source s_local { > >>>> system(); > >>>> internal(); > >>>> }; > >>>> > >>>> source s_network { > >>>> udp(); > >>>> }; > >>>> > >>>> destination d_local { > >>>> file("/var/log/messages"); > >>>> }; > >>>> > >>>> log { > >>>> source(s_local); > >>>> > >>>> # uncomment this line to open port 514 to receive messages > >>>> #source(s_network); > >>>> destination(d_local); > >>>> }; > >>>> > >>>> > >>>> > >>>> Here is the file: > >>>> > >>>> > >>>> $ sudo file /var/log/messages > >>>> /var/log/messages: data > >>>> $ sudo cat /var/log/messages > >>>> Mar 13 10:43:00 syslog-ng[18451]: syslog-ng starting up; version='3.5.3' > >>>> Mar 13 10:43:01 CROND[18454]: (root) CMD (/util/avail/get_avail.sh > >>> 1>/util/avail/logs/get_avail.out 2>&1) > >>>> Mar 13 10:43:16 sudo: a0142566 : TTY=pts/2 ; PWD=/home/a0142566 ; > >>> USER=root ; COMMAND=/usr/bin/file /var/log/messages > >>>> And this is a capture from vi in hex mode on that file. > >>>> > >>>> > >>>> 0000000: 4d61 7220 3133 2031 303a 3433 3a30 3020 Mar 13 10:43:00 > >>>> 0000010: 0000 0000 0000 0000 0000 0000 0000 0000 ................ > >>>> 0000020: 0000 0000 0000 0000 0000 0000 0000 0000 ................ > >>>> 0000030: 0000 0000 0000 0000 0000 0000 0000 0000 ................ > >>>> 0000040: 0000 0000 0000 0000 0000 0000 0000 0000 ................ > >>>> 0000050: 0000 0000 0000 0000 0000 0000 0000 0000 ................ > >>>> 0000060: 0000 0000 0000 0000 0000 0000 0000 0000 ................ > >>>> 0000070: 0000 0000 0000 0000 0000 0000 0000 0000 ................ > >>>> 0000080: 0000 0000 0000 0000 0000 0000 0000 0000 ................ > >>>> 0000090: 0000 0000 0000 0000 0000 0000 0000 0000 ................ > >>>> 00000a0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ > >>>> 00000b0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ > >>>> 00000c0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ > >>>> 00000d0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ > >>>> 00000e0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ > >>>> 00000f0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ > >>>> 0000100: 0000 0000 0000 0000 0000 0000 0000 0020 ............... > >>>> 0000110: 7379 736c 6f67 2d6e 675b 3138 3435 315d syslog-ng[18451] > >>>> 0000120: 3a20 7379 736c 6f67 2d6e 6720 7374 6172 : syslog-ng star > >>>> 0000130: 7469 6e67 2075 703b 2076 6572 7369 6f6e ting up; version > >>>> 0000140: 3d27 332e 352e 3327 0a4d 6172 2031 3320 ='3.5.3'.Mar 13 > >>>> 0000150: 3130 3a34 333a 3031 2000 0000 0000 0000 10:43:01 ....... > >>>> 0000160: 0000 0000 0000 0000 0000 0000 0000 0000 ................ > >>>> 0000170: 0000 0000 0000 0000 0000 0000 0000 0000 ................ > >>>> 0000180: 0000 0000 0000 0000 0000 0000 0000 0000 ................ > >>>> 0000190: 0000 0000 0000 0000 0000 0000 0000 0000 ................ > >>>> 00001a0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ > >>>> 00001b0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ > >>>> 00001c0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ > >>>> 00001d0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ > >>>> 00001e0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ > >>>> 00001f0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ > >>>> 0000200: 0000 0000 0000 0000 0000 0000 0000 0000 ................ > >>>> 0000210: 0000 0000 0000 0000 0000 0000 0000 0000 ................ > >>>> 0000220: 0000 0000 0000 0000 0000 0000 0000 0000 ................ > >>>> 0000230: 0000 0000 0000 0000 0000 0000 0000 0000 ................ > >>>> 0000240: 0000 0000 0000 0000 0000 0000 0000 0000 ................ > >>>> 0000250: 0000 0000 0000 0000 2043 524f 4e44 5b31 ........ CROND[1 > >>>> 0000260: 3834 3534 5d3a 2028 726f 6f74 2920 434d 8454]: (root) CM > >>>> 0000270: 4420 282f 7574 696c 2f61 7661 696c 2f67 D (/util/avail/g > >>>> 0000280: 6574 5f61 7661 696c 2e73 6820 313e 2f75 et_avail.sh 1>/u > >>>> 0000290: 7469 6c2f 6176 6169 6c2f 6c6f 6773 2f67 til/avail/logs/g > >>>> 00002a0: 6574 5f61 7661 696c 2e6f 7574 2032 3e26 et_avail.out 2>& > >>>> 00002b0: 3129 0a4d 6172 2031 3320 3130 3a34 333a 1). > >>>> > >>>> > >>>> Any ideas? > >>>> Thanks in advance folks!! > >>>> Jim > >>>> > >>>> > >>>> > >>> ______________________________________________________________________________ > >>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng > >>>> Documentation: > >>> http://www.balabit.com/support/documentation/?product=syslog-ng > >>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq > >>>> > >>> > >>> ______________________________________________________________________________ > >>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng > >>> Documentation: > >>> http://www.balabit.com/support/documentation/?product=syslog-ng > >>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq > >>> > >>> > >>> > >>> > >>> ______________________________________________________________________________ > >>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng > >>> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng > >>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq > >>> >

2 1

syslog-ng ports in FreeBSD
by Peter Czanik 25 Mar '14

25 Mar '14

Hi, Right now syslog-ng 3.4 is the default syslog-ng port in FreeBSD (sysutils/syslog-ng) and 3.5 is available as sysutils/sysog-ng-devel. With the latest bugfix release I consider 3.5 good enough to become the default. Before making the switch, I'd wait one more week and would like you to test the 3.5 port and share your experiences. Using 3.5 also has the additional benefit, that one can use the freshly introduced sysutils/syslog-ng-incubator port. The incubator adds many interesting modules to syslog-ng: https://github.com/balabit/syslog-ng-incubator , this port supports most of it, except for Riemann. Bye, -- Peter Czanik (CzP) <czanik(a)balabit.hu> BalaBit IT Security / syslog-ng upstream http://czanik.blogs.balabit.com/ https://twitter.com/PCzanik

1 1

GSoC2014: Information for students
by Gergely Nagy 20 Mar '14

20 Mar '14

Hi! I believe I contacted everyone who submitted a proposal by now, but I know there are some, who are still writing it, and I'd like to reach out to everyone interested and involved: since we would like to allow our students to improve their proposal past the application deadline, and would also like to re-use the same page for progress reports and other informations (such as repo urls) during the programme itself, I'd like to ask all of you, to write your proposals on our GitHub wiki[1], and on Melange, the content of the proposal should be a link to the created wiki page. [1]: https://github.com/balabit/syslog-ng/wiki/GSoC2014-Proposals If you already wrote a proposal on Melange, copying it over should be simple, as both Melange and GitHub accept Markdown. But if you need help, and the linked page above isn't clear or complete enough, please let me know, and I'll do my best to help. Apologies for the inconvenience! Also, all students who submitted a proposal will receive a small, topic-related task this afternoon, which we would like you to complete. It shouldn't take more than an afternoon, I hope, but I'll be available on all the usual channels (IRC, twitter, email, etc) to answer any questions. Cheers, and thanks in advance! -- |8]

1 0

GSoC: Proposal Submitted
by Rajul Srivastava 17 Mar '14

17 Mar '14

Hello, I have submitted my proposal regarding ' Type Hinting from the start' for GSoC 2014. It is at the link https://www.google-melange.com/gsoc/proposal/review/student/google/gsoc2014… I shall request you all to review it and make some suggestions and comments on how I can improve it. I am particularly worried about the timeline, if I have not given ample time to tasks. Any suggestion or criticism is more that welcome. Regards, Rajul -- Rajul Final year Undergraduate Student, Department of Chemistry, Indian Institute of Technology Kharagpur

2 3

Background/foreground problem
by Hardcastle, Iain (CBC - Hamilton) 17 Mar '14

17 Mar '14

Hi all, Odd one I've come across, cant find any clues on the web so figured I'd post here. Trying to get syslog-ng integrated into the Observium NMS package. Not sure whether the problem lies with syslog-ng, PHP or the sucking of data into Observium. Let me explain: With syslog-ng started in foreground mode 'syslog-ng -Fevd' everything works as expected. I see syslog packets coming in via tcpdump, I see the log flash up on the terminal and it correctly routes into the right part of the Observium front end. With syslog-ng started as a daemon (from boot - the desired config setup - or with 'service syslog-ng start'), nothing gets to Observium. I can still see the syslog packets in tcpdump obviously but I'm blind as to what happens to the messages after that. One thing I have noticed is that when running syslog-ng in foreground mode, I see "Starting destination program; cmdline='/opt/observium/syslog.php'" in the startup messages and a corresponding process in the 'ps -lax' output "php /opt/observium/syslog.php" With syslog-ng started as a daemon, I do not see this php process in the ps output.. Bit stuck as to where to look next. Has anyone seen this behavior before? Centos6.5 and standard packages and configs. System works great otherwise. Syslog-ng config file in its entirety below: options { chain_hostnames(0); flush_lines (0); time_reopen (10); log_fifo_size (1000); long_hostnames (off); use_dns (no); use_fqdn (no); create_dirs (no); keep_hostname (1); }; source s_net { udp(); }; destination d_observium { program("/opt/observium/syslog.php" template ("$HOST||$FACILITY||$PRIORITY||$LEVEL||$TAG||$YEAR-$MONTH-$DAY $HOUR:$MIN:$SEC||$MS G||$PROGRAM\n") template-escape(yes)); }; log { source(s_net); destination(d_observium); }; Any ideas? iain Deloitte refers to one or more of Deloitte Touche Tohmatsu Limited, a UK private company limited by guarantee, and its network of member firms, each of which is a legally separate and independent entity. Please see http://www.deloitte.com/about for a detailed description of the legal structure of Deloitte Touche Tohmatsu Limited and its member firms.

2 2