- syslog-ng - lists.balabit.hu

[Bug 269] New: some code doesn't follow glib recomendations
by bugzilla＠bugzilla.balabit.com 04 Feb '14

04 Feb '14

https://bugzilla.balabit.com/show_bug.cgi?id=269 Summary: some code doesn't follow glib recomendations Product: syslog-ng Version: 3.5.x Platform: PC OS/Version: Linux Status: NEW Severity: major Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: iippolitov(a)gmail.com Type of the Report: bug Estimated Hours: 0.0 I'm using rhel 5.9 and syslog-ng 3.5.3. tests/unit/test_nvtable fails on line 69. It is a know bug: https://bugzilla.redhat.com/show_bug.cgi?id=714409#c4 . As per https://bugzilla.redhat.com/show_bug.cgi?id=716447 : >From the glib2 documentation: ... Note that neither keys nor values are copied when inserted into the GHashTable, so they must exist for the lifetime of the GHashTable. This means that the use of static strings is OK, but temporary strings (i.e. those created in buffers and those returned by GTK+ widgets) should be copied with g_strdup() before being inserted. If keys or values are dynamically allocated, you must be careful to ensure that they are freed when they are removed from the GHashTable, and also when they are overwritten by new insertions into the GHashTable. It is also not advisable to mix static strings and dynamically-allocated strings in a GHashTable, because it then becomes difficult to determine whether the string should be freed. ... The code in lib/nv_table.c uses a pointer to a string that is changed during a program execution to insert values into GHashTable. This causes some unit tests to fail at least in rhel5. nv_registry_alloc_handle was obviously fixed to g_strdup() string from function parameters, while nv_registry_add_alias still uses a pointer the function gets as a parameter: Here is a workaround: void nv_registry_add_alias(NVRegistry *self, NVHandle handle, const gchar *alias) { g_static_mutex_lock(&nv_registry_lock); - g_hash_table_insert(self->name_map, (gchar*)alias, GUINT_TO_POINTER((glong) handle)); + gchar * st_alias = g_strdup(alias); + g_hash_table_insert(self->name_map, st_alias, GUINT_TO_POINTER((glong) handle)); g_static_mutex_unlock(&nv_registry_lock); } AFAIK, g_hash_table_insert is used widely around syslog-ng code. Could someone please look it through and ensure that any value added into gHashTable is not changed during hashtable lifetime? Feel free to contact me via email or gtalk to find out any details I might miss. Thanks in advance. -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 7

Help with long log messages
by Edward Gleeck 03 Feb '14

03 Feb '14

Syslog-ng seems to only allow up to 8192 bytes of log_msg_size and after which it splits up the log message into multiple log messages. Setting this up in the global options or on the source option to use more than 8192 does not seem to work. I was wondering if there are other options that I need to put in order that very long logs aren't split up. I realize this might be a very rare case on the need for long log messages, but the application logging was designed poorly and we need this functionality currently while the logging is being fixed. In looking at the source code, it seems that the log_msg_size is stored as a gint type, which in my recollection allows me to store up to +32787 right? If the max I could put is 8192, then I guess I'll have to come up with something else to process the split logs, otherwise any help would be appreciated.

2 1

[Bug 267] New: $YEAR on event in RFC 3164 format
by bugzilla＠bugzilla.balabit.com 31 Jan '14

31 Jan '14

https://bugzilla.balabit.com/show_bug.cgi?id=267 Summary: $YEAR on event in RFC 3164 format Product: syslog-ng Version: 3.3.x Platform: PC OS/Version: Linux Status: NEW Severity: normal Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: dtgarry(a)hotmail.com Type of the Report: --- Estimated Hours: 0.0 I am using syslog-ng 3.3.4 Revision: /3.3.4.dfsg-2ubuntu1 Revision: /3.3.4.dfsg-2ubuntu1 I have a problem on the macro $YEAR when the event is in RFC 3164 format sometime the $YEAR extract the correct value of year, but not always. I test this problem on a computer with the current date, 9 january 2014, sending some event with these dates: - Dec 31 23:59:59, syslog-ng return the 2013 year; - Jan 1 00:00:01, syslog-ng return the 2014 year; - Feb 1 00:00:01, syslog-ng return the 2014 year; - Mar 1 00:00:01, syslog-ng return the 2013 year; how syslog-ng extract the year from an event when there is no year? Is it possible to configure correctly the syslog-ng? for testing I have configured syslog-ng to call my program "mytest" like this: destination d_tst { program("/etc/syslog-ng/sbin/mytest" template("<$PRI>$DATE $HOST $PROGRAM $MSGONLY $HOST $FACILITY $PRIORITY $LEVEL $TAG $YEAR-$MONTH-$DAY $HOUR:$MIN:$SEC $PROGRAM\n") template-escape(yes) ); }; on mytest program I log what syslog-ng has parsed; today, that is the 9th january 2014, I have sent to syslog-ng this string: <131>Dec 31 23:59:59 hostTest processTest some test on mytest I will receive: <131>Dec 31 23:59:59 hostTest processTest some test hostTest local0 err err 83 2013-12-31 23:59:59 processTest note the year is 2013 If I send this: <131>Jan 1 00:00:01 hostTest processTest some test I will receive: <131>Jan 1 00:00:01 hostTest processTest some test hostTest local0 err err 83 2014-01-01 00:00:01 processTest now the year is 2014 But I send this event that is in the future: <131>Feb 1 00:00:01 hostTest processTest some test I will receive: <131>Feb 1 00:00:01 hostTest processTest some test hostTest local0 err err 83 2014-02-01 00:00:01 processTest the year is 2014 If I send an event of march: <131>Mar 1 00:00:01 hostTest processTest some test I will receive: <131>Mar 1 00:00:01 hostTest processTest some test hostTest local0 err err 83 2013-03-01 00:00:01 processTest where the year is 2013 thank you. -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 6

Re: [syslog-ng] Issue with pdbtool patterndb and percent symbols
by Mark Shetka 29 Jan '14

29 Jan '14

My Zsh shell was somehow interfering with pdbtool and the % symbols. I reran with Bash and the patterns match just fine, so I'm all good. Thanks! -- Mark Shetka Information Technology Systems & Services University of Minnesota - Duluth (218) 726-7682 On Wed, Jan 29, 2014 at 8:40 AM, Mark Shetka <mshetka(a)d.umn.edu> wrote: > I am setting up some patterns to parse Cisco syslog messages. I noticed > that pdbtool will not complete if I have a "%F" anywhere in the string. > > Example log message: > %FWSM-1-109006: Authentication failed for user 'test' from > 131.212.1.1/43250 to 10.1.1.1/22 on interface management > > This does not complete: > pdbtool match -p cisco.xml -M "%FWSM-1-109006: Authentication failed for > user 'test' from 131.212.1.1/43250 to 10.1.1.1/22 on interface management" > > Nor does simply %F: > pdbtool match -p cisco.xml -M "%F" > > It is fine without the %: > pdbtool match -p cisco.xml -M "FWSM-1-109006: Authentication failed for > user 'test' from 131.212.1.1/43250 to 10.1.1.1/22 on interface management" > > MESSAGE=FWSM-1-109006: Authentication failed for user 'test' from > 131.212.1.1/43250 to 10.1.1.1/22 on interface management > .classifier.class=login > .classifier.rule_id=5cfbcb23-cfe4-4120-85c1-918df65c0edc > usracct.username=test > usracct.device=131.212.1.1 > usracct.service=22 > usracct.type=login > usracct.sessionid= > usracct.application= > secevt.verdict=REJECT > TAGS=.classifier.login,usracct,secevt > > It also seems to have issues with "%S", although not quite in the same > way. Any ideas what could be causing this? > > Mark > > > -- > Mark Shetka > Information Technology Systems & Services > University of Minnesota - Duluth > (218) 726-7682 >

1 0

Issue with pdbtool patterndb and percent symbols
by Mark Shetka 29 Jan '14

29 Jan '14

I am setting up some patterns to parse Cisco syslog messages. I noticed that pdbtool will not complete if I have a "%F" anywhere in the string. Example log message: %FWSM-1-109006: Authentication failed for user 'test' from 131.212.1.1/43250to 10.1.1.1/22 on interface management This does not complete: pdbtool match -p cisco.xml -M "%FWSM-1-109006: Authentication failed for user 'test' from 131.212.1.1/43250 to 10.1.1.1/22 on interface management" Nor does simply %F: pdbtool match -p cisco.xml -M "%F" It is fine without the %: pdbtool match -p cisco.xml -M "FWSM-1-109006: Authentication failed for user 'test' from 131.212.1.1/43250 to 10.1.1.1/22 on interface management" MESSAGE=FWSM-1-109006: Authentication failed for user 'test' from 131.212.1.1/43250 to 10.1.1.1/22 on interface management .classifier.class=login .classifier.rule_id=5cfbcb23-cfe4-4120-85c1-918df65c0edc usracct.username=test usracct.device=131.212.1.1 usracct.service=22 usracct.type=login usracct.sessionid= usracct.application= secevt.verdict=REJECT TAGS=.classifier.login,usracct,secevt It also seems to have issues with "%S", although not quite in the same way. Any ideas what could be causing this? Mark -- Mark Shetka Information Technology Systems & Services University of Minnesota - Duluth (218) 726-7682

1 0

[Bug 268] New: filter on tag does partial match
by bugzilla＠bugzilla.balabit.com 27 Jan '14

27 Jan '14

https://bugzilla.balabit.com/show_bug.cgi?id=268 Summary: filter on tag does partial match Product: syslog-ng Version: 3.4.x Platform: PC OS/Version: Linux Status: NEW Severity: normal Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: erempel(a)uvic.ca Type of the Report: --- Estimated Hours: 0.0 According to the documentation ------- tags() Synopsis: tag Description: Select messages labeled with the specified tag. Every message automatically has the tag of its source in .source.<id_of_the_source_statement> format. This option is available only in syslog-ng 3.1 and later. ------------ I interpret this to mean that an exact and full match of the tag is required. I have a tag on a message of "collection.required" and a filter filter f_collection { tags("collection"); }; this filter matches the messages with the tag "collection.required" which I think is a bug. I expect this filter on only match tags that exactly match "collection". -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 4

syslog-ng and OS library path?
by Matt Zagrabelny 25 Jan '14

25 Jan '14

Hello syslog-ng users! I might be getting a little out of the scope of syslog-ng usage here, but I thought I'd start here and work my way elsewhere. I've got a Debian unstable workstation that I do some syslog-ng testing on. I recently started using the sql destination driver and am hitting an issue on my unstable workstation, but not on a Debian "old-stable" squeeze system. # syslog-ng -F -e Log pattern database reloaded; file='/var/lib/syslog-ng/patterndb.xml', version='3', pub_date='2013-06-27' Unable to initialize database access (DBI); rc='-1', error='No such file or directory (2)' Error initializing message pipeline; Which I run under strace and get: openat(AT_FDCWD, "/usr/lib/x86_64-linux-gnu/dbd", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = -1 ENOENT (No such file or directory) The location looks okay for a Debian multiarch system, but according to dpkg the library isn't under the multiarch library path: $ dpkg -L libdbd-pgsql | grep x86_64 $ dpkg -L libdbd-pgsql | grep '^/usr/lib' /usr/lib /usr/lib/dbd /usr/lib/dbd/libdbdpgsql.la /usr/lib/dbd/libdbdpgsql.so I am under the impression that /usr/lib is still searched by the system loader. Why is syslog-ng not able to find the library file. What am I missing? Thanks for any suggestions or hints. -mz

3 3

[ANN]: Incubator news, Debian/Ubuntu packages
by Gergely Nagy 24 Jan '14

24 Jan '14

Hi! I have recently released version 0.1.1 of the syslog-ng Incubator project[1], which includes a brand new destination: Lua. With this, you can write destination plugins without having to touch C, or compile anything. It does have its limitations and issues still, but it is usable to the point that an example ElasticSearch plugin is included with the sources. I also made Debian and Ubuntu packages[2] out of the new release, which includes all modules (the new Lua one, and the ones I described on my blog[3] some time ago). However, unlike the syslog-ng packages, these are only available for Ubuntu Precise and newer, or Debian Wheezy and newer. Ubuntu Lucid and Debian Squeeze are not supported, and will not be. If you're using syslog-ng 3.5 packages, the new modules will automatically appear, as they are put into the same repository (they will not be automatically installed, however, not even when using the syslog-ng meta-package). Also, while I was there, I disabled Ubuntu Raring in the repository: it will be reaching its end of life this month, so no new updates will be made (and the Incubator is not available for it, either). It will stay in the repository for a while, though. [1]: https://github.com/balabit/syslog-ng-incubator [2]: http://asylum.madhouse-project.org/projects/debian/ [3]: http://asylum.madhouse-project.org/blog/2013/12/29/the-incubator/ -- |8]

1 0

Re: [syslog-ng] how to use macros from patterndb in syslog-ng.conf
by Jim Hendrick 10 Jan '14

10 Jan '14

Thanks. Seems obvious now. The pdbtool test *was* parsing but syslog-ng was not... Again *doh* :-) Sent from my Verizon Wireless 4G LTE Smartphone -------- Original message -------- From: Evan Rempel <erempel(a)uvic.ca> Date: 01/10/2014 2:03 PM (GMT-05:00) To: syslog-ng(a)lists.balabit.hu Subject: Re: [syslog-ng] how to use macros from patterndb in syslog-ng.conf With no-parse you will not get a program name, which will prevent the pattern from matching in patterndb Evan. On 01/10/2014 10:59 AM, jrhendri(a)roadrunner.com wrote: > DOH!!! > quoth the bard Homer Simpson... > > in my testing, I was appending syslog messages to "testfile" like this: > head -10 user.info.2014.01.08 > testfile > > and using "flags(no-parse)" on that file source which looks like this: > > [n0142566@VDDP13E-0D6B677 ~]$ cat testfile > Jan 8 20:00:00 10.192.225.12 RT_FLOW: RT_FLOW_SESSION_CREATE: session created 146.150.19.13/46188->10.178.128.38/11000 None 146.150.19.13/46188->10.178.128.38/11000 None None 6 3798 Int_NonProd_CORE_Liberty Int_NonProd_CORE_Secure 180869281 N/A(N/A) reth0.3931 UNKNOWN UNKNOWN UNKNOWN > Jan 8 20:00:00 10.192.225.12 RT_FLOW: RT_FLOW_SESSION_CLOSE: session closed TCP RST: 10.178.200.127/43833->10.181.72.46/2059 junos-tcp-any 10.178.200.127/43833->10.181.72.46/2059 None None 6 3384 Int_NonProd_CORE_Secure Int_NonProd_CORE_Liberty 180781521 2(100) 1(60) 3 UNKNOWN UNKNOWN N/A(N/A) reth0.3930 UNKNOWN > [n0142566@VDDP13E-0D6B677 ~]$ > > > > not quite sure how / why - but that was breaking this even though this worked: > > [n0142566@VDDP13E-0D6B677 ~]$ pdbtool match -f testfile -p ./log/juniper_db.xml > HOST=10.192.225.12 > MESSAGE=RT_FLOW_SESSION_CREATE: session created 146.150.19.13/46188->10.178.128.38/11000 None 146.150.19.13/46188->10.178.128.38/11000 None None 6 3798 Int_NonProd_CORE_Liberty Int_NonProd_CORE_Secure 180869281 N/A(N/A) reth0.3931 UNKNOWN UNKNOWN UNKNOWN > PROGRAM=RT_FLOW > LEGACY_MSGHDR=RT_FLOW: > .classifier.class=system > .classifier.rule_id=2a270520-2ff7-7048-a088-b03d5b3b5f7d > J.MSG.SRC=146.150.19.13 > J.MSG.SPORT=46188 > J.MSG.DST=10.178.128.38 > J.MSG.DPORT=11000 > TAGS=.classifier.system > > HOST=10.192.225.12 > MESSAGE=RT_FLOW_SESSION_CLOSE: session closed TCP RST: 10.178.200.127/43833->10.181.72.46/2059 junos-tcp-any 10.178.200.127/43833->10.181.72.46/2059 None None 6 3384 Int_NonProd_CORE_Secure Int_NonProd_CORE_Liberty 180781521 2(100) 1(60) 3 UNKNOWN UNKNOWN N/A(N/A) reth0.3930 UNKNOWN > PROGRAM=RT_FLOW > LEGACY_MSGHDR=RT_FLOW: > .classifier.class=system > .classifier.rule_id=2a270520-2ff7-7048-a088-b03d5b3b5f7d > J.MSG.CLOSE.REASON=TCP RST > J.MSG.SRC=10.178.200.127 > J.MSG.SPORT=43833 > J.MSG.DST=10.181.72.46 > J.MSG.DPORT=2059 > TAGS=.classifier.system > > > After I took out flags(no-parse) on the file source, it works as (I) expected ... > > I am still curious what I overlooked - misunderstood. But it appears to be related to how the messages were parsed on the source end. > > Thanks again! > Jim > > > > > ---- jrhendri(a)roadrunner.com wrote: >> this seems like a simple thing to do, but a few days of searching and some hours of testing has not shown me the answer. >> >> Essentially I want to parse log events and output select fields, so I have created a basic patterndb xml file. >> >> <patterndb version='3' pub_date='2014-01-09'> >> <ruleset name='patternize' id='6cb77f11-6c9b-ee4e-9f62-b97224d4384c'> >> <rules> >> <rule id='2a270520-2ff7-7048-a088-b03d5b3b5f7d' class='system' provider='patternize'> >>  >> <patterns> >> <pattern>RT_FLOW_SESSION_CLOSE: session closed @ESTRING:J.MSG.CLOSE.REASON::@ @IPv4:J.MSG.SRC@/@NUMBER:J.MSG.SPORT@->@IPv4:J.MSG.DST@/@NUMBER:J.MSG.DPORT@ </pattern> >> <pattern>RT_FLOW_SESSION_CREATE: session created @IPv4:J.MSG.SRC@/@NUMBER:J.MSG.SPORT@->@IPv4:J.MSG.DST@/@NUMBER:J.MSG.DPORT@ </pattern> >> </patterns> >> </rule> >> </rules> >> </ruleset> >> </patterndb> >> >> This tests fine using pdbtool on sample data in a file. (cut from a real syslog file of logs) >> Inside syslog-ng.conf I want to use the parsed values as a template: >> source s_testfile { >> file("/home/n0142566/testfile" >> flags(no-parse) ); >> }; >> >> filter f_juniper_session_create { >> match("RT_FLOW_SESSION_CREATE" value( "MESSAGE" ) ); >> }; >> filter f_juniper_session_close { >> match("RT_FLOW_SESSION_CLOSE" value( "MESSAGE" ) ); >> }; >> >> parser juniper_db { >> db-parser ( >> file("/home/n0142566/log/juniper_db.xml") >> ); >> }; >> >> destination d_local_create { >> file("/home/n0142566/log/messages-create-$HOST" >> template("${J.MSG.SRC}, ${J.MSG.SPORT}, ${J.MSG.DST}, ${J.MSG.DPORT}\n") ); >> }; >> >> destination d_local_close { >> file("/home/n0142566/log/messages-close-$HOST" >> #!# template("${J.MSG.SRC}, ${J.MSG.SPORT}, ${J.MSG.DST}, ${J.MSG.DPORT}, ${J.MSG.CLOSE.REASON}\n") ); >> template("${J.MSG}\n") ); >> }; >> >> >> log { >> source(s_testfile); >> filter(f_juniper_session_create); >> parser(juniper_db); >> destination(d_local_create); >> }; >> log { >> source(s_testfile); >> filter(f_juniper_session_close); >> parser(juniper_db); >> destination(d_local_close); >> }; >> >> >> But when I run syslog-ng and append events to "testfile" the output is simply comma separated blanks :-( >> >> I am quite sure I am missing something *extremely* basic - but at this point I thought I would ask clearer minds for help!! >> >> Thanks, ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq

1 0

how to use macros from patterndb in syslog-ng.conf
by jrhendri＠roadrunner.com 10 Jan '14

10 Jan '14

this seems like a simple thing to do, but a few days of searching and some hours of testing has not shown me the answer. Essentially I want to parse log events and output select fields, so I have created a basic patterndb xml file. <patterndb version='3' pub_date='2014-01-09'> <ruleset name='patternize' id='6cb77f11-6c9b-ee4e-9f62-b97224d4384c'> <rules> <rule id='2a270520-2ff7-7048-a088-b03d5b3b5f7d' class='system' provider='patternize'>  <patterns> <pattern>RT_FLOW_SESSION_CLOSE: session closed @ESTRING:J.MSG.CLOSE.REASON::@ @IPv4:J.MSG.SRC@/@NUMBER:J.MSG.SPORT@->@IPv4:J.MSG.DST@/@NUMBER:J.MSG.DPORT@ </pattern> <pattern>RT_FLOW_SESSION_CREATE: session created @IPv4:J.MSG.SRC@/@NUMBER:J.MSG.SPORT@->@IPv4:J.MSG.DST@/@NUMBER:J.MSG.DPORT@ </pattern> </patterns> </rule> </rules> </ruleset> </patterndb> This tests fine using pdbtool on sample data in a file. (cut from a real syslog file of logs) Inside syslog-ng.conf I want to use the parsed values as a template: source s_testfile { file("/home/n0142566/testfile" flags(no-parse) ); }; filter f_juniper_session_create { match("RT_FLOW_SESSION_CREATE" value( "MESSAGE" ) ); }; filter f_juniper_session_close { match("RT_FLOW_SESSION_CLOSE" value( "MESSAGE" ) ); }; parser juniper_db { db-parser ( file("/home/n0142566/log/juniper_db.xml") ); }; destination d_local_create { file("/home/n0142566/log/messages-create-$HOST" template("${J.MSG.SRC}, ${J.MSG.SPORT}, ${J.MSG.DST}, ${J.MSG.DPORT}\n") ); }; destination d_local_close { file("/home/n0142566/log/messages-close-$HOST" #!# template("${J.MSG.SRC}, ${J.MSG.SPORT}, ${J.MSG.DST}, ${J.MSG.DPORT}, ${J.MSG.CLOSE.REASON}\n") ); template("${J.MSG}\n") ); }; log { source(s_testfile); filter(f_juniper_session_create); parser(juniper_db); destination(d_local_create); }; log { source(s_testfile); filter(f_juniper_session_close); parser(juniper_db); destination(d_local_close); }; But when I run syslog-ng and append events to "testfile" the output is simply comma separated blanks :-( I am quite sure I am missing something *extremely* basic - but at this point I thought I would ask clearer minds for help!! Thanks, Jim

2 2