- syslog-ng - lists.balabit.hu

Introduction
by Rajul Srivastava 25 Feb '14

25 Feb '14

Hi all, My name in Rajul, and I am a final year undergraduate student at the Indian Institute of Technology Kharagpur. I wish to participate in Google Summer of Code 2014, and while going through the list of organisations, I came across The syslog-ng project". I am proficient with programming languages C/C++, Python, Java, Groovy, Ruby. I am very interested in the fields of Algorithms, Machine Learning, Complex Networks, Information Retrieval. I have always been interested in programming and in the past I have participated in Google Summer of Code 2012, with the organisation Network Time Foundation,working on the project "improving the Logging/Debugging System of Network Time Protocol Software". I have also interned in the Global Technology division of Barclays, during the summers of 2013, working with the Market Risk IT team. Besides I have worked on a few Research projects in the fields of Computational Finance, Complex Networks, and Computational Chemistry. I am currently working on my Thesis project in the field of Computational Sciences on a project titled "Network Analysis of Chemical Reactions". I have had courses in the fields of Programming and Data Structures, Complex Networks, Distributed Systems, Algorithms, Operations Research in the past. I have gone through the list of project ideas and I found most of the project ideas very interesting. Although I find all the projects listed worth a while, I am particularly interested in the "Configuration Visualization" and "Type Hinting from the start" projects. I suppose that my programming background is suitable for these projects. I shall be grateful if anyone can help me and give me reference to the literature that I may use and also shed some light on how I can go about making a successful proposal. Thanks!! Best Regards, Rajul -- Rajul Final year Undergraduate Student, Indian Institute of Technology Kharagpur

1 0

Per-destination log_msg_size?
by David Schweikert 21 Feb '14

21 Feb '14

Hi, We have some setups where we want to have a large log_msg_size, but need to send some logs to an external party which only can handle standard BSD syslog messages with a maximum size of 1024 bytes. Question: would it be possible to add a per-destination log-msg-size to truncate the messages on a per-destination basis? I think that it would be very useful. Something like: destination d_remote { udp("192.168.65.10" port(514) log_msg_size(1024)); }; Do you find it a good idea and is there any chance that this might be implemented? Thanks David

5 7

Re: [syslog-ng] syslog-ng Digest, Vol 106, Issue 12
by Terry Slattery 18 Feb '14

18 Feb '14

Why not stand up a test server and add it to the logging configs of the non-working and a couple of working switches? It could be a VM, since it won't have much load. You could also capture the log messages and play them back at the test server without impacting your production server or data archives. -tcs On 2/18/14 3:12 PM, syslog-ng-request(a)lists.balabit.hu wrote: > Date: Tue, 18 Feb 2014 15:12:50 -0500 > From: Chris Moody<chris(a)node-nine.com> > Subject: [syslog-ng] need help debugging some network received logs > that aren't writing to files > To: Syslog-ng users' and developers' mailing list > <syslog-ng(a)lists.balabit.hu> > Message-ID:<5303BEC2.9010405(a)node-nine.com> > Content-Type: text/plain; charset=ISO-8859-1; format=flowed > > Hello. > > First off, thanks a __TON__ for syslog-ng. I've sworn by this awesome > code for years now. I've built all sorts of logging infrastructure with > it. > > I seem to have hit on something though that's got me scratching my head > and lacking for explanation. Perhaps I've just been staring at it and > debugging it too long and am missing something obvious. -- Terry Slattery CCIE# 1026

1 0

log path with two destinations - flow control problem
by Tomáš Novosad 18 Feb '14

18 Feb '14

Hello to all i'm trying to set syslog to store sendmaill messages into PostgreSQL tables. One simple table (tableA) which stores all messages, and another table (tableB) which stores only some relevant iformation. All messages are parsed in my parser, then logged into tableA, then checked against the filter, an those messages which pass are logged into tableB. it all looks like this: log { source(pipe_load); filter(f_sendmail); #-- basic filter for sendmail program parser(maillog); #-- parse message details destination(maillog_base); #-- store message in tableA - basic info only filter(f_sendmail_from_or_to); #-- filter only messages which have "from" or "to" address defined destination(maillog_detail); #-- strore the deatiled info into tableB flags(flow-control); } filterinhg itself works fine, but i have problem with flow-control Messages are send from file (old archived maillog) into syslog pipe source. Problemm is, that alhough flow-control is enabled, messages get droped on the maillog_base destination. Im doing "cat old_maillog > syslog-pile" where old_maillog has approx 1 800 000 records, but only about 400 000 records gets logged into tableA. Flow control does not seem to work in this scenario. In case when i comment-out lines with filter(f_sendmail_from_or_to) and destination(maillog_detail) effectively disabling logging into the 2nd table, all messages from input file are processed and stored into tableA. The flow control works fine in this case. Is there something wrong with such filtering and storing only some messages in the 2nd destination? I also thought about having two different log path for each table, but in that case, the parsing of the message should be done twice, which i want to avoid. My question is, if is it OK to use filtering like in my example, - filter - than destination - then filter again - then another destination + flow control Many thanks in advance -- Tomáš Novosad LinuxBox.cz, s.r.o. 28. října 168, 709 00 Ostrava tel.: +420 591 166 221 mobil: +420 737 238 655 email: tomas.novosad(a)linuxbox.cz jabber: novosad(a)linuxbox.cz www.linuxbox.cz mobil servis: +420 737 238 656 email servis: servis(a)linuxbox.cz

3 3

source plugin udp not found
by Costa Farber 16 Feb '14

16 Feb '14

Hello, I have compiled syslog-ng ver 3.4.3 from source. syslog-ng -V syslog-ng 3.4.3 Installer-Version: 3.4.3 Revision: ssh+git://algernon@git.balabit /var/scm/git/syslog-ng/syslog-ng-ose--mainline--3.4#no_branch#64d670f3cbfb90769f3c7f0fdd9c70bb9136ec5b Compile-Date: Feb 10 2014 15:39:36 Available-Modules: Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-IPv6: on Enable-Spoof-Source: off Enable-TCP-Wrapper: off Enable-Linux-Caps: off Enable-Pcre: on When I run it: syslog-ng --cfgfile /opt/wix-log-service/configs/wix-log-service.conf I get: Error parsing source, source plugin udp not found in /opt/wix-log-service/configs/wix-log-service.conf at line 66, column 9: udp(port(514) so_rcvbuf(33554430)); What is wrong? Regards, Constantin -

2 3

Adding syslog-ng OSE 3.5.1 to buildroot
by Tamayo, Andres 16 Feb '14

16 Feb '14

Hello, I am working on adding syslog-ng OSE 3.5.1 to a cross-compiled image using buildroot. I was wondering if anybody knows if there is a repository where I can get the Config.in and syslog-ng.mk files needed to configure buildroot. Thank you for your help.

2 1

Multiple rewrite conditions under 3.4.x
by C. L. Martinez 14 Feb '14

14 Feb '14

Hi all, Is it possible to add multiple rewrite conditions in syslog-ng 3.4.x like this: rewrite r_rewrite_set{ set("myhost1", value("HOST") condition(program("myapplication1"))); set("myhost2", value("HOST") condition(program("myapplication2"))); set("myhost3", value("HOST") condition(program("myapplication3"))); set("myhost4", value("HOST") condition(program("myapplication4"))); }; ?? Is this a good option or maybe a performance penalty?? Thanks.

3 4

Re: [syslog-ng] [RFC] procfile Dezso Bagi
by Balazs Scheidler 14 Feb '14

14 Feb '14

Algernon is working on the debian subdir. Thanks for the feedback. Hope your experience is going to be bugfree :) On Feb 11, 2014 3:52 PM, "Valentijn Sessink" <valentyn(a)blub.net> wrote: > Hi, > > Finally found some time to test this. My setup is rather simple (I > think), the config-file now says: > destination d_syslogblock { pseudofile > ("/proc/net/xt_recent/syslogblock" template("+${usracct.device}\n")); > file("/var/log/syslogblock"); file("/tmp/blocker" > template("+${usracct.device}\n")); }; > > The "blocker" file is just for testing purposes. and "syslogblock" is to > log all evil to a regular log file as well. > > I got it to work rather easily and will report my findings. > > In the mean time, a question: is there anyone who guards the debian/ > directory inside the source tree? There seem to be some (mostly minor) > issues, and as I'm fixing them locally in order to get syslog-ng up and > running, I may as well send them upstream. If anyone cares ;-) that is. > > Best regards, > > Valentijn > > On 12/30/13 19:16, Balazs Scheidler wrote: > > I have now submitted a pull request to merge the "pseudofile" > > destination, I've now skipped the support for destination file > > templates, but that can be added later on when a more concrete usecase > > shows up. It already solves Valentijn's use-case, so it is useful as it > is. > > ______________________________________________________________________________ > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng > Documentation: > http://www.balabit.com/support/documentation/?product=syslog-ng > FAQ: http://www.balabit.com/wiki/syslog-ng-faq > >

1 0

insider 2014-02: FIPS; GSoC; Incubator; Conferences;
by Peter Czanik 13 Feb '14

13 Feb '14

Dear syslog-ng users, This is the 31th issue of the syslog-ng Insider, a monthly newsletter that brings you syslog-ng related news. FEATURED NEWS FIPS ---- Certain departments of the US government and those conducting business on certain regulated markets are required to use tools utilizing cryptography certified according to the FIPS 140-2 standard to handle sensitive data. As logs often contain sensitive information, this regulation can apply to the log management system, too. Starting with the 5.0.4 release, we are offering a version of syslog-ng Premium Edition to customers that is built and shipped with the OpenSSL FIPS Object Module and as thus usable in environments where such validation is required. (For more information about the OpenSSL FIPS Object Module, see _http://openssl.com/fips/_.) This is a separate version and the original version is available that uses the standard OpenSSL library. It is currently available on major Linux platforms and has the same functionality as the non-compliant version apart from a few limitations listed in the documentation. Please visit _http://www.balabit.com/lp/fips_if for more information. Google Summer of Code (GsoC) ---------------------------- Based on the success of last years syslog-ng Gsoc participation (in co-operation with the openSUSE project), this year we plan to participate in GsoC again with a number of syslog-ng related projects. The current list of ideas range from new sources and destination through under the hood improvements to integration with configuration management systems. The syslog-ng GsoC page is available at _https://github.com/balabit/syslog-ng/wiki/GSoC2014_ The ideas page is available at _https://github.com/balabit/syslog-ng/wiki/GSoC2014-Idea-&-Project-list_ We also created a central hub for syslog-ng OSE related material, as part of our preparations for Google Summer of Code and onwards: it is available at_http://syslog-ng.org/_ syslog-ng incubator updates --------------------------- The syslog-ng incubator is a collection of tools and modules which are not (yet) part of the official repository. Since last month there are many smaller changes and brand new features. The master branch goes now hand in hand with syslog-ng's master branch, which is currently 3.6 pre-alpha code. If you are working on syslog-ng 3.5, use the 3.5/master branch of syslog-ng-incubator. On the feature side the Lua destination was enhanced, a monitoring source was added, together with a Graphite output. Among the Lua examples one can also find a script to push logs into ElasticSearch. To try the new features, check out _https://github.com/balabit/syslog-ng-incubator_from git. You can read more about the Lua destination at _https://talien.blogs.balabit.com/2014/02/lua-the-undiscovered-country/_ syslog-ng at conferences ------------------------ In the past few weeks we visited three conferences. While there we talked about syslog-ng to many people, some during organized presentations, and many more between presentations and social events of the conferences. A look back at FOSDEM and Config Management Camp: _https://czanik.blogs.balabit.com/2014/02/fosdem-2014-and-config-management… The next confirmed event is: Open Source Data Center Conference, 8-10 April, Berlin, Germany: _http://www.netways.de/osdc/_, Giving a talk titled "Monitoring with syslog-ng, Riemann and Kibana" There will be two developers present, so if anyone attends, and has Incubator, Lua, Riemann, etc. related questions, we'll be happy to answer. SHORT NEWS: * Measuring log collection performance: _https://pzolee.blogs.balabit.com/2014/01/measuring-log-collection-performa… NEW RELEASES * syslog-ng 5.0.4: _https://lists.balabit.hu/pipermail/syslog-ng-announce/2014-February/000180.html_ Your feedback and news tips about the next issue is welcome at _documentation(a)balabit.com <mailto:documentation@balabit.com>_To read this newsletter on-line, visit: _http://insider.blogs.balabit.com/_ -- Peter Czanik (CzP) <czanik(a)balabit.hu> BalaBit IT Security / syslog-ng upstream http://czanik.blogs.balabit.com/ https://twitter.com/PCzanik

1 0

debian init files gone wrong? (Due to git reorg?)
by Valentijn Sessink 12 Feb '14

12 Feb '14

Hi, Building syslog-ng from source posed a few problems, one of them being debian/syslog-ng.init that missed a crucial option: It's function syslogng_start() in Git currently says: start-stop-daemon --start --quiet --exec "$SYSLOGNG" --pidfile "$PIDFILE" -- $SYSLOGNG_OPTS \ ... while the Debian source I viewed says: start-stop-daemon --start --quiet --exec "$SYSLOGNG" --pidfile "$PIDFILE" -- -p "$PIDFILE" $SYSLOGNG_OPTS \ As the Git repository tells the file is from 2007, I suspect the git reorganisation from november to have caused this - but I'm not sure. Also, the Debian init files seem to be much more up to date than this file - the file in Git still starts with the "skeleton" stuff. Best regards, Valentijn

2 3

[RFC] procfile destination
by Balazs Scheidler 11 Feb '14

11 Feb '14

Hi, Based on a request I have now created a new destination driver, called "procfile". A working PoC is available on this branch: https://github.com/balabit/syslog-ng/tree/3.6/f/procfile It can be used to change kernel settings dynamically, based on a trigger within syslog-ng. The difference between procfile() and file(): - no template in the name of the file - processing is performed immediately as read by the source, no queueing is performed (e.g. no loss is possible, but takes CPU time from the source) - opens and closes the destination file after each and every write, writes to offset zero e.g. it doesn't append further data. I'd be grateful if you could provide feedback around these questions or the feature in general: - name of the driver, right now it is called procfile as it was the original use-case, however it works with stuff in /dev too, and might work with simple named pipes. - right now the file must exist and is not created (e.g. it really requires an existing file); do we need that? - do we need append mode in some cases - whether we need append mode (if we think of general files, this would probably be needed) Thanks in advance, Bazsi

5 6

[Bug 269] New: some code doesn't follow glib recomendations
by bugzilla＠bugzilla.balabit.com 04 Feb '14

04 Feb '14

https://bugzilla.balabit.com/show_bug.cgi?id=269 Summary: some code doesn't follow glib recomendations Product: syslog-ng Version: 3.5.x Platform: PC OS/Version: Linux Status: NEW Severity: major Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: iippolitov(a)gmail.com Type of the Report: bug Estimated Hours: 0.0 I'm using rhel 5.9 and syslog-ng 3.5.3. tests/unit/test_nvtable fails on line 69. It is a know bug: https://bugzilla.redhat.com/show_bug.cgi?id=714409#c4 . As per https://bugzilla.redhat.com/show_bug.cgi?id=716447 : >From the glib2 documentation: ... Note that neither keys nor values are copied when inserted into the GHashTable, so they must exist for the lifetime of the GHashTable. This means that the use of static strings is OK, but temporary strings (i.e. those created in buffers and those returned by GTK+ widgets) should be copied with g_strdup() before being inserted. If keys or values are dynamically allocated, you must be careful to ensure that they are freed when they are removed from the GHashTable, and also when they are overwritten by new insertions into the GHashTable. It is also not advisable to mix static strings and dynamically-allocated strings in a GHashTable, because it then becomes difficult to determine whether the string should be freed. ... The code in lib/nv_table.c uses a pointer to a string that is changed during a program execution to insert values into GHashTable. This causes some unit tests to fail at least in rhel5. nv_registry_alloc_handle was obviously fixed to g_strdup() string from function parameters, while nv_registry_add_alias still uses a pointer the function gets as a parameter: Here is a workaround: void nv_registry_add_alias(NVRegistry *self, NVHandle handle, const gchar *alias) { g_static_mutex_lock(&nv_registry_lock); - g_hash_table_insert(self->name_map, (gchar*)alias, GUINT_TO_POINTER((glong) handle)); + gchar * st_alias = g_strdup(alias); + g_hash_table_insert(self->name_map, st_alias, GUINT_TO_POINTER((glong) handle)); g_static_mutex_unlock(&nv_registry_lock); } AFAIK, g_hash_table_insert is used widely around syslog-ng code. Could someone please look it through and ensure that any value added into gHashTable is not changed during hashtable lifetime? Feel free to contact me via email or gtalk to find out any details I might miss. Thanks in advance. -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 7

Help with long log messages
by Edward Gleeck 03 Feb '14

03 Feb '14

Syslog-ng seems to only allow up to 8192 bytes of log_msg_size and after which it splits up the log message into multiple log messages. Setting this up in the global options or on the source option to use more than 8192 does not seem to work. I was wondering if there are other options that I need to put in order that very long logs aren't split up. I realize this might be a very rare case on the need for long log messages, but the application logging was designed poorly and we need this functionality currently while the logging is being fixed. In looking at the source code, it seems that the log_msg_size is stored as a gint type, which in my recollection allows me to store up to +32787 right? If the max I could put is 8192, then I guess I'll have to come up with something else to process the split logs, otherwise any help would be appreciated.

2 1

[Bug 267] New: $YEAR on event in RFC 3164 format
by bugzilla＠bugzilla.balabit.com 31 Jan '14

31 Jan '14

https://bugzilla.balabit.com/show_bug.cgi?id=267 Summary: $YEAR on event in RFC 3164 format Product: syslog-ng Version: 3.3.x Platform: PC OS/Version: Linux Status: NEW Severity: normal Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: dtgarry(a)hotmail.com Type of the Report: --- Estimated Hours: 0.0 I am using syslog-ng 3.3.4 Revision: /3.3.4.dfsg-2ubuntu1 Revision: /3.3.4.dfsg-2ubuntu1 I have a problem on the macro $YEAR when the event is in RFC 3164 format sometime the $YEAR extract the correct value of year, but not always. I test this problem on a computer with the current date, 9 january 2014, sending some event with these dates: - Dec 31 23:59:59, syslog-ng return the 2013 year; - Jan 1 00:00:01, syslog-ng return the 2014 year; - Feb 1 00:00:01, syslog-ng return the 2014 year; - Mar 1 00:00:01, syslog-ng return the 2013 year; how syslog-ng extract the year from an event when there is no year? Is it possible to configure correctly the syslog-ng? for testing I have configured syslog-ng to call my program "mytest" like this: destination d_tst { program("/etc/syslog-ng/sbin/mytest" template("<$PRI>$DATE $HOST $PROGRAM $MSGONLY $HOST $FACILITY $PRIORITY $LEVEL $TAG $YEAR-$MONTH-$DAY $HOUR:$MIN:$SEC $PROGRAM\n") template-escape(yes) ); }; on mytest program I log what syslog-ng has parsed; today, that is the 9th january 2014, I have sent to syslog-ng this string: <131>Dec 31 23:59:59 hostTest processTest some test on mytest I will receive: <131>Dec 31 23:59:59 hostTest processTest some test hostTest local0 err err 83 2013-12-31 23:59:59 processTest note the year is 2013 If I send this: <131>Jan 1 00:00:01 hostTest processTest some test I will receive: <131>Jan 1 00:00:01 hostTest processTest some test hostTest local0 err err 83 2014-01-01 00:00:01 processTest now the year is 2014 But I send this event that is in the future: <131>Feb 1 00:00:01 hostTest processTest some test I will receive: <131>Feb 1 00:00:01 hostTest processTest some test hostTest local0 err err 83 2014-02-01 00:00:01 processTest the year is 2014 If I send an event of march: <131>Mar 1 00:00:01 hostTest processTest some test I will receive: <131>Mar 1 00:00:01 hostTest processTest some test hostTest local0 err err 83 2013-03-01 00:00:01 processTest where the year is 2013 thank you. -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 6

Re: [syslog-ng] Issue with pdbtool patterndb and percent symbols
by Mark Shetka 29 Jan '14

29 Jan '14

My Zsh shell was somehow interfering with pdbtool and the % symbols. I reran with Bash and the patterns match just fine, so I'm all good. Thanks! -- Mark Shetka Information Technology Systems & Services University of Minnesota - Duluth (218) 726-7682 On Wed, Jan 29, 2014 at 8:40 AM, Mark Shetka <mshetka(a)d.umn.edu> wrote: > I am setting up some patterns to parse Cisco syslog messages. I noticed > that pdbtool will not complete if I have a "%F" anywhere in the string. > > Example log message: > %FWSM-1-109006: Authentication failed for user 'test' from > 131.212.1.1/43250 to 10.1.1.1/22 on interface management > > This does not complete: > pdbtool match -p cisco.xml -M "%FWSM-1-109006: Authentication failed for > user 'test' from 131.212.1.1/43250 to 10.1.1.1/22 on interface management" > > Nor does simply %F: > pdbtool match -p cisco.xml -M "%F" > > It is fine without the %: > pdbtool match -p cisco.xml -M "FWSM-1-109006: Authentication failed for > user 'test' from 131.212.1.1/43250 to 10.1.1.1/22 on interface management" > > MESSAGE=FWSM-1-109006: Authentication failed for user 'test' from > 131.212.1.1/43250 to 10.1.1.1/22 on interface management > .classifier.class=login > .classifier.rule_id=5cfbcb23-cfe4-4120-85c1-918df65c0edc > usracct.username=test > usracct.device=131.212.1.1 > usracct.service=22 > usracct.type=login > usracct.sessionid= > usracct.application= > secevt.verdict=REJECT > TAGS=.classifier.login,usracct,secevt > > It also seems to have issues with "%S", although not quite in the same > way. Any ideas what could be causing this? > > Mark > > > -- > Mark Shetka > Information Technology Systems & Services > University of Minnesota - Duluth > (218) 726-7682 >

1 0

Issue with pdbtool patterndb and percent symbols
by Mark Shetka 29 Jan '14

29 Jan '14

I am setting up some patterns to parse Cisco syslog messages. I noticed that pdbtool will not complete if I have a "%F" anywhere in the string. Example log message: %FWSM-1-109006: Authentication failed for user 'test' from 131.212.1.1/43250to 10.1.1.1/22 on interface management This does not complete: pdbtool match -p cisco.xml -M "%FWSM-1-109006: Authentication failed for user 'test' from 131.212.1.1/43250 to 10.1.1.1/22 on interface management" Nor does simply %F: pdbtool match -p cisco.xml -M "%F" It is fine without the %: pdbtool match -p cisco.xml -M "FWSM-1-109006: Authentication failed for user 'test' from 131.212.1.1/43250 to 10.1.1.1/22 on interface management" MESSAGE=FWSM-1-109006: Authentication failed for user 'test' from 131.212.1.1/43250 to 10.1.1.1/22 on interface management .classifier.class=login .classifier.rule_id=5cfbcb23-cfe4-4120-85c1-918df65c0edc usracct.username=test usracct.device=131.212.1.1 usracct.service=22 usracct.type=login usracct.sessionid= usracct.application= secevt.verdict=REJECT TAGS=.classifier.login,usracct,secevt It also seems to have issues with "%S", although not quite in the same way. Any ideas what could be causing this? Mark -- Mark Shetka Information Technology Systems & Services University of Minnesota - Duluth (218) 726-7682

1 0

[Bug 268] New: filter on tag does partial match
by bugzilla＠bugzilla.balabit.com 27 Jan '14

27 Jan '14

https://bugzilla.balabit.com/show_bug.cgi?id=268 Summary: filter on tag does partial match Product: syslog-ng Version: 3.4.x Platform: PC OS/Version: Linux Status: NEW Severity: normal Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: erempel(a)uvic.ca Type of the Report: --- Estimated Hours: 0.0 According to the documentation ------- tags() Synopsis: tag Description: Select messages labeled with the specified tag. Every message automatically has the tag of its source in .source.<id_of_the_source_statement> format. This option is available only in syslog-ng 3.1 and later. ------------ I interpret this to mean that an exact and full match of the tag is required. I have a tag on a message of "collection.required" and a filter filter f_collection { tags("collection"); }; this filter matches the messages with the tag "collection.required" which I think is a bug. I expect this filter on only match tags that exactly match "collection". -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 4

syslog-ng and OS library path?
by Matt Zagrabelny 25 Jan '14

25 Jan '14

Hello syslog-ng users! I might be getting a little out of the scope of syslog-ng usage here, but I thought I'd start here and work my way elsewhere. I've got a Debian unstable workstation that I do some syslog-ng testing on. I recently started using the sql destination driver and am hitting an issue on my unstable workstation, but not on a Debian "old-stable" squeeze system. # syslog-ng -F -e Log pattern database reloaded; file='/var/lib/syslog-ng/patterndb.xml', version='3', pub_date='2013-06-27' Unable to initialize database access (DBI); rc='-1', error='No such file or directory (2)' Error initializing message pipeline; Which I run under strace and get: openat(AT_FDCWD, "/usr/lib/x86_64-linux-gnu/dbd", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = -1 ENOENT (No such file or directory) The location looks okay for a Debian multiarch system, but according to dpkg the library isn't under the multiarch library path: $ dpkg -L libdbd-pgsql | grep x86_64 $ dpkg -L libdbd-pgsql | grep '^/usr/lib' /usr/lib /usr/lib/dbd /usr/lib/dbd/libdbdpgsql.la /usr/lib/dbd/libdbdpgsql.so I am under the impression that /usr/lib is still searched by the system loader. Why is syslog-ng not able to find the library file. What am I missing? Thanks for any suggestions or hints. -mz

3 3

[ANN]: Incubator news, Debian/Ubuntu packages
by Gergely Nagy 24 Jan '14

24 Jan '14

Hi! I have recently released version 0.1.1 of the syslog-ng Incubator project[1], which includes a brand new destination: Lua. With this, you can write destination plugins without having to touch C, or compile anything. It does have its limitations and issues still, but it is usable to the point that an example ElasticSearch plugin is included with the sources. I also made Debian and Ubuntu packages[2] out of the new release, which includes all modules (the new Lua one, and the ones I described on my blog[3] some time ago). However, unlike the syslog-ng packages, these are only available for Ubuntu Precise and newer, or Debian Wheezy and newer. Ubuntu Lucid and Debian Squeeze are not supported, and will not be. If you're using syslog-ng 3.5 packages, the new modules will automatically appear, as they are put into the same repository (they will not be automatically installed, however, not even when using the syslog-ng meta-package). Also, while I was there, I disabled Ubuntu Raring in the repository: it will be reaching its end of life this month, so no new updates will be made (and the Incubator is not available for it, either). It will stay in the repository for a while, though. [1]: https://github.com/balabit/syslog-ng-incubator [2]: http://asylum.madhouse-project.org/projects/debian/ [3]: http://asylum.madhouse-project.org/blog/2013/12/29/the-incubator/ -- |8]

1 0

Re: [syslog-ng] how to use macros from patterndb in syslog-ng.conf
by Jim Hendrick 10 Jan '14

10 Jan '14

Thanks. Seems obvious now. The pdbtool test *was* parsing but syslog-ng was not... Again *doh* :-) Sent from my Verizon Wireless 4G LTE Smartphone -------- Original message -------- From: Evan Rempel <erempel(a)uvic.ca> Date: 01/10/2014 2:03 PM (GMT-05:00) To: syslog-ng(a)lists.balabit.hu Subject: Re: [syslog-ng] how to use macros from patterndb in syslog-ng.conf With no-parse you will not get a program name, which will prevent the pattern from matching in patterndb Evan. On 01/10/2014 10:59 AM, jrhendri(a)roadrunner.com wrote: > DOH!!! > quoth the bard Homer Simpson... > > in my testing, I was appending syslog messages to "testfile" like this: > head -10 user.info.2014.01.08 > testfile > > and using "flags(no-parse)" on that file source which looks like this: > > [n0142566@VDDP13E-0D6B677 ~]$ cat testfile > Jan 8 20:00:00 10.192.225.12 RT_FLOW: RT_FLOW_SESSION_CREATE: session created 146.150.19.13/46188->10.178.128.38/11000 None 146.150.19.13/46188->10.178.128.38/11000 None None 6 3798 Int_NonProd_CORE_Liberty Int_NonProd_CORE_Secure 180869281 N/A(N/A) reth0.3931 UNKNOWN UNKNOWN UNKNOWN > Jan 8 20:00:00 10.192.225.12 RT_FLOW: RT_FLOW_SESSION_CLOSE: session closed TCP RST: 10.178.200.127/43833->10.181.72.46/2059 junos-tcp-any 10.178.200.127/43833->10.181.72.46/2059 None None 6 3384 Int_NonProd_CORE_Secure Int_NonProd_CORE_Liberty 180781521 2(100) 1(60) 3 UNKNOWN UNKNOWN N/A(N/A) reth0.3930 UNKNOWN > [n0142566@VDDP13E-0D6B677 ~]$ > > > > not quite sure how / why - but that was breaking this even though this worked: > > [n0142566@VDDP13E-0D6B677 ~]$ pdbtool match -f testfile -p ./log/juniper_db.xml > HOST=10.192.225.12 > MESSAGE=RT_FLOW_SESSION_CREATE: session created 146.150.19.13/46188->10.178.128.38/11000 None 146.150.19.13/46188->10.178.128.38/11000 None None 6 3798 Int_NonProd_CORE_Liberty Int_NonProd_CORE_Secure 180869281 N/A(N/A) reth0.3931 UNKNOWN UNKNOWN UNKNOWN > PROGRAM=RT_FLOW > LEGACY_MSGHDR=RT_FLOW: > .classifier.class=system > .classifier.rule_id=2a270520-2ff7-7048-a088-b03d5b3b5f7d > J.MSG.SRC=146.150.19.13 > J.MSG.SPORT=46188 > J.MSG.DST=10.178.128.38 > J.MSG.DPORT=11000 > TAGS=.classifier.system > > HOST=10.192.225.12 > MESSAGE=RT_FLOW_SESSION_CLOSE: session closed TCP RST: 10.178.200.127/43833->10.181.72.46/2059 junos-tcp-any 10.178.200.127/43833->10.181.72.46/2059 None None 6 3384 Int_NonProd_CORE_Secure Int_NonProd_CORE_Liberty 180781521 2(100) 1(60) 3 UNKNOWN UNKNOWN N/A(N/A) reth0.3930 UNKNOWN > PROGRAM=RT_FLOW > LEGACY_MSGHDR=RT_FLOW: > .classifier.class=system > .classifier.rule_id=2a270520-2ff7-7048-a088-b03d5b3b5f7d > J.MSG.CLOSE.REASON=TCP RST > J.MSG.SRC=10.178.200.127 > J.MSG.SPORT=43833 > J.MSG.DST=10.181.72.46 > J.MSG.DPORT=2059 > TAGS=.classifier.system > > > After I took out flags(no-parse) on the file source, it works as (I) expected ... > > I am still curious what I overlooked - misunderstood. But it appears to be related to how the messages were parsed on the source end. > > Thanks again! > Jim > > > > > ---- jrhendri(a)roadrunner.com wrote: >> this seems like a simple thing to do, but a few days of searching and some hours of testing has not shown me the answer. >> >> Essentially I want to parse log events and output select fields, so I have created a basic patterndb xml file. >> >> <patterndb version='3' pub_date='2014-01-09'> >> <ruleset name='patternize' id='6cb77f11-6c9b-ee4e-9f62-b97224d4384c'> >> <rules> >> <rule id='2a270520-2ff7-7048-a088-b03d5b3b5f7d' class='system' provider='patternize'> >>  >> <patterns> >> <pattern>RT_FLOW_SESSION_CLOSE: session closed @ESTRING:J.MSG.CLOSE.REASON::@ @IPv4:J.MSG.SRC@/@NUMBER:J.MSG.SPORT@->@IPv4:J.MSG.DST@/@NUMBER:J.MSG.DPORT@ </pattern> >> <pattern>RT_FLOW_SESSION_CREATE: session created @IPv4:J.MSG.SRC@/@NUMBER:J.MSG.SPORT@->@IPv4:J.MSG.DST@/@NUMBER:J.MSG.DPORT@ </pattern> >> </patterns> >> </rule> >> </rules> >> </ruleset> >> </patterndb> >> >> This tests fine using pdbtool on sample data in a file. (cut from a real syslog file of logs) >> Inside syslog-ng.conf I want to use the parsed values as a template: >> source s_testfile { >> file("/home/n0142566/testfile" >> flags(no-parse) ); >> }; >> >> filter f_juniper_session_create { >> match("RT_FLOW_SESSION_CREATE" value( "MESSAGE" ) ); >> }; >> filter f_juniper_session_close { >> match("RT_FLOW_SESSION_CLOSE" value( "MESSAGE" ) ); >> }; >> >> parser juniper_db { >> db-parser ( >> file("/home/n0142566/log/juniper_db.xml") >> ); >> }; >> >> destination d_local_create { >> file("/home/n0142566/log/messages-create-$HOST" >> template("${J.MSG.SRC}, ${J.MSG.SPORT}, ${J.MSG.DST}, ${J.MSG.DPORT}\n") ); >> }; >> >> destination d_local_close { >> file("/home/n0142566/log/messages-close-$HOST" >> #!# template("${J.MSG.SRC}, ${J.MSG.SPORT}, ${J.MSG.DST}, ${J.MSG.DPORT}, ${J.MSG.CLOSE.REASON}\n") ); >> template("${J.MSG}\n") ); >> }; >> >> >> log { >> source(s_testfile); >> filter(f_juniper_session_create); >> parser(juniper_db); >> destination(d_local_create); >> }; >> log { >> source(s_testfile); >> filter(f_juniper_session_close); >> parser(juniper_db); >> destination(d_local_close); >> }; >> >> >> But when I run syslog-ng and append events to "testfile" the output is simply comma separated blanks :-( >> >> I am quite sure I am missing something *extremely* basic - but at this point I thought I would ask clearer minds for help!! >> >> Thanks, ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq

1 0

how to use macros from patterndb in syslog-ng.conf
by jrhendri＠roadrunner.com 10 Jan '14

10 Jan '14

this seems like a simple thing to do, but a few days of searching and some hours of testing has not shown me the answer. Essentially I want to parse log events and output select fields, so I have created a basic patterndb xml file. <patterndb version='3' pub_date='2014-01-09'> <ruleset name='patternize' id='6cb77f11-6c9b-ee4e-9f62-b97224d4384c'> <rules> <rule id='2a270520-2ff7-7048-a088-b03d5b3b5f7d' class='system' provider='patternize'>  <patterns> <pattern>RT_FLOW_SESSION_CLOSE: session closed @ESTRING:J.MSG.CLOSE.REASON::@ @IPv4:J.MSG.SRC@/@NUMBER:J.MSG.SPORT@->@IPv4:J.MSG.DST@/@NUMBER:J.MSG.DPORT@ </pattern> <pattern>RT_FLOW_SESSION_CREATE: session created @IPv4:J.MSG.SRC@/@NUMBER:J.MSG.SPORT@->@IPv4:J.MSG.DST@/@NUMBER:J.MSG.DPORT@ </pattern> </patterns> </rule> </rules> </ruleset> </patterndb> This tests fine using pdbtool on sample data in a file. (cut from a real syslog file of logs) Inside syslog-ng.conf I want to use the parsed values as a template: source s_testfile { file("/home/n0142566/testfile" flags(no-parse) ); }; filter f_juniper_session_create { match("RT_FLOW_SESSION_CREATE" value( "MESSAGE" ) ); }; filter f_juniper_session_close { match("RT_FLOW_SESSION_CLOSE" value( "MESSAGE" ) ); }; parser juniper_db { db-parser ( file("/home/n0142566/log/juniper_db.xml") ); }; destination d_local_create { file("/home/n0142566/log/messages-create-$HOST" template("${J.MSG.SRC}, ${J.MSG.SPORT}, ${J.MSG.DST}, ${J.MSG.DPORT}\n") ); }; destination d_local_close { file("/home/n0142566/log/messages-close-$HOST" #!# template("${J.MSG.SRC}, ${J.MSG.SPORT}, ${J.MSG.DST}, ${J.MSG.DPORT}, ${J.MSG.CLOSE.REASON}\n") ); template("${J.MSG}\n") ); }; log { source(s_testfile); filter(f_juniper_session_create); parser(juniper_db); destination(d_local_create); }; log { source(s_testfile); filter(f_juniper_session_close); parser(juniper_db); destination(d_local_close); }; But when I run syslog-ng and append events to "testfile" the output is simply comma separated blanks :-( I am quite sure I am missing something *extremely* basic - but at this point I thought I would ask clearer minds for help!! Thanks, Jim

2 2

insider 2014-01: Conferences; 3.6 pre-alpha ; incubator ; PCI-DSS
by Peter Czanik 09 Jan '14

09 Jan '14

Dear syslog-ng users, This is the 30th issue of the syslog-ng Insider, a monthly newsletter that brings you syslog-ng related news. FEATURED NEWS syslog-ng at conferences ------------------------ This year, we plan to present syslog-ng in many conferences, or just participate conferences where many syslog-ng users are expected to show up. If you want to meet someone from the syslog-ng team, come to one of our presentations or drop an e-mail so we can find each other. We are also very interested to hear suggestions, where syslog-ng should be presented! Here is a list of confirmed events, which will be updated regularly in later newsletters: FOSDEM, 1-2 February, Brussels, Belgium: https://fosdem.org/2014/, giving a talk titled: “Babelfish for DevOps: syslog-ng” Fedora, JBoss and RedHat developers conference, 7-9 February, Brno, Czech Republic: http://devconf.cz/, Participating only Open Source Data Center Conference, 8-10 April, Berlin, Germany: http://www.netways.de/osdc/, Giving a talk titled "Monitoring with syslog-ng, Riemann and Kibana" Check out syslog-ng 3.6 pre-alpha! ---------------------------------- While syslog-ng 3.6 still did not even reach an alpha release, it received already a lot of development. So, while it might still eat your logs for lunch, those who are interested in where syslog-ng is heading should check out syslog-ng 3.6 from git. Unlike previous syslog-ng versions, it is developed in a unified git repository without a version string attached: https://github.com/balabit/syslog-ng While most changes are under the hood, there are also some user visible features like pseudofile destination: https://github.com/balabit/syslog-ng/commit/be381a4e3f3eca0695f5976678ea26c… Nodejs support was also added tosyslog-ng: use the widespread winston logging API, and syslog-ng will process its JSON formatted messages: https://bazsi.blogs.balabit.com/2014/01/nodejs-support-in-syslog-ng/ And if you look at the stats at http://www.ohloh.net/p/syslog-ng you can see a healthy growth of the code base and in the number of contributors. Thank you for your support! syslog-ng incubator ------------------- The syslog-ng incubator is a collection of tools and modules which are not (yet) part of the official repository. It has some very interesting code in it, like a riemann or an RSS destination, but until now it was completely undocumented. Not any more: http://asylum.madhouse-project.org/blog/2013/12/29/the-incubator/ The Incubator also includes a Lua destination, which makes it possible to write simple destination drivers without a line of C. It is still a work in progress, but is an important step into writing modules in other languages. And to make your life easier packages are available in Debian testing, Ubuntu Trustyand for openSUSE in the 3^rd party repositories ( http://www.balabit.com/network-security/syslog-ng/opensource-logging-system… ). PCI DSS 3.0 Continues to Emphasize the Importance of Log Management ------------------------------------------------------------------- The Payment Card Industry Security Standards Council recently released the Data Security Standard 3.0, three years after the prior version. As one of the most important international data security standards, the latest release was eagerly awaited by IT security practitioners. Clarifications make up the bulk of the changes but the standards council changed most of the 12 major requirements to include modified or additional sub-requirements. With PCI DSS 3.0 the standards council has reiterated that log management is a critical part of security best practices. You can read more about it at https://jluby.blogs.balabit.com/2013/12/09/pci-dss-3-0-continues-to-emphasi… SHORT NEWS: * Testing syslog-ng PE on Windows was never so easy: Windows Quick Start Guide for syslog-ng Premium Edition: http://czanik.blogs.balabit.com/2013/12/syslog-ng-on-windows-quick-start-gu… * TLS vulnerabilities in log traffic: syslog-ng vs. BEAST, CRIME and BREACH:http://gyp.blogs.balabit.com/2013/12/syslog-ng-vs-beast-crime-and-br… * Top 6 SANS Essential Categories of Log Reports 2013: http://czanik.blogs.balabit.com/2013/12/top-6-sans-essential-categories-of-… * Shortening Breach Resolution Times with Distributed Log Management: http://jluby.blogs.balabit.com/2013/12/30/shortening-breach-resolution-time… NEW RELEASES * syslog-ng 3.5.3: https://lists.balabit.hu/pipermail/syslog-ng-announce/2013-December/000177.… * syslog-ng 3.4.7: https://lists.balabit.hu/pipermail/syslog-ng-announce/2013-December/000178.… Your feedback and news tips about the next issue is welcome at documentation(a)balabit.com <mailto:documentation@balabit.com> To read this newsletter on-line, visit: http://insider.blogs.balabit.com/ -- Peter Czanik (CzP) <czanik(a)balabit.hu> BalaBit IT Security / syslog-ng upstream http://czanik.blogs.balabit.com/ https://twitter.com/PCzanik

1 0

[PATCH] confgen: backticks weren't processed in generated configuration part
by Juhász Viktor 28 Dec '13

28 Dec '13

confgen: backticks weren't processed in generated configuration part produced by confgen module. The reason of this problem was, that the cfg_lexer_subst_args was called only for read lines from configuration files (YY_INPUT) and for the processed blocks. If process the backticks for a cfg input buffer is a must, the cfg_lexer_subst_args has to be called explicit. Now the confgen module call the it before set the include buffer to the lexer Signed-off-by: Juhász Viktor <jviktor(a)balabit.hu> --- modules/confgen/confgen-plugin.c | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/modules/confgen/confgen-plugin.c b/modules/confgen/confgen-plugin.c index 519a660..9831962 100644 --- a/modules/confgen/confgen-plugin.c +++ b/modules/confgen/confgen-plugin.c @@ -40,6 +40,7 @@ confgen_generate(CfgLexer *lexer, gint type, const gchar *name, CfgArgs *args, g gchar *exec = (gchar *) user_data; gsize res; gchar buf[256]; + gchar *old_value; g_snprintf(buf, sizeof(buf), "%s confgen %s", cfg_lexer_lookup_context_name_by_type(type), name); if (!cfg_args_validate(args, NULL, buf)) @@ -79,6 +80,10 @@ confgen_generate(CfgLexer *lexer, gint type, const gchar *name, CfgArgs *args, g g_free(value); return FALSE; } + + old_value=value; + value = cfg_lexer_subst_args(lexer->globals, NULL, NULL, value, &value_len); + g_free(old_value); if (!cfg_lexer_include_buffer(lexer, buf, value, value_len)) { g_free(value); -- 1.8.3.2

2 2

syslog-ng 3.5.3 has been released
by devel＠balabit.hu 25 Dec '13

25 Dec '13

------------------------------------------------------------------------------ PACKAGE : syslog-ng VERSION : 3.5.3 SUMMARY : new stable release DATE : Dec 24, 2013 ------------------------------------------------------------------------------ DESCRIPTION: A new stable version of syslog-ng Open Source Edition (3.5.3) has been released. For latest fixes in the 3.5.x feature branch you are recommended to upgrade to this version. CHANGES: 3.5.3 Tue, 24 Dec 2013 19:53:05 +0100 This is the second bug-fix release for the 3.5.x series, with a number of important changes over the previous release. Upgrading to this version is highly recommended. Bugfixes ======== * Node flag propagation has been fixed, so flags(final) works the same as it did in 3.3, when embedded log statements are used too. * Improvements were made to the build system, to make it more reliable when building in parallel. * Using the value-pairs() framework, either via the $(format-json) template function or any of the destinations that make use of it, will now correctly skip and ignore empty values, even in SDATA. * The configure script will correctly detect newer versions of json-c too, without compatibility symlinks being present. * A file descriptor and memory leak has been fixed in the syslog() destination. * The pdbtool program will warn in test mode when a test message has no program attribute. [BugZilla #263] * If the PID property is overwritten, syslog-ng will now stop using the cached MSGHDR value, and therefore, use the newly set PID instead, as it was supposed to. * The localport() option of various network destinations has been restored, and will work correctly now. [BugZilla #266] * The upstart service file in contrib has been updated to use a different process mode (background), as we do not need a supervisor when running under upstart. This also means that upstart will be far less confused. * Manual pages are now always included in the tarball. Credits ======= syslog-ng is developed as a community project, and as such it relies on volunteers to do the work necessarily to produce syslog-ng. Reporting bugs, testing changes, writing code or simply providing feedback are all important contributions, so please if you are a user of syslog-ng, contribute. These people have helped in this release: Andras Mitzki <micek(a)balabit.hu> Balazs Scheidler <bazsi(a)balabit.hu> Chris Johnson <chris.johnson3(a)hp.com> Fabien Wernli <cpan(a)faxm0dem.org> Gergely Nagy <algernon(a)balabit.hu> Lucas McLane <lucas(a)clicksecurity.com> Peter Czanik <czanik(a)balabit.hu> Sebastiaan Hoogeveen <zebaz(a)xs4all.nl> Tobias Schwab <tobias.schwab(a)dynport.de> DOWNLOAD: You can download the source or binary packages from: http://www.balabit.com/network-security/syslog-ng/opensource-logging-system… The documentation of the syslog-ng Open Source Edition is available in The syslog-ng Open Source Edition Administrator's Guide at http://www.balabit.com/support/documentation/

3 2

syslog-ng 3.4.7, 3.5.3 and incubator packages for openSUSE / SLES
by Peter Czanik 25 Dec '13

25 Dec '13

Hello, Here is my Christmas present for you: openSUSE packages for syslog-ng updated and syslog-ng incubator package added. As it's Christmas, it received only very limited testing (only openSUSE 13.1 x86_65) and incubator lacks riemann support (that needs packaging an additional support library). 3.5 man pages are still a kind of broken, so SLES support is missing there. Packages are available at: https://build.opensuse.org/project/show/home:czanik:syslog-ng34 https://build.opensuse.org/project/show/home:czanik:syslog-ng35 Bye, -- Peter Czanik (CzP) <czanik(a)balabit.hu> BalaBit IT Security / syslog-ng upstream http://czanik.blogs.balabit.com/ https://twitter.com/PCzanik

1 0