- syslog-ng - lists.balabit.hu

follow_freq() On A Static File
by Hidayath Basha 09 Mar '11

09 Mar '11

Hello, Is it fine to apply follow_freq(1) on a static file (not tailing one) Or One should use follow_freq() only on a tailing file? Thanks and regards, Hidayath

2 1

log4j
by Peter Czanik 08 Mar '11

08 Mar '11

Hello, As I don't code in Java, I don't have any personal experiences with log4j. But from e-mail and forum messages, and also from discussions at FOSDEM I know, that log4j is often used together with syslog-ng. I ran into this log4j related question this morning: http://serverfault.com/questions/242972/getting-syslog-ng-to-recognize-a-ja… I'd like to ask, if there is anybody who already has a solution for the above question or any other useful tips about log4j & syslog-ng to share. Thanks, bye, -- Peter Czanik (CzP) <czanik(a)balabit.hu> BalaBit IT Security / syslog-ng upstream http://czanik.blogs.balabit.com/

2 1

code-commenting log messages
by Peter Gyongyosi 07 Mar '11

07 Mar '11

Hi, most of the pattern database-related projects I've seen here started off from the log side of the whole issue: investigating the messages an application produces and trying to create patterns based on them (manually or via patternize). Of course, this makes perfect sense if you're an administrator or you're trying to analyze logs from already-written applications, but what if you're the one who writes these applications and you'd like to make them easier to analyze later on? What if you're the extremely nice guy who wants to ship an up-to-date pattern database for syslog-ng along with your application? :) Or, to be more realistic, at least a list of the most important log messages your program can emit that the user can search to find out what an actual log message means. The idea I've been toying with is defining some kind of a standardized commenting scheme for logging-related calls in source code. Using this, the code could be parsed automatically and a list of log messages could easily be generated that could be used later for documentation or even generating a pattern database. I've done multiple rounds of investigation but I only found custom, application-specific solutions for this. So on one hand, I'm asking you if you've seen any widespread commenting schemes for log messages that can fulfil this task? Or even a custom, application-specific solution that is so incredibly cool that we should all adopt it instead of reinventing the wheel? :) Until then, I'm proposing a commenting scheme I think would be usable and I'm asking for your feedback. It is based on Javadoc, which is widespread enough to be familiar for the majority of developers and has mutations available for virtually all programming languages. I'm going to show the concept through the now-standard example of the logging of the login in sshd: /** * @log * * Signals that the authentication has succeeded or failed for a user. * * @class system * @tags useracct, secevt * @url http://www.openssh.com * * @example Accepted password for bazsi from 127.0.0.1 port 48650 ssh2 * @example Failed password for bazsi from 127.0.1.1 port 44637 ssh2 * * @pattern Accepted @ESTRING:usracct.authmethod: @for @ESTRING:usracct.username: @from @ESTRING:usracct.device: @port @ESTRING:: @@ANYSTRING:usracct.service@ * @pattern Failed @ESTRING:usracct.authmethod: @for @ESTRING:usracct.username: @from @ESTRING:usracct.device: @port @ESTRING:: @@ANYSTRING:usracct.service@ * * @regexp (Accepted|Failed) (gssapi(-with-mic|-keyex)?|rsa|dsa|password|publickey|keyboard-interactive/pam|hostbased) for [^[:space:]]+ from [^[:space:]]+ port [[:digit:]]+( (ssh|ssh2))?$ */ authlog("%s %s for %s%.100s from %.200s port %d%s", authmsg, method, authctxt->valid ? "" : "invalid user ", authctxt->user, get_remote_ipaddr(), get_remote_port(), info); To formalize it a bit: - comments for the log messages should follow the standard Javadoc/Doxygen formatting, with the main indentifier token being "@log" (or \log if you prefer, all further parameters should be usable in the backslash form as well) - the allowed parameters are @class, @url, @example, @regexp, @pattern and @tags - multiple @example, @pattern and @regexp fields are allowed - it is mandatory to specify either @example, @pattern or @regexp to give the application some kind of an identifier, all other params can be omitted I have my own concerns, though: - Is it even the right thing to do to spam source code like this? Is it the proper place for the semantic description of log messages? Sure, it's the easiest place to add such things while the app is being developed and if the comment's right there in the code it's more likely that any changes will be followed up during an update of the code, but still, it seems a bit of an overkill even for me now that I look at it... But maybe it'd worth it for the most important messages. - Maybe and @id would be useful to allow for manually specifying an identifier for the log messages (we all love Oracle's error codes, don't we? :)) - What should be done if a single log() call can emit multiple different messages with different meanings that should be getting their own separate entry? The example above is the perfect candidate for this: the "login accepted" and "login failed" events are definitely different. Maybe two separate @log comments could be added in this case. - How does it relate to more advanced db-parser features like correlation or the self-testing ability of pdbtool of name-value pairs in the <examples> section? (- Isn't it a bit too push-y to call it @pattern instead of @syslog-ng-pattern? :)) However, if such a scheme could be adopted, with some little hacking several cool things could be done (which I'm willing to do): - create "pdbtool sourceextract" or something like that that can generate ready-to-use pattern databases from the source code - this can be added to the build processes to update them along with manpages and similar stuff - Doxygen & co. could be patched to parse and display them - vim, Eclipse and Your Favourite Editor could get a one-click way to add such comment blocks, just as it is possible to add header comments easily What do you think? greets, Peter

3 2

cisco/squid feedback
by Alexander Clouter 07 Mar '11

07 Mar '11

Hi, Digging around, I could not find anything on how to 'sensibly' log the junk that Cisco IOS devices (and their infernal WLC) spit out plus I was keen to rewrite squid HTTP proxy server logs to make use of the epoch+msec timestamp found in the MSG. I have documented, very roughly, my current solution on my website and would welcome amendments/fixes/flames/etc on the approach: http://www.digriz.org.uk/syslog-ng-integration Before you ask, I use 'match("fqdn.example.com" value("HOST_FROM") ...)' as netmask() is broken for IPv6 :) Cheers -- Alexander Clouter .sigmonster says: When pleasure remains, does it remain a pleasure?

2 5

[PATCH]: support setting a default suppress() globally
by Gergely Nagy 04 Mar '11

04 Mar '11

Hi! Attached is a patch that makes it possible to set a default value for suppress() in the global options block, which will then be inherited by all drivers that support this option. Of course, it can still be overridden on a per-destination basis. What this means is, that if one has a larger number of destinations that all need the same suppress settings, we'd write that like this before this patch: destination d_1 { file("/var/log/messages-1" suppress(10)); }; destination d_2 { file("/var/log/messages-2" suppress(10)); }; ... destination d_N { file("/var/log/messages-N" suppress(10)); }; destination d_XX { file("/var/log/messages-XX" suppress(5)); }; After the patch, this can be shortened to: options { suppress(10); }; destination d_1 { file("/var/log/messages-1"); }; ... destination d_N { file("/var/log/messages-N"); }; destination d_XX { file("/var/log/messages-XX" suppress(5)); }; Both result in exactly the same behaviour, but the second is a lot shorter, and easier to maintain, as long as the majority of destinations need the same suppress() setting. Hopefully people will find this useful. The patch is also available from the work/global-suppress branch of my tree at git://git.balabit.hu/algernon/syslog-ng-3.3.git -- |8]

2 4

Reading Multiple Files Using Regex
by Hidayath Basha 04 Mar '11

04 Mar '11

Hi, I have multiple files... some-host:/some/path $ ls -l -rw------- 1 root root 90K Mar 3 11:37 mylogs-2011-03-03 -rw------- 1 root root 121 Mar 4 11:40 mylogs-2011-03-04 Can I read these files in the source driver with following source driver? source s_files{ file("/some/path/mylogs*"); }; I use SLES 11.0 and syslog-ng 2.0.9 Thanks and regards, Hidayath Saven Technologies: A leader in technology driven business solutions ---------------------------------------------------------------------------- ----------------------------------- #302, 3rd Floor, MyHome Sarovar Plaza, Hyderabad- 500 063, India Off: +91 040 23237303/7304 ext 115 email: hidayath.basha(a)saventech.com | www.saventech.com follow me: http://twitter.com/hidayath26 ---------------------------------------------------------------------------- -----------------------------------

2 1

patterndb context-timeout problem
by Zoltán Pallagi 04 Mar '11

04 Mar '11

Hi, I've found a problem in handling context-timeout in a special case. If context-timeout=10 (10 sec) and I am using a trigger, where trigger="timeout", it can happen that the timeout won't expire: I sent the following two messages to syslog-ng before expiring timeout (but the difference between the two timestamp is bigger than the timeout value): "<38>1990-01-01T14:45:25 customhostname program6[1234]: program6 testmessage" "<38>1990-01-01T14:55:25 customhostname program6[1234]: program6 testmessage" I receive the timeout message immediately, when the second message arrives (it's ok), but there is no timeout for the second message.

2 1

Message correlation, everything needs an "@"? And a grep problem.
by Valentijn Sessink 04 Mar '11

04 Mar '11

Hello list, After playing with correlation for a while, here are a few remarks on it's workings. I have played with syslog-ng 3.2.2, built from source on an Ubuntu 10.04 machine. For some reason, the "@1", "@2"... etcetera specifiers needed to be all over the place to be able to see any variable at all. For example, the rule below describes the last message of a Postfix log trail. Note that even the context-id is specified as "${queueid}@1". All other references also have an "@" sign. Without the @, the variables seem just empty. <rule id='X' context-id="${queueid}@1" context-scope="host" provider="openoffice" class="maildelivery"> <patterns> <pattern>@ESTRING:queueid: @removed</pattern> </patterns> <actions> <action> <message> <values> <value name="MESSAGE">${queueid}@1 mail to=<${toaddress}@2> from=<${pffrom}@3> connection from=${pfremotehost}@5[${pfremoteip}@5] msgid=<${msgid}@4> orig_to=<${origto}@2> delay=${deliverystatus}@2</value> </values> </message> </action> </actions> </rule> Is this how it's meant to be? Then, when trying to use the "grep" template function in this message, I can't seem to find out how it works. Using the exact example from the documentation, $(grep ("${username}" == "root") ${auth_method}), I only get an error: Error parsing filter expression, syntax error, unexpected $undefined in <string> at line 1, column 3: (${ ^ Trying to alter the message, using "$something" == "value", doesn't seem to have an effect, i.e. I can't get "grep" to find any values. In fact, I'm not even sure if I can use the names from the (E/Q)STRING parsers right away, or if I need to have <value name="something"> tags before I can grep for them. In fact, most of the time, the grep I would like to use is something like: $(grep ( "$variable" != "" ) "$variable"). But whatever I try, no results from grep. Could anyone please clarify? Best regards, Valentijn

2 15

Permission Denied (13)
by Hidayath Basha 04 Mar '11

04 Mar '11

Hi, we are using SLES 11 We are trying to read a tailing file and send it to the server syslog-ng.conf (on client 192.168.18.10 some-host) #syslog-ng.conf options { long_hostnames(off); sync(1); perm(0640); stats(3600); }; source s_mysrc{ file("/some/file.log" ); }; destination d_mydest { tcp("192.168.18.7" port(5140)); }; log { source(s_mysrc); destination(d_mydest); }; #syslog-ng.conf ends syslog-ng.conf (on Server 192.168.18.7) #syslog-ng.conf options { long_hostnames(off); sync(0); perm(0640); stats(3600); }; source s_mysrc{ tcp(ip("192.168.18.7") port(5140) keep-alive(yes)); }; destination d_mydest { file ("/var/log/hosts/$YEAR-$MONTH/$HOST/$FACILITY-$YEAR-$MONTH-$DAY" owner(root) group(root) perm(0600) dir_perm(0700) create_dirs(yes) ); }; log { source(s_mysrc); destination(d_mydest); }; #syslog-ng.conf ends When I start the service at client, I'm getting the following error Starting syslog servicesError opening file for reading; filename='/some/file.log', error='Permission denied (13)' Error initializing source driver; source='s_mysrc' startproc: exit status of parent of /sbin/syslog-ng: 2 The file /some/file.log has -rw-rw-rw- permissions When I modify the source driver to ... file("/some/file.log" follow_freq(1)); I'm able to start the syslog service, but I don't see anything on the server (192.162.18.7) at /var/log/hosts/2011-03/some-host/ Please help Thanks and regards, Hidayath Saven Technologies: A leader in technology driven business solutions ---------------------------------------------------------------------------- ----------------------------------- #302, 3rd Floor, MyHome Sarovar Plaza, Hyderabad- 500 063, India Off: +91 040 23237303/7304 ext 115 email: hidayath.basha(a)saventech.com | www.saventech.com follow me: http://twitter.com/hidayath26 ---------------------------------------------------------------------------- -----------------------------------

2 3

[Bug 117] New: syslog-ng.persist should not reside in /var/lib
by bugzilla＠bugzilla.balabit.com 03 Mar '11

03 Mar '11

https://bugzilla.balabit.com/show_bug.cgi?id=117 Summary: syslog-ng.persist should not reside in /var/lib Product: syslog-ng Version: unspecified Platform: PC OS/Version: Linux Status: NEW Severity: normal Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: wlt(a)obsidian-studios.com Type of the Report: --- Estimated Hours: 0.0 syslog-ng uses a syslog-ng.persist file that is created in /var/lib. Per FHS 2.3 that should exist in a subdir, if not one specific to syslog-ng, then /var/lib/misc. http://www.pathname.com/fhs/pub/fhs-2.3.html#VARLIBVARIABLESTATEINFORMATION An application (or a group of inter-related applications) must use a subdirectory of /var/lib for its data. There is one required subdirectory, /var/lib/misc, which is intended for state files that don't need a subdirectory; the other subdirectories should only be present if the application in question is included in the distribution. [38] [38] An important difference between this version of this standard and previous ones is that applications are now required to use a subdirectory of /var/lib. This causes problems on diskless systems with a shared /var/lib. Subdirs that contain unique per system data are not shared, and instead mounted on top of the shared /var/lib. I could do the same thing even for a single file, syslog-ng.persist in this case. However given FHS, it should be written in a subdir. Which I can then make that unique per system, like other subdirs in /var/lib that are not shared amongst all diskless systems. -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 2