- syslog-ng - lists.balabit.hu

Some messages are missing
by Tinnis G 14 Mar '11

14 Mar '11

Hi, I am not getting all the messages in the message log files . If someone helps me , it will be really appreciated. We have remote servers , where we are keeping the log files too. I feel that the problem is in the filter. Please correct me. I want all the logs will be reported . Mentioning below the configuration file on each server. These are all same throughout. Thanks in advance. _________________________________________________________ options { sync (0); time_reopen (10); log_fifo_size (1000); long_hostnames(off); use_dns (yes); use_fqdn (no); create_dirs (yes); keep_hostname (yes); stats (3600); }; source src { unix-stream("/dev/log"); internal(); pipe("/proc/kmsg"); }; # Remote logging ################ ## Auth log destination loghost1 { tcp("log1.xx.org" port(514)); }; destination loghost2 { tcp("log2.xx.org" port(514)); }; destination loghost3 { tcp("log3.xx.org" port(514)); }; filter f_auth { facility(auth); }; log { source(src); filter(f_auth); destination(loghost1); }; log { source(src); filter(f_auth); destination(loghost2); }; log { source(src); filter(f_auth); destination(loghost3); }; # ## Authpriv log destination loghost1 { tcp("log1.xx.org" port(514)); }; destination loghost2 { tcp("log2.xx.org" port(514)); }; destination loghost3 { tcp("log3.xx.org" port(514)); }; filter f_authpriv { facility(auth, authpriv); }; log { source(src); filter(f_authpriv); destination(loghost1); }; log { source(src); filter(f_authpriv); destination(loghost2); }; log { source(src); filter(f_authpriv); destination(loghost3); }; ## Everything log destination loghost1 { tcp("log1.xx.org" port(514)); }; destination loghost2 { tcp("log2.xx.org" port(514)); }; destination loghost3 { tcp("log3.xx.org" port(514)); }; filter f_everything { level(debug..emerg); }; log { source(src); filter(f_everything); destination(loghost1); }; log { source(src); filter(f_everything); destination(loghost2); }; log { source(src); filter(f_everything); destination(loghost3); }; # # Local logging ################ # Local Destinations #%%%%%%%%%%%%%%%%%%% destination authpriv { file("/var/log/authpriv.log"); }; destination auth { file("/var/log/auth.log"); }; destination syslog { file("/var/log/syslog.log"); }; destination messages { file("/var/log/messages.log"); }; destination cron { file("/var/log/cron.log"); }; destination daemon { file("/var/log/daemon.log"); }; destination kernel { file("/var/log/kernel.log"); }; destination lpr { file("/var/log/lpr.log"); }; destination mail { file("/var/log/mail.log"); }; destination debug { file("/var/log/debug.log"); }; # # destination console { usertty("root"); }; destination console_all { file("/dev/tty12"); }; # Local Filters ############### filter f_syslog { not facility(authpriv, mail); }; filter f_messages { level(info..warn) and not facility(auth, authpriv, mail, news); }; filter f_authpriv { facility(auth, authpriv); }; filter f_auth { facility(auth); }; filter f_cron { facility(cron); }; filter f_daemon { facility(daemon); }; filter f_kernel { facility(kern); }; filter f_lpr { facility(lpr); }; filter f_mail { facility(mail); }; filter f_debug { not facility(auth, authpriv, news, mail); }; # # filter f_emergency { level(emerg); }; filter f_info { level(info); }; filter f_notice { level(notice); }; filter f_warn { level(warn); }; filter f_crit { level(crit); }; filter f_err { level(err); }; # Local Log ############ log { source(src); filter(f_syslog); destination(syslog); }; log { source(src); filter(f_messages); destination(messages); }; log { source(src); filter(f_authpriv); destination(authpriv); }; log { source(src); filter(f_auth); destination(auth); }; log { source(src); filter(f_cron); destination(cron); }; log { source(src); filter(f_daemon); destination(daemon); }; log { source(src); filter(f_kernel); destination(kernel); }; log { source(src); filter(f_lpr); destination(lpr); }; log { source(src); filter(f_mail); destination(mail); }; log { source(src); filter(f_debug); destination(debug); }; # log { source(src); filter(f_emergency); destination(console); }; log { source(src); destination(console_all); }; # ## END /etc/syslog-ng/syslog-ng.conf _________________________________________________________

3 2

Reading Log file
by Shawn Cannon 14 Mar '11

14 Mar '11

Can someone tell me what the numbers mean for processed and dropped? Am I logging everything to SQL or am I dropping some? Mar 9 17:12:03 syslog01 syslog-ng[1738]: Log statistics; processed='source(s_gms)=2279331', dropped='dst.sql(d_mssql#0,freetds, rawsql.adbcefg.net,1785,Syslog)=2213605', stored='dst.sql(d_mssql#0,freetds, rawsql.abcdefg.net,1785,Syslog)=0',

3 4

[PATCH]: Experimental SMTP destination
by Gergely Nagy 11 Mar '11

11 Mar '11

Hi! I've had a little free time, and started toying with the idea of implementing an SMTP destination for syslog-ng. So far, it's progressing nicely, but there are a few shortcomings, as I will explain later. For anyone interested, it can be found on the 'work/modules/afsmtp' branch of my syslog-ng-3.3 tree: git://git.balabit.hu/algernon/syslog-ng-3.3.git Or, one can browse the branch on the web: http://git.balabit.hu/?p=algernon/syslog-ng-3.3.git;a=shortlog;h=work/modul… One will need the libesmtp library installed for the module to work. It should be available in most distributions, but if one doesn't have it, it can be downloaded from http://www.stafford.uklinux.net/libesmtp/ (although, as of this writing, the site seems to be down). The usage is something like this: destination d_smtp { smtp( host("localhost") port(25) mail_from("syslog-ng alert service <noreply(a)example.com>") rcpt_to("Admin #1 <admin(a)example.com>") rcpt_to("Admin #2 <admin2(a)example.com>") subject("[ALERT] $LEVEL condition in $PROGRAM!") header("X-Program" "$PROGRAM") header("X-Facility" "$FACILITY") body("Hi!\nThe syslog-ng alerting service thinks you might be interested in the following message:\n $MSG\n-- \nsyslog-ng\n") ); }; One can specify any number of recipients, and the mail will be sent to all of them. Custom headers can also be specified, but at this time, a selected few headers do not work as one would expect, due to libesmtp handling them specially. These are: Date, From, To, Cc, Bcc, Reply-To - trying to set either of these will result in unspecified behaviour (ranging from mangled headers to crashes, in case of Date). I will fix these at a later point. They will probably be need to set via their own statements (header_date, header_from, etc, or something along those lines). Specifying mail_from or subject multiple times will override any previous values. Adding the same custom header multiple times will only add the first (though, plans include changing that, so that one can include the same header multiple times, with different values - if I can find a use-case for that). There are also a few other minor issues here and there, but nothing major, as far as I could check. Further plans include authentication & starttls support. libesmtp supports both, I just didn't have time to write the neccessary glue code yet. -- |8]

8 26

Destination MySQL for Syslog-ng 2.0.x
by Hidayath Basha 11 Mar '11

11 Mar '11

Hi, How can I collect the logs in MySQL db with syslog-ng 2.0.x? There is a destination 'sql' in syslog-ng 3+, but how can I do it with 2.0? Thanks and regards, Hidayath

3 2

Transmitting a Huge Log File
by Hidayath Basha 11 Mar '11

11 Mar '11

Hi all, I'm trying to transmit a huge log file (of about 80k lines) to a centralized syslog server over TCP But, on the syslog server, I'm receiver only the last part of the log file (of about 7000 lines) How can I transmit the whole log file On client, the syslog-ng.conf file ... destination d_mydest { tcp("192.168.18.7" port(5140)); }; source s_src { file("/some/huge/file.log" follow_freq(1) log_prefix("mylogs:")); }; log { source(s_src); destination(d_mydest); }; On server (192.168.18.7) , the conf file ... source s_mysrc{ tcp(ip("192.168.18.7") port(5140) keep-alive(yes)); }; filter f_filter{ match("mylogs:"); }; destination d_mydest { file ("/var/log/hosts/$HOST/$YEAR-$MONTH/$YEAR-$MONTH-$DAY/hugefile.log" owner(root) group(root) perm(0600) dir_perm(0700) create_dirs(yes) ); }; log { source(s_mysrc); filter(f_filter); destination(d_mydest); }; Thanks and regards, Hidayath

4 4

Dropped messages to MSSQL?
by Shawn Cannon 10 Mar '11

10 Mar '11

I really need help on this. I am collecting tons of SYSLOG data from over 500 firewalls. I have these syslogs going to a Barracuda Load Balancer first which will then send messages to 2 syslog-ng open source servers. Here is how I have my config file setup: @version: 3.0 #Default configuration file for syslog-ng. # # For a description of syslog-ng configuration file directives, please read # the syslog-ng Administrator's guide at: # # http://www.balabit.com/dl/html/syslog-ng-admin-guide_en.html/bk01-toc.html # options { use_dns(no); }; ###### # sources source s_gms { # message generated by Syslog-NG # internal(); # standard Linux log source (this is the default place for the syslog() # function to send logs to) # unix-stream("/dev/log"); # messages from the kernel #file("/proc/kmsg" program_override("kernel")); # messages destined to udp514 udp(ip(0.0.0.0) port(514)); }; source s_syslogng { internal(); }; ###### # destinations destination d_messages { file("/var/log/messages"); }; destination d_mssql { sql(type(mssql) host("rawsql.abcdefg.net") port("1785") username("username") password("password") database("Syslog") table("syslogng")columns("datetime varchar(16)", "host varchar(32)", "program varchar(32)", "pid varchar(8)", "message varchar(4096)") values("$R_DATE", "$HOST", "$PROGRAM", "$PID", "$MSGONLY") indexes("datetime", "host", "program", "pid")); }; log { source(s_gms); destination(d_mssql); }; log { source(s_syslogng); destination(d_messages); }; When messages are coming in, I am showing the following in the local syslog-ng messages: Log statistics; processed='source(s_gms)=2155636', dropped='dst.sql(d_mssql#0,freetds,rawsql.abcdefg.net,1785,Syslog)=1717472', stored='dst.sql(d_mssql#0,freetds,rawsql.abcdefg.net,1785,Syslog)=0', I assume this is telling me that I am dropping the majority of my messages instead of them getting inserted into my MS SQL database? The MS SQL Database runs on a very beefy server with plenty of memory. I am trying to determine why this is being dropped. Please help as I am huge newbie when it comes to syslog-ng. Thanks!

5 14

syslog-ng 3.3.0alpha2 released
by Balazs Scheidler 10 Mar '11

10 Mar '11

Hi, I'm happy to announce that I've just uploaded syslog-ng 3.3.0alpha2 to our website. It contains a couple of important bugfixes and some new features compared to 3.3.0alpha1. It worked for me quite well, and it endured the first round of testing by the BalaBit syslog-ng PE team, when they've started working with the code. Highlights ========== syslog-ng 3.3 has a couple of new and interesting features, already in this alpha release: * It uses multiple threads to do its work, thus capable of scaling to multiple CPUs. Add 'threaded(yes)' to your option block and you're good to go. * Built in mongodb destination for folks who prefer NoSQL over traditional SQL. * This version has a new and improved "pdbtool patternize" functionality More exciting stuff is coming. Known issues ============ I know for certain that it will not work on FreeBSD unless the following environment variable is set: IV_EXCLUDE_POLL_METHOD=kqueue Otherwise syslog-ng will crash at startup, immediately. Stability ========= Certainly you wouldn't want to run your production system on this version, however this code base had endured a couple of test rounds, so if it compiles and starts up for you (which can be a challenge I can admit), then the basic functions will most certainly work. Changes since 3.3.0alpha1 ========================= 3.3.0alpha2 Thu, 10 Mar 2011 12:51:36 +0100 Features: ======== * Support for systemd activation added. * Add support for customizable token delimiters in "pdbtool patternize", which makes the resulting patterns to have much better quality. * Added support for a --no-parse command line option for "pdbtool patternize" to avoid parsing the input as normal syslog messages. * Added a new flag 'ignore-errors' to LogWriter based destinations (file, pipe, tcp, udp, syslog) * Added support for specifying the suppress() option globally. Bugfixes: ========= * Fixed a tcp()/udp()/syslog() destination driver issue that caused aborts during startup. * Handle non-epollable devices like /dev/null normally by failling back to the regular file-like handling. Earlier these caused an abort inside ivykis. * Makefile fixes to make it possible to compile syslog-ng from a "make dist" tarball. * Added error messages about libdbi initialization failures, that would prevent the SQL destination from working. * Make it possible to compile against PCRE not in a standard location (caused compilation failures on FreeBSD). * Fixed several $(grep) related bugs: - not to cause a segmentation faule when the filter expression supplied contains syntax errors - the template parsing code removed quotes required by filter expressions, with the current change it is possible to use quotes in the filter expressions directly: $(grep ('$username' == 'root') $MSG) - fixed the handling of template function invocations within correllation that had multiple messages in its context. * Fixed a segmentation fault in pdbtool match --debug-pattern in case there was no matching rule. * Fixed "pdbtool test". * Added a new macro named $CONTEXT_ID that expands to the current context-id in correllated rules. Other changes: ============== * Changed the default syslog-ng.conf version number to 3.3 to match the current version. * The pipe() destination used to override the default value of flush_lines() by explicitly setting it to 0. This behaviour has been removed, now the pipe() driver will properly use the default, unless overridden explicitly. * Ported the 3.2 linking changes to 3.3, which means that: - libsyslog-ng-patterndb.so is gone, both the command line pdbtool command and the patterndb plugin (libpatterndb.so) links its contents statically, to improve portability on Cygwin. - unit tests & command line utilities link properly in --enable-mixed-linking mode * Added debug messages in the correllation code to make it easier to debug correllation rules. * Added debug/verbose options to pdbtool. Credits: ======== Code, bugreports, testing, documentation suggestions and other improvements were contributed by: * Arkadiusz Miśkiewicz (PLD Linux) * Balazs Scheidler (BalaBit) * Balint Kovacs (BalaBit) * Corinna Vinschen (RedHat) * Dalibor Toman (fortech.cz) * Gergely Nagy (BalaBit) * Laszlo Boszormenyi (lsc.hu) * Marius Tomaschewski (Novell) * Peter Czanik (BalaBit) * Peter Gyongyosi (BalaBit) * Tom Gundersen (jklm.no) * Valentijn Sessink (blub.net) * Zoltan Pallagi (BalaBit) -- Bazsi

1 1

PostgreSQL, MonitorWare Schema and log presentation/analysis
by Hendrik Visage 10 Mar '11

10 Mar '11

Hi there, I've searched hi and low for a log analyzer/presenter for PostgreSql backend (still looking) but keep bumping into stuff like LogAnalyzer wanting a MonitorWare Schema.. Searching for that with syslog-ng, I've been unsuccessful (or I am missing something), other than getting the SQL schema below on a chinese site. Now my quetion: Anybody else used syslog-ng to pump directly into a PostgreSQL db a Monitorware Schema? Any other place I can get some more information on this schema usage? Anybody using PostgreSQL as a storage back end with a Web based GUI? (Before we enter the MySQL vs PostgreSQL debates, the answer is simple: we know postgresql, we don't know mysql, and if we hit performance issues in the current environment, we know we will need to procure the necessary HW and handle those decisions there) CREATE TABLE SystemEvents ( ID int unsigned not null auto_increment primary key, CustomerID bigint, ReceivedAt datetime NULL, DeviceReportedTime datetime NULL, Facility smallint NULL, Priority smallint NULL, FromHost varchar(60) NULL, Message text, NTSeverity int NULL, Importance int NULL, EventSource varchar(60), EventUser varchar(60) NULL, EventCategory int NULL, EventID int NULL, EventBinaryData text NULL, MaxAvailable int NULL, CurrUsage int NULL, MinUsage int NULL, MaxUsage int NULL, InfoUnitID int NULL , SysLogTag varchar(60), EventLogType varchar(60), GenericFileName VarChar(60), SystemID int NULL ); CREATE TABLE SystemEventsProperties ( ID int unsigned not null auto_increment primary key, SystemEventID int NULL , ParamName varchar(255) NULL , ParamValue text NULL );

2 1

syslog-ng 3.0.7 shows 2 daemons on restart
by Rakesh Rajasekharan 10 Mar '11

10 Mar '11

Hi, I have compiled syslog-ng 3.0.7 on an HP-UX machine. Now, when I try to restart, it shows 2 daemons. # /sbin/init.d/syslog_ng restart syslog-ng service starting. # ps -ef|grep -i syslog root 3635 3634 0 09:17:58 ? 0:00 /opt/soe/local/syslog_ng/sbin/syslog-ng -f /opt/soe/local/syslog_ng/etc/syslog-ng.conf -p /var/run/syslog-ng.pid root 3640# /sbin/init.d/syslog_ng restart syslog-ng service starting. # ps -ef|grep -i syslog *root 3635 3634 0 09:17:58 ? 0:00 /opt/soe/local/syslog_ng/sbin/syslog-ng -f /opt/soe/local/syslog_ng/etc/syslog-ng.conf -p /var/run/syslog-ng.pid root 3640 2701 0 09:18:01 pts/0 0:00 grep -i syslog root 3634 1 0 09:17:58 ? 0:00 /opt/soe/local/syslog_ng/sbin/syslog-ng -f /opt/soe/local/syslog_ng/etc/syslog-ng.conf -p /var/run/syslog-ng.pid 2701 0 09:18:01 pts/0 0:00 grep -i syslog root 3634 1 0 09:17:58 ? 0:00 /opt/soe/local/syslog_ng/sbin/syslog-ng -f /opt/soe/local/syslog_ng/etc/syslog-ng.conf -p /var/run/syslog-ng.pid * ** Is this soemthing expected . Please confirm. Thanks, Rakesh

4 4

AIX 5.3 syslog-ng 3.0.7
by Rakesh Rajasekharan 09 Mar '11

09 Mar '11

Hi, I am facing a somewhat strange issue on AIX 5.3. I am unable to restart the daemon whenever i try to set it up as a central server. On a a normal case it works fine . After copying my syslog-ng.conf.server-sample as syslog-ng.conf., when i restart the daemon, it does not comlian about anything and gives me a PID as well. But the daemon does not seem to have started. Below are the logs. Setting Up as A Central Server ======================= cscaixlgg059:root:/opt/soe/local/syslog_ng/etc>cp syslog-ng.conf.server-sample syslog-ng.conf cscaixlgg059:root:/opt/soe/local/syslog_ng/etc>/*opt/soe/local/syslog_ng/etc/init.d/syslog-ng.d restart *0513-004 The Subsystem or Group, syslog_ng, is currently inoperative. 0513-059 The syslog_ng Subsystem has been started. Subsystem PID is 233502. cscaixlgg059:root:/opt/soe/local/syslog_ng/etc>ps -ef|grep -i syslog cscaixlgg059:root:/opt/soe/local/syslog_ng/etc>lssrc -a|grep -i syslog * syslog_ng local inoperative * But , otherwise as in a normal case or even as a client it wors perfectly fine. cscaixlgg059:root:/opt/soe/local/syslog_ng/etc>cp syslog-ng.conf.bkp syslog-ng.conf cscaixlgg059:root:/opt/soe/local/syslog_ng/etc>/opt/soe/local/syslog_ng/etc/init.d/syslog-ng.d restart 0513-004 The Subsystem or Group, syslog_ng, is currently inoperative. 0513-059 The syslog_ng Subsystem has been started. Subsystem PID is 155708. cscaixlgg059:root:/opt/soe/local/syslog_ng/etc>ps -ef|grep -i syslog root 155708 139370 0 05:43:40 - 0:00 /opt/soe/local/syslog_ng/sbin/syslog-ng cscaixlgg059:root:/opt/soe/local/syslog_ng/etc>lssrc -a|grep -i syslog syslog_ng local 155708 active I created the syslog_ng substem as below */bin/mkssys -s syslog_ng -p /opt/soe/local/syslog_ng/sbin/syslog-ng -a ' -F -f /opt/soe/local/syslog_ng/etc/syslog-ng.conf ' -u 0 -S -n 15 -f 9 -G local * Can someone please advice? Thanks & Regards, Rakesh

2 1

follow_freq() On A Static File
by Hidayath Basha 09 Mar '11

09 Mar '11

Hello, Is it fine to apply follow_freq(1) on a static file (not tailing one) Or One should use follow_freq() only on a tailing file? Thanks and regards, Hidayath

2 1

log4j
by Peter Czanik 08 Mar '11

08 Mar '11

Hello, As I don't code in Java, I don't have any personal experiences with log4j. But from e-mail and forum messages, and also from discussions at FOSDEM I know, that log4j is often used together with syslog-ng. I ran into this log4j related question this morning: http://serverfault.com/questions/242972/getting-syslog-ng-to-recognize-a-ja… I'd like to ask, if there is anybody who already has a solution for the above question or any other useful tips about log4j & syslog-ng to share. Thanks, bye, -- Peter Czanik (CzP) <czanik(a)balabit.hu> BalaBit IT Security / syslog-ng upstream http://czanik.blogs.balabit.com/

2 1

code-commenting log messages
by Peter Gyongyosi 07 Mar '11

07 Mar '11

Hi, most of the pattern database-related projects I've seen here started off from the log side of the whole issue: investigating the messages an application produces and trying to create patterns based on them (manually or via patternize). Of course, this makes perfect sense if you're an administrator or you're trying to analyze logs from already-written applications, but what if you're the one who writes these applications and you'd like to make them easier to analyze later on? What if you're the extremely nice guy who wants to ship an up-to-date pattern database for syslog-ng along with your application? :) Or, to be more realistic, at least a list of the most important log messages your program can emit that the user can search to find out what an actual log message means. The idea I've been toying with is defining some kind of a standardized commenting scheme for logging-related calls in source code. Using this, the code could be parsed automatically and a list of log messages could easily be generated that could be used later for documentation or even generating a pattern database. I've done multiple rounds of investigation but I only found custom, application-specific solutions for this. So on one hand, I'm asking you if you've seen any widespread commenting schemes for log messages that can fulfil this task? Or even a custom, application-specific solution that is so incredibly cool that we should all adopt it instead of reinventing the wheel? :) Until then, I'm proposing a commenting scheme I think would be usable and I'm asking for your feedback. It is based on Javadoc, which is widespread enough to be familiar for the majority of developers and has mutations available for virtually all programming languages. I'm going to show the concept through the now-standard example of the logging of the login in sshd: /** * @log * * Signals that the authentication has succeeded or failed for a user. * * @class system * @tags useracct, secevt * @url http://www.openssh.com * * @example Accepted password for bazsi from 127.0.0.1 port 48650 ssh2 * @example Failed password for bazsi from 127.0.1.1 port 44637 ssh2 * * @pattern Accepted @ESTRING:usracct.authmethod: @for @ESTRING:usracct.username: @from @ESTRING:usracct.device: @port @ESTRING:: @@ANYSTRING:usracct.service@ * @pattern Failed @ESTRING:usracct.authmethod: @for @ESTRING:usracct.username: @from @ESTRING:usracct.device: @port @ESTRING:: @@ANYSTRING:usracct.service@ * * @regexp (Accepted|Failed) (gssapi(-with-mic|-keyex)?|rsa|dsa|password|publickey|keyboard-interactive/pam|hostbased) for [^[:space:]]+ from [^[:space:]]+ port [[:digit:]]+( (ssh|ssh2))?$ */ authlog("%s %s for %s%.100s from %.200s port %d%s", authmsg, method, authctxt->valid ? "" : "invalid user ", authctxt->user, get_remote_ipaddr(), get_remote_port(), info); To formalize it a bit: - comments for the log messages should follow the standard Javadoc/Doxygen formatting, with the main indentifier token being "@log" (or \log if you prefer, all further parameters should be usable in the backslash form as well) - the allowed parameters are @class, @url, @example, @regexp, @pattern and @tags - multiple @example, @pattern and @regexp fields are allowed - it is mandatory to specify either @example, @pattern or @regexp to give the application some kind of an identifier, all other params can be omitted I have my own concerns, though: - Is it even the right thing to do to spam source code like this? Is it the proper place for the semantic description of log messages? Sure, it's the easiest place to add such things while the app is being developed and if the comment's right there in the code it's more likely that any changes will be followed up during an update of the code, but still, it seems a bit of an overkill even for me now that I look at it... But maybe it'd worth it for the most important messages. - Maybe and @id would be useful to allow for manually specifying an identifier for the log messages (we all love Oracle's error codes, don't we? :)) - What should be done if a single log() call can emit multiple different messages with different meanings that should be getting their own separate entry? The example above is the perfect candidate for this: the "login accepted" and "login failed" events are definitely different. Maybe two separate @log comments could be added in this case. - How does it relate to more advanced db-parser features like correlation or the self-testing ability of pdbtool of name-value pairs in the <examples> section? (- Isn't it a bit too push-y to call it @pattern instead of @syslog-ng-pattern? :)) However, if such a scheme could be adopted, with some little hacking several cool things could be done (which I'm willing to do): - create "pdbtool sourceextract" or something like that that can generate ready-to-use pattern databases from the source code - this can be added to the build processes to update them along with manpages and similar stuff - Doxygen & co. could be patched to parse and display them - vim, Eclipse and Your Favourite Editor could get a one-click way to add such comment blocks, just as it is possible to add header comments easily What do you think? greets, Peter

3 2

cisco/squid feedback
by Alexander Clouter 07 Mar '11

07 Mar '11

Hi, Digging around, I could not find anything on how to 'sensibly' log the junk that Cisco IOS devices (and their infernal WLC) spit out plus I was keen to rewrite squid HTTP proxy server logs to make use of the epoch+msec timestamp found in the MSG. I have documented, very roughly, my current solution on my website and would welcome amendments/fixes/flames/etc on the approach: http://www.digriz.org.uk/syslog-ng-integration Before you ask, I use 'match("fqdn.example.com" value("HOST_FROM") ...)' as netmask() is broken for IPv6 :) Cheers -- Alexander Clouter .sigmonster says: When pleasure remains, does it remain a pleasure?

2 5

[PATCH]: support setting a default suppress() globally
by Gergely Nagy 04 Mar '11

04 Mar '11

Hi! Attached is a patch that makes it possible to set a default value for suppress() in the global options block, which will then be inherited by all drivers that support this option. Of course, it can still be overridden on a per-destination basis. What this means is, that if one has a larger number of destinations that all need the same suppress settings, we'd write that like this before this patch: destination d_1 { file("/var/log/messages-1" suppress(10)); }; destination d_2 { file("/var/log/messages-2" suppress(10)); }; ... destination d_N { file("/var/log/messages-N" suppress(10)); }; destination d_XX { file("/var/log/messages-XX" suppress(5)); }; After the patch, this can be shortened to: options { suppress(10); }; destination d_1 { file("/var/log/messages-1"); }; ... destination d_N { file("/var/log/messages-N"); }; destination d_XX { file("/var/log/messages-XX" suppress(5)); }; Both result in exactly the same behaviour, but the second is a lot shorter, and easier to maintain, as long as the majority of destinations need the same suppress() setting. Hopefully people will find this useful. The patch is also available from the work/global-suppress branch of my tree at git://git.balabit.hu/algernon/syslog-ng-3.3.git -- |8]

2 4

Reading Multiple Files Using Regex
by Hidayath Basha 04 Mar '11

04 Mar '11

Hi, I have multiple files... some-host:/some/path $ ls -l -rw------- 1 root root 90K Mar 3 11:37 mylogs-2011-03-03 -rw------- 1 root root 121 Mar 4 11:40 mylogs-2011-03-04 Can I read these files in the source driver with following source driver? source s_files{ file("/some/path/mylogs*"); }; I use SLES 11.0 and syslog-ng 2.0.9 Thanks and regards, Hidayath Saven Technologies: A leader in technology driven business solutions ---------------------------------------------------------------------------- ----------------------------------- #302, 3rd Floor, MyHome Sarovar Plaza, Hyderabad- 500 063, India Off: +91 040 23237303/7304 ext 115 email: hidayath.basha(a)saventech.com | www.saventech.com follow me: http://twitter.com/hidayath26 ---------------------------------------------------------------------------- -----------------------------------

2 1

patterndb context-timeout problem
by Zoltán Pallagi 04 Mar '11

04 Mar '11

Hi, I've found a problem in handling context-timeout in a special case. If context-timeout=10 (10 sec) and I am using a trigger, where trigger="timeout", it can happen that the timeout won't expire: I sent the following two messages to syslog-ng before expiring timeout (but the difference between the two timestamp is bigger than the timeout value): "<38>1990-01-01T14:45:25 customhostname program6[1234]: program6 testmessage" "<38>1990-01-01T14:55:25 customhostname program6[1234]: program6 testmessage" I receive the timeout message immediately, when the second message arrives (it's ok), but there is no timeout for the second message.

2 1

Message correlation, everything needs an "@"? And a grep problem.
by Valentijn Sessink 04 Mar '11

04 Mar '11

Hello list, After playing with correlation for a while, here are a few remarks on it's workings. I have played with syslog-ng 3.2.2, built from source on an Ubuntu 10.04 machine. For some reason, the "@1", "@2"... etcetera specifiers needed to be all over the place to be able to see any variable at all. For example, the rule below describes the last message of a Postfix log trail. Note that even the context-id is specified as "${queueid}@1". All other references also have an "@" sign. Without the @, the variables seem just empty. <rule id='X' context-id="${queueid}@1" context-scope="host" provider="openoffice" class="maildelivery"> <patterns> <pattern>@ESTRING:queueid: @removed</pattern> </patterns> <actions> <action> <message> <values> <value name="MESSAGE">${queueid}@1 mail to=<${toaddress}@2> from=<${pffrom}@3> connection from=${pfremotehost}@5[${pfremoteip}@5] msgid=<${msgid}@4> orig_to=<${origto}@2> delay=${deliverystatus}@2</value> </values> </message> </action> </actions> </rule> Is this how it's meant to be? Then, when trying to use the "grep" template function in this message, I can't seem to find out how it works. Using the exact example from the documentation, $(grep ("${username}" == "root") ${auth_method}), I only get an error: Error parsing filter expression, syntax error, unexpected $undefined in <string> at line 1, column 3: (${ ^ Trying to alter the message, using "$something" == "value", doesn't seem to have an effect, i.e. I can't get "grep" to find any values. In fact, I'm not even sure if I can use the names from the (E/Q)STRING parsers right away, or if I need to have <value name="something"> tags before I can grep for them. In fact, most of the time, the grep I would like to use is something like: $(grep ( "$variable" != "" ) "$variable"). But whatever I try, no results from grep. Could anyone please clarify? Best regards, Valentijn

2 15

Permission Denied (13)
by Hidayath Basha 04 Mar '11

04 Mar '11

Hi, we are using SLES 11 We are trying to read a tailing file and send it to the server syslog-ng.conf (on client 192.168.18.10 some-host) #syslog-ng.conf options { long_hostnames(off); sync(1); perm(0640); stats(3600); }; source s_mysrc{ file("/some/file.log" ); }; destination d_mydest { tcp("192.168.18.7" port(5140)); }; log { source(s_mysrc); destination(d_mydest); }; #syslog-ng.conf ends syslog-ng.conf (on Server 192.168.18.7) #syslog-ng.conf options { long_hostnames(off); sync(0); perm(0640); stats(3600); }; source s_mysrc{ tcp(ip("192.168.18.7") port(5140) keep-alive(yes)); }; destination d_mydest { file ("/var/log/hosts/$YEAR-$MONTH/$HOST/$FACILITY-$YEAR-$MONTH-$DAY" owner(root) group(root) perm(0600) dir_perm(0700) create_dirs(yes) ); }; log { source(s_mysrc); destination(d_mydest); }; #syslog-ng.conf ends When I start the service at client, I'm getting the following error Starting syslog servicesError opening file for reading; filename='/some/file.log', error='Permission denied (13)' Error initializing source driver; source='s_mysrc' startproc: exit status of parent of /sbin/syslog-ng: 2 The file /some/file.log has -rw-rw-rw- permissions When I modify the source driver to ... file("/some/file.log" follow_freq(1)); I'm able to start the syslog service, but I don't see anything on the server (192.162.18.7) at /var/log/hosts/2011-03/some-host/ Please help Thanks and regards, Hidayath Saven Technologies: A leader in technology driven business solutions ---------------------------------------------------------------------------- ----------------------------------- #302, 3rd Floor, MyHome Sarovar Plaza, Hyderabad- 500 063, India Off: +91 040 23237303/7304 ext 115 email: hidayath.basha(a)saventech.com | www.saventech.com follow me: http://twitter.com/hidayath26 ---------------------------------------------------------------------------- -----------------------------------

2 3

[Bug 117] New: syslog-ng.persist should not reside in /var/lib
by bugzilla＠bugzilla.balabit.com 03 Mar '11

03 Mar '11

https://bugzilla.balabit.com/show_bug.cgi?id=117 Summary: syslog-ng.persist should not reside in /var/lib Product: syslog-ng Version: unspecified Platform: PC OS/Version: Linux Status: NEW Severity: normal Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: wlt(a)obsidian-studios.com Type of the Report: --- Estimated Hours: 0.0 syslog-ng uses a syslog-ng.persist file that is created in /var/lib. Per FHS 2.3 that should exist in a subdir, if not one specific to syslog-ng, then /var/lib/misc. http://www.pathname.com/fhs/pub/fhs-2.3.html#VARLIBVARIABLESTATEINFORMATION An application (or a group of inter-related applications) must use a subdirectory of /var/lib for its data. There is one required subdirectory, /var/lib/misc, which is intended for state files that don't need a subdirectory; the other subdirectories should only be present if the application in question is included in the distribution. [38] [38] An important difference between this version of this standard and previous ones is that applications are now required to use a subdirectory of /var/lib. This causes problems on diskless systems with a shared /var/lib. Subdirs that contain unique per system data are not shared, and instead mounted on top of the shared /var/lib. I could do the same thing even for a single file, syslog-ng.persist in this case. However given FHS, it should be written in a subdir. Which I can then make that unique per system, like other subdirs in /var/lib that are not shared amongst all diskless systems. -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 2

internal-sftp and logs files
by Naira Kaieski 03 Mar '11

03 Mar '11

Good afternoon, Staff set up openssh to direct users to a certain group members to a chroot environment and these users will have access only to the server using sftp protocol. Put in the sshd_config file: Match Group customers ChrootDirectory% h ForceCommand internal-sftp-l VERBOSE f-AUTH Thus each user is directed to the chroot environment indicated in the variable% h (home directory defined in / etc / passwd) An example of directory is: User: naira Home directory: /var/www/naira.com.br The problem is that I am not able to capture logs of the user group "clients" that are targeted to the chroot environment. Access via internal-sftp from other users who do not belong to the "client" I get the logs in auth.log files. I just create dir and file /var/www/naira.com.br/dev/log but i not had log files... I'm using syslog-ng. Has anyone ever made this kind of setup? Thanks, -- Naira Kaieski Nucleo de Internet/Redes - Faccat Linux Professional Institute - LPI000223834

2 2

linking changes
by Balazs Scheidler 03 Mar '11

03 Mar '11

Hi, I've tried to put a closure on how syslog-ng 3.2 and above is being linked. There were numerous problems as it started to get into Linux distributions. This patch documents the intent with --enable-mixed-linking and how that behaves wrt the executables & shared libraries that syslog-ng installs. commit f2c7f3ea16d81e82b79582d3f2fe3882add4eb23 Author: Balazs Scheidler <bazsi(a)balabit.hu> Date: Wed Feb 23 13:01:02 2011 +0100 configure.in: reorganize LIBS variables so that unit tests/programs can be linked in mixed mode too Previously if --enable-mixed-linking was specified, unit tests and other tools (like pdbtool) couldn't be linked properly. The situation with --enable-mixed-linking is: * the primary aim here is to have a syslog-ng executable that can be started early in the boot process, and not a set of syslog-ng executables that can be distributed to different hosts that lack the required dependencies. * syslog-ng contains all static libs required by libsyslog-ng.so * therefore libsyslog-ng.so cannot be used outside the syslog-ng binary (except if the executable is linked against the required libs) * tools like pdbtool will be linked dynamically * unit tests will be linked dynamically too This patch cleans up and documents the purpose of various _DEPS_LIBS variables and updates Makefile.am files to use the updated names. A static GLib detection bug was also fixed. Reported-By: Arkadiusz Miśkiewicz <arekm(a)maven.pl> Signed-off-by: Balazs Scheidler <bazsi(a)balabit.hu> A related change is that the shared library libsyslog-ng-patterndb.so that was introduced in 3.2.2 will be dropped. Also, the main syslog-ng executable is not linked against libssl/libcrypto on purpose, it is only used by the afsocket module (which implements the tcp/syslog destinations). Could anyone who is in charge of maintaining syslog-ng binaries in various distributions give this a test so 3.2.3 will compile everywhere out-of-the-box? Thanks in advance, -- Bazsi

4 29

patternize and IPv6 hosts
by Valentijn Sessink 02 Mar '11

02 Mar '11

Hi, Just a small remark. If you run pdbtool with "patternize" on a log file with logs from an IPv6 host, pdbtool thinks that everything after the first ":" is the log message. For example, the log message: Feb 24 11:39:26 2a01:4f8:8a0:5141::3c2 named[31090]: lame server resolving .... ... will be patternized as if the logging host were "2a01" and if the pattern should be: <pattern>4f8:8a0:5141::3c2 named[31090]: lame server resolving ....</pattern> I searched around in the patternize code, but could not find the particular code where the host got cut off. However, for someone a bit at home in this code, I think the fix should be trivial. Best regards, Valentijn

3 6

afsql dbi initialization errors checks
by Hendrik Visage 02 Mar '11

02 Mar '11

Hi there, Busy compiling (more like porting) syslog-ng 3.2 to HP-UX 11.31, and I've been hitting a few snags that I since over come, but now I hit one where I see the source code is not checking whether the afsql got a proper DBI initialization, nor whether the actual driver is available before starting the db thread/etc. I've modified the lines 988-990 below, where they were just a simple "dbi_initialize();". On my setup, it "initialized", but with 0 drivers available :( (No, I DO have installed the Postgrsql dbi-driver, so yes I'll be debugging the DBI side now :() 986 if (!dbi_initialized) 987 { 988+ msg_error("HVT: dbi_initialization", 989= evt_tag_int("status",dbi_initialize("/opt/hvsyslog-ng3/lib/dbd") ), 990+ NULL); 991 dbi_initialized = TRUE; 992 } I also would like syslog-ng to a check at startup time and to rather exit with an error if the requested DBI driver is not available, than to wait till the messages starts to stream in and then to throw screens full of errors :( Hendrik

2 3

[Bug 116] New: Gentoo QA warning: app-admin/syslog-ng-3.1.4: logpatterns.c: 220: warning: implicit declaration of function ‘atoi’
by bugzilla＠bugzilla.balabit.com 02 Mar '11

02 Mar '11

https://bugzilla.balabit.com/show_bug.cgi?id=116 Summary: Gentoo QA warning: app-admin/syslog-ng-3.1.4: logpatterns.c:220: warning: implicit declaration of function ‘atoi’ Product: syslog-ng Version: 3.1.x Platform: PC OS/Version: Linux Status: NEW Severity: minor Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: mrdanwallis(a)gmail.com Type of the Report: bug Estimated Hours: 0.0 When installing the app-admin/syslog-ng-3.1.4 package on my Gentoo system, I received the following notification: * QA Notice: Package has poor programming practices which may compile * fine but exhibit random runtime failures. * logpatterns.c:220: warning: implicit declaration of function ‘atoi’ * Please do not file a Gentoo bug and instead report the above QA * issues directly to the upstream developers of this software. * Homepage: http://www.balabit.com/products/syslog_ng/ * ERROR: app-admin/syslog-ng-3.1.4 failed: * install aborted due to poor programming practices shown above Looking through the build output, here is the warning in context: i686-pc-linux-gnu-gcc -std=gnu99 -DHAVE_CONFIG_H -I. -I.. -I/usr/include/glib-2.0 -I/usr/lib/glib-2.0/include -I/usr/include/eventlog -D_GNU_SOURCE -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -O2 -march=i686 -pipe -Wall -c apphook.c i686-pc-linux-gnu-gcc -std=gnu99 -DHAVE_CONFIG_H -I. -I.. -I/usr/include/glib-2.0 -I/usr/lib/glib-2.0/include -I/usr/include/eventlog -D_GNU_SOURCE -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -O2 -march=i686 -pipe -Wall -c radix.c i686-pc-linux-gnu-gcc -std=gnu99 -DHAVE_CONFIG_H -I. -I.. -I/usr/include/glib-2.0 -I/usr/lib/glib-2.0/include -I/usr/include/eventlog -D_GNU_SOURCE -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -O2 -march=i686 -pipe -Wall -c logpatterns.c logpatterns.c: In function ‘log_classifier_xml_start_element’: logpatterns.c:220: warning: implicit declaration of function ‘atoi’ i686-pc-linux-gnu-gcc -std=gnu99 -DHAVE_CONFIG_H -I. -I.. -I/usr/include/glib-2.0 -I/usr/lib/glib-2.0/include -I/usr/include/eventlog -D_GNU_SOURCE -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -O2 -march=i686 -pipe -Wall -c cfg-lex.c cfg-lex.c:1562: warning: ‘input’ defined but not used i686-pc-linux-gnu-gcc -std=gnu99 -DHAVE_CONFIG_H -I. -I.. -I/usr/include/glib-2.0 -I/usr/lib/glib-2.0/include -I/usr/include/eventlog -D_GNU_SOURCE -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -O2 -march=i686 -pipe -Wall -c cfg-grammar.c -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 1