March 2011 - syslog-ng - lists.balabit.hu

logging stopped
by Len Conrad 15 Mar '11

15 Mar '11

uname -a FreeBSD 7.0-RELEASE syslog-ng --version syslog-ng 2.0.10 change date on syslog-ng.conf is "Apr 20 2009" been running untouched for at least that long. about 00:20 today Friday, all syslogging to syslog-ng stopped. chkrootkit shows nothing wrong stop syslog-ng then pkg_delete, and then cd /usr/ports/sysutils/syslog-ng2 make && make install start it, no change I rebooted the syslog server. no change trafshow -i bce0 -n then filter 514 shows 100KBs arriving from our syslog clients. df shows plenty of disk space for /var suggestions? Len

2 1

Re: [syslog-ng] afsql module error
by Eshan Bhide 15 Mar '11

15 Mar '11

Updated to the latest 3.3.0alpha2 build, libdbi-drivers and latest ivykis from git, and after much messing with the source I got it to compile and install. Now I get the error: "The database access library (DBI) reports no usable SQL drivers, perhaps DBI drivers are not installed properly; Error initializing dest driver; dest='d_sql', id='d_sql#0' Error initializing message pipeline;" I'm pulling my hair out at this point - could I get some help here please :) ? On 15 March 2011 14:13, Eshan Bhide <ebhide(a)globalgossip.net> wrote: > Here's another try: > > http://pastebin.com/7TV0nHzD > > On 14 March 2011 21:54, Hendrik Visage <hvjunk(a)gmail.com> wrote: >> >> >> On Mon, Mar 14, 2011 at 6:37 AM, Eshan Bhide <ebhide(a)globalgossip.net> >> wrote: >>> >>> Hi Hendrik, >>> Still having this problem of "No such DBI driver; type='mysql'" >>> >>> Strace is attached at: >>> http://pastebin.com/qaJZ2ZbE >> >> I see libdbi getting loaded, but what I am missing in that strace, is the >> libdbi searching for it's drivers.... >> >> >> > ------------------------------------------------- IMPORTANT NOTICE : The information in this email is confidential and may also be privileged. If you are not the intended recipient, any use or dissemination of the information and any disclosure or copying of this email is unauthorised and strictly prohibited. If you have received this email in error, please promptly inform us by reply email or telephone. You should also delete this email and destroy any hard copies produced.

2 1

Some messages are missing
by Tinnis G 14 Mar '11

14 Mar '11

Hi, I am not getting all the messages in the message log files . If someone helps me , it will be really appreciated. We have remote servers , where we are keeping the log files too. I feel that the problem is in the filter. Please correct me. I want all the logs will be reported . Mentioning below the configuration file on each server. These are all same throughout. Thanks in advance. _________________________________________________________ options { sync (0); time_reopen (10); log_fifo_size (1000); long_hostnames(off); use_dns (yes); use_fqdn (no); create_dirs (yes); keep_hostname (yes); stats (3600); }; source src { unix-stream("/dev/log"); internal(); pipe("/proc/kmsg"); }; # Remote logging ################ ## Auth log destination loghost1 { tcp("log1.xx.org" port(514)); }; destination loghost2 { tcp("log2.xx.org" port(514)); }; destination loghost3 { tcp("log3.xx.org" port(514)); }; filter f_auth { facility(auth); }; log { source(src); filter(f_auth); destination(loghost1); }; log { source(src); filter(f_auth); destination(loghost2); }; log { source(src); filter(f_auth); destination(loghost3); }; # ## Authpriv log destination loghost1 { tcp("log1.xx.org" port(514)); }; destination loghost2 { tcp("log2.xx.org" port(514)); }; destination loghost3 { tcp("log3.xx.org" port(514)); }; filter f_authpriv { facility(auth, authpriv); }; log { source(src); filter(f_authpriv); destination(loghost1); }; log { source(src); filter(f_authpriv); destination(loghost2); }; log { source(src); filter(f_authpriv); destination(loghost3); }; ## Everything log destination loghost1 { tcp("log1.xx.org" port(514)); }; destination loghost2 { tcp("log2.xx.org" port(514)); }; destination loghost3 { tcp("log3.xx.org" port(514)); }; filter f_everything { level(debug..emerg); }; log { source(src); filter(f_everything); destination(loghost1); }; log { source(src); filter(f_everything); destination(loghost2); }; log { source(src); filter(f_everything); destination(loghost3); }; # # Local logging ################ # Local Destinations #%%%%%%%%%%%%%%%%%%% destination authpriv { file("/var/log/authpriv.log"); }; destination auth { file("/var/log/auth.log"); }; destination syslog { file("/var/log/syslog.log"); }; destination messages { file("/var/log/messages.log"); }; destination cron { file("/var/log/cron.log"); }; destination daemon { file("/var/log/daemon.log"); }; destination kernel { file("/var/log/kernel.log"); }; destination lpr { file("/var/log/lpr.log"); }; destination mail { file("/var/log/mail.log"); }; destination debug { file("/var/log/debug.log"); }; # # destination console { usertty("root"); }; destination console_all { file("/dev/tty12"); }; # Local Filters ############### filter f_syslog { not facility(authpriv, mail); }; filter f_messages { level(info..warn) and not facility(auth, authpriv, mail, news); }; filter f_authpriv { facility(auth, authpriv); }; filter f_auth { facility(auth); }; filter f_cron { facility(cron); }; filter f_daemon { facility(daemon); }; filter f_kernel { facility(kern); }; filter f_lpr { facility(lpr); }; filter f_mail { facility(mail); }; filter f_debug { not facility(auth, authpriv, news, mail); }; # # filter f_emergency { level(emerg); }; filter f_info { level(info); }; filter f_notice { level(notice); }; filter f_warn { level(warn); }; filter f_crit { level(crit); }; filter f_err { level(err); }; # Local Log ############ log { source(src); filter(f_syslog); destination(syslog); }; log { source(src); filter(f_messages); destination(messages); }; log { source(src); filter(f_authpriv); destination(authpriv); }; log { source(src); filter(f_auth); destination(auth); }; log { source(src); filter(f_cron); destination(cron); }; log { source(src); filter(f_daemon); destination(daemon); }; log { source(src); filter(f_kernel); destination(kernel); }; log { source(src); filter(f_lpr); destination(lpr); }; log { source(src); filter(f_mail); destination(mail); }; log { source(src); filter(f_debug); destination(debug); }; # log { source(src); filter(f_emergency); destination(console); }; log { source(src); destination(console_all); }; # ## END /etc/syslog-ng/syslog-ng.conf _________________________________________________________

3 2

Reading Log file
by Shawn Cannon 14 Mar '11

14 Mar '11

Can someone tell me what the numbers mean for processed and dropped? Am I logging everything to SQL or am I dropping some? Mar 9 17:12:03 syslog01 syslog-ng[1738]: Log statistics; processed='source(s_gms)=2279331', dropped='dst.sql(d_mssql#0,freetds, rawsql.adbcefg.net,1785,Syslog)=2213605', stored='dst.sql(d_mssql#0,freetds, rawsql.abcdefg.net,1785,Syslog)=0',

3 4

[PATCH]: Experimental SMTP destination
by Gergely Nagy 11 Mar '11

11 Mar '11

Hi! I've had a little free time, and started toying with the idea of implementing an SMTP destination for syslog-ng. So far, it's progressing nicely, but there are a few shortcomings, as I will explain later. For anyone interested, it can be found on the 'work/modules/afsmtp' branch of my syslog-ng-3.3 tree: git://git.balabit.hu/algernon/syslog-ng-3.3.git Or, one can browse the branch on the web: http://git.balabit.hu/?p=algernon/syslog-ng-3.3.git;a=shortlog;h=work/modul… One will need the libesmtp library installed for the module to work. It should be available in most distributions, but if one doesn't have it, it can be downloaded from http://www.stafford.uklinux.net/libesmtp/ (although, as of this writing, the site seems to be down). The usage is something like this: destination d_smtp { smtp( host("localhost") port(25) mail_from("syslog-ng alert service <noreply(a)example.com>") rcpt_to("Admin #1 <admin(a)example.com>") rcpt_to("Admin #2 <admin2(a)example.com>") subject("[ALERT] $LEVEL condition in $PROGRAM!") header("X-Program" "$PROGRAM") header("X-Facility" "$FACILITY") body("Hi!\nThe syslog-ng alerting service thinks you might be interested in the following message:\n $MSG\n-- \nsyslog-ng\n") ); }; One can specify any number of recipients, and the mail will be sent to all of them. Custom headers can also be specified, but at this time, a selected few headers do not work as one would expect, due to libesmtp handling them specially. These are: Date, From, To, Cc, Bcc, Reply-To - trying to set either of these will result in unspecified behaviour (ranging from mangled headers to crashes, in case of Date). I will fix these at a later point. They will probably be need to set via their own statements (header_date, header_from, etc, or something along those lines). Specifying mail_from or subject multiple times will override any previous values. Adding the same custom header multiple times will only add the first (though, plans include changing that, so that one can include the same header multiple times, with different values - if I can find a use-case for that). There are also a few other minor issues here and there, but nothing major, as far as I could check. Further plans include authentication & starttls support. libesmtp supports both, I just didn't have time to write the neccessary glue code yet. -- |8]

8 26

Destination MySQL for Syslog-ng 2.0.x
by Hidayath Basha 11 Mar '11

11 Mar '11

Hi, How can I collect the logs in MySQL db with syslog-ng 2.0.x? There is a destination 'sql' in syslog-ng 3+, but how can I do it with 2.0? Thanks and regards, Hidayath

3 2

Transmitting a Huge Log File
by Hidayath Basha 11 Mar '11

11 Mar '11

Hi all, I'm trying to transmit a huge log file (of about 80k lines) to a centralized syslog server over TCP But, on the syslog server, I'm receiver only the last part of the log file (of about 7000 lines) How can I transmit the whole log file On client, the syslog-ng.conf file ... destination d_mydest { tcp("192.168.18.7" port(5140)); }; source s_src { file("/some/huge/file.log" follow_freq(1) log_prefix("mylogs:")); }; log { source(s_src); destination(d_mydest); }; On server (192.168.18.7) , the conf file ... source s_mysrc{ tcp(ip("192.168.18.7") port(5140) keep-alive(yes)); }; filter f_filter{ match("mylogs:"); }; destination d_mydest { file ("/var/log/hosts/$HOST/$YEAR-$MONTH/$YEAR-$MONTH-$DAY/hugefile.log" owner(root) group(root) perm(0600) dir_perm(0700) create_dirs(yes) ); }; log { source(s_mysrc); filter(f_filter); destination(d_mydest); }; Thanks and regards, Hidayath

4 4

Dropped messages to MSSQL?
by Shawn Cannon 10 Mar '11

10 Mar '11

I really need help on this. I am collecting tons of SYSLOG data from over 500 firewalls. I have these syslogs going to a Barracuda Load Balancer first which will then send messages to 2 syslog-ng open source servers. Here is how I have my config file setup: @version: 3.0 #Default configuration file for syslog-ng. # # For a description of syslog-ng configuration file directives, please read # the syslog-ng Administrator's guide at: # # http://www.balabit.com/dl/html/syslog-ng-admin-guide_en.html/bk01-toc.html # options { use_dns(no); }; ###### # sources source s_gms { # message generated by Syslog-NG # internal(); # standard Linux log source (this is the default place for the syslog() # function to send logs to) # unix-stream("/dev/log"); # messages from the kernel #file("/proc/kmsg" program_override("kernel")); # messages destined to udp514 udp(ip(0.0.0.0) port(514)); }; source s_syslogng { internal(); }; ###### # destinations destination d_messages { file("/var/log/messages"); }; destination d_mssql { sql(type(mssql) host("rawsql.abcdefg.net") port("1785") username("username") password("password") database("Syslog") table("syslogng")columns("datetime varchar(16)", "host varchar(32)", "program varchar(32)", "pid varchar(8)", "message varchar(4096)") values("$R_DATE", "$HOST", "$PROGRAM", "$PID", "$MSGONLY") indexes("datetime", "host", "program", "pid")); }; log { source(s_gms); destination(d_mssql); }; log { source(s_syslogng); destination(d_messages); }; When messages are coming in, I am showing the following in the local syslog-ng messages: Log statistics; processed='source(s_gms)=2155636', dropped='dst.sql(d_mssql#0,freetds,rawsql.abcdefg.net,1785,Syslog)=1717472', stored='dst.sql(d_mssql#0,freetds,rawsql.abcdefg.net,1785,Syslog)=0', I assume this is telling me that I am dropping the majority of my messages instead of them getting inserted into my MS SQL database? The MS SQL Database runs on a very beefy server with plenty of memory. I am trying to determine why this is being dropped. Please help as I am huge newbie when it comes to syslog-ng. Thanks!

5 14

syslog-ng 3.3.0alpha2 released
by Balazs Scheidler 10 Mar '11

10 Mar '11

Hi, I'm happy to announce that I've just uploaded syslog-ng 3.3.0alpha2 to our website. It contains a couple of important bugfixes and some new features compared to 3.3.0alpha1. It worked for me quite well, and it endured the first round of testing by the BalaBit syslog-ng PE team, when they've started working with the code. Highlights ========== syslog-ng 3.3 has a couple of new and interesting features, already in this alpha release: * It uses multiple threads to do its work, thus capable of scaling to multiple CPUs. Add 'threaded(yes)' to your option block and you're good to go. * Built in mongodb destination for folks who prefer NoSQL over traditional SQL. * This version has a new and improved "pdbtool patternize" functionality More exciting stuff is coming. Known issues ============ I know for certain that it will not work on FreeBSD unless the following environment variable is set: IV_EXCLUDE_POLL_METHOD=kqueue Otherwise syslog-ng will crash at startup, immediately. Stability ========= Certainly you wouldn't want to run your production system on this version, however this code base had endured a couple of test rounds, so if it compiles and starts up for you (which can be a challenge I can admit), then the basic functions will most certainly work. Changes since 3.3.0alpha1 ========================= 3.3.0alpha2 Thu, 10 Mar 2011 12:51:36 +0100 Features: ======== * Support for systemd activation added. * Add support for customizable token delimiters in "pdbtool patternize", which makes the resulting patterns to have much better quality. * Added support for a --no-parse command line option for "pdbtool patternize" to avoid parsing the input as normal syslog messages. * Added a new flag 'ignore-errors' to LogWriter based destinations (file, pipe, tcp, udp, syslog) * Added support for specifying the suppress() option globally. Bugfixes: ========= * Fixed a tcp()/udp()/syslog() destination driver issue that caused aborts during startup. * Handle non-epollable devices like /dev/null normally by failling back to the regular file-like handling. Earlier these caused an abort inside ivykis. * Makefile fixes to make it possible to compile syslog-ng from a "make dist" tarball. * Added error messages about libdbi initialization failures, that would prevent the SQL destination from working. * Make it possible to compile against PCRE not in a standard location (caused compilation failures on FreeBSD). * Fixed several $(grep) related bugs: - not to cause a segmentation faule when the filter expression supplied contains syntax errors - the template parsing code removed quotes required by filter expressions, with the current change it is possible to use quotes in the filter expressions directly: $(grep ('$username' == 'root') $MSG) - fixed the handling of template function invocations within correllation that had multiple messages in its context. * Fixed a segmentation fault in pdbtool match --debug-pattern in case there was no matching rule. * Fixed "pdbtool test". * Added a new macro named $CONTEXT_ID that expands to the current context-id in correllated rules. Other changes: ============== * Changed the default syslog-ng.conf version number to 3.3 to match the current version. * The pipe() destination used to override the default value of flush_lines() by explicitly setting it to 0. This behaviour has been removed, now the pipe() driver will properly use the default, unless overridden explicitly. * Ported the 3.2 linking changes to 3.3, which means that: - libsyslog-ng-patterndb.so is gone, both the command line pdbtool command and the patterndb plugin (libpatterndb.so) links its contents statically, to improve portability on Cygwin. - unit tests & command line utilities link properly in --enable-mixed-linking mode * Added debug messages in the correllation code to make it easier to debug correllation rules. * Added debug/verbose options to pdbtool. Credits: ======== Code, bugreports, testing, documentation suggestions and other improvements were contributed by: * Arkadiusz Miśkiewicz (PLD Linux) * Balazs Scheidler (BalaBit) * Balint Kovacs (BalaBit) * Corinna Vinschen (RedHat) * Dalibor Toman (fortech.cz) * Gergely Nagy (BalaBit) * Laszlo Boszormenyi (lsc.hu) * Marius Tomaschewski (Novell) * Peter Czanik (BalaBit) * Peter Gyongyosi (BalaBit) * Tom Gundersen (jklm.no) * Valentijn Sessink (blub.net) * Zoltan Pallagi (BalaBit) -- Bazsi

1 1

PostgreSQL, MonitorWare Schema and log presentation/analysis
by Hendrik Visage 10 Mar '11

10 Mar '11

Hi there, I've searched hi and low for a log analyzer/presenter for PostgreSql backend (still looking) but keep bumping into stuff like LogAnalyzer wanting a MonitorWare Schema.. Searching for that with syslog-ng, I've been unsuccessful (or I am missing something), other than getting the SQL schema below on a chinese site. Now my quetion: Anybody else used syslog-ng to pump directly into a PostgreSQL db a Monitorware Schema? Any other place I can get some more information on this schema usage? Anybody using PostgreSQL as a storage back end with a Web based GUI? (Before we enter the MySQL vs PostgreSQL debates, the answer is simple: we know postgresql, we don't know mysql, and if we hit performance issues in the current environment, we know we will need to procure the necessary HW and handle those decisions there) CREATE TABLE SystemEvents ( ID int unsigned not null auto_increment primary key, CustomerID bigint, ReceivedAt datetime NULL, DeviceReportedTime datetime NULL, Facility smallint NULL, Priority smallint NULL, FromHost varchar(60) NULL, Message text, NTSeverity int NULL, Importance int NULL, EventSource varchar(60), EventUser varchar(60) NULL, EventCategory int NULL, EventID int NULL, EventBinaryData text NULL, MaxAvailable int NULL, CurrUsage int NULL, MinUsage int NULL, MaxUsage int NULL, InfoUnitID int NULL , SysLogTag varchar(60), EventLogType varchar(60), GenericFileName VarChar(60), SystemID int NULL ); CREATE TABLE SystemEventsProperties ( ID int unsigned not null auto_increment primary key, SystemEventID int NULL , ParamName varchar(255) NULL , ParamValue text NULL );

2 1