- syslog-ng - lists.balabit.hu

[PATCH] tfhash: New template function to calculate the SHA1 of its input
by Peter Gyongyosi 12 Aug '12

12 Aug '12

tfhash: New template function to calculate the SHA1 of its input This patch adds a new template function called $(hash) which can be used to calculate the SHA1 hash of its inputs. As it uses the SHA1() function from OpenSSL, it is only available if syslog-ng is compiled using --enable-ssl. One use case for it can be when you'd like to anonymize sensitive parts of the log message (eg. the user name) you parsed out using patterndb before storing or forwarding it for further analysis but would like to retain the ability to correlate messages along this value. Beware though that such simple anonymization can be broken if there's enough data available, see http://arxiv.org/pdf/cs/0610105v2.pdf for references and examples. Thanks for the original idea to Martin Grauel. Signed-off-by: Peter Gyongyosi <gyp(a)balabit.hu> --- [see the patch itself at https://github.com/gyp/syslog-ng-3.4/commit/5c6e97d94694eef2aa76fa61a1c8928… ] <https://github.com/gyp/syslog-ng-3.4/commit/5c6e97d94694eef2aa76fa61a1c8928…>

5 12

lib/ivykis submodule pointing to stale commit?
by Conrad Hoffmann 12 Aug '12

12 Aug '12

Hi folks, apparently the commit that lib/ivykis submodule is set to (syslog-ng-3.4, master branch) does not exist in the tree anymore? GitHub actually still shows it, but it's not checked out: (See lines beginning with *) dionysos:~/tmp $>git clone https://github.com/balabit/syslog-ng-3.4.git Cloning into 'syslog-ng-3.4'... remote: Counting objects: 13157, done. remote: Compressing objects: 100% (2834/2834), done. remote: Total 13157 (delta 10256), reused 13100 (delta 10213) Receiving objects: 100% (13157/13157), 3.18 MiB | 1.12 MiB/s, done. Resolving deltas: 100% (10256/10256), done. dionysos:~/tmp $>cd syslog-ng-3.4/ dionysos:~/tmp/syslog-ng-3.4 $>git submodule init Submodule 'lib/ivykis' (git://github.com/buytenh/ivykis.git) registered for path 'lib/ivykis' Submodule 'modules/afmongodb/libmongo-client' (git://git.balabit.hu/bazsi/libmongo-client.git) registered for path 'modules/afmongodb/libmongo-client' dionysos:~/tmp/syslog-ng-3.4 $>git submodule update Cloning into 'lib/ivykis'... remote: Counting objects: 3678, done. remote: Compressing objects: 100% (917/917), done. remote: Total 3678 (delta 2764), reused 3635 (delta 2732) Receiving objects: 100% (3678/3678), 1.06 MiB | 682 KiB/s, done. Resolving deltas: 100% (2764/2764), done. *fatal: reference is not a tree: 73c0c27a19755f700b268081a5a59acdd04b421e Cloning into 'modules/afmongodb/libmongo-client'... remote: Generating pack... remote: Done counting 4658 objects. remote: Deltifying 4658 objects... remote: 100% (4658/4658) done remote: Total 4658 (delta 3123), reused 4585 (delta 3053) Receiving objects: 100% (4658/4658), 702.36 KiB | 1.17 MiB/s, done. Resolving deltas: 100% (3123/3123), done. Submodule path 'modules/afmongodb/libmongo-client': checked out '58f3814cad94bcd78216c7ac971c8435d17a9242' *Unable to checkout '73c0c27a19755f700b268081a5a59acdd04b421e' in submodule path 'lib/ivykis' I can't find a similar in the tree neither. Any ideas? Regards, Conrad

3 2

Re: [syslog-ng] Issue about using perl script in Receiving messages from external applications
by Evan Rempel 10 Aug '12

10 Aug '12

You are probably suffering from perl's buffering. I forget the exact syntax right now but a quick google of disabling Perl auto flush should give you what you need. Jian Zhu <zhujian0805(a)gmail.com> wrote: Hello, Could anyone please help me on this below issue? my syslog-ng.conf file look like below: The test.sh script work fine, I can see it's output in the test.sh.log file. But the test.pl<http://test.pl> doesn't work. the test.sh and test.pl<http://test.pl> have the same function of printing something to STDOUT. ------------------------------ ------------------------------------------------------------------------------------------------------------------- @version: 3.0 #Default configuration file for syslog-ng. # # For a description of syslog-ng configuration file directives, please read # the syslog-ng Administrator’s guide at: # # http://www.balabit.com/dl/html/syslog-ng-admin-guide_en.html/bk01-toc.html # options { # Number of syslog lines stored in memory before being written to files flush_lines (0); log_fifo_size (2048); create_dirs (yes); perm (0640); dir_perm (0750); }; source ppp { program("/root/perl/test.pl<http://test.pl>"); }; destination ppp { file ("/var/log/syslog-ng/$HOST/perl.pl.log" create_dirs(yes)); }; log { source(ppp); destination(ppp); }; source sss { program("/root/shell/test.sh"); }; destination sss { file ("/var/log/syslog-ng/$HOST/test.sh.log" create_dirs(yes)); }; log { source(sss); destination(sss); }; ------------------------------------------------------------------------------------------------------------------------------------------------- -- Thanks & Best Regards James Zhu Mobile: +86 186 8482 2818 Email: zhujian0805(a)gmail.com<mailto:zhujian0805@gmail.com>

3 3

[Bug 178] New: Spoofed source address bug introduced in 3.3.5
by bugzilla＠bugzilla.balabit.com 10 Aug '12

10 Aug '12

https://bugzilla.balabit.com/show_bug.cgi?id=178 Summary: Spoofed source address bug introduced in 3.3.5 Product: syslog-ng Version: 3.3.x Platform: PC OS/Version: Solaris Status: NEW Severity: major Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: marvin.nipper(a)stream.com Type of the Report: regression Estimated Hours: 0.0 Hi. Sorry if this is not the right way to do this. There was an email thread regarding this on 4/30, but I figured that the right way to do this way to simply open an official bug report. As this is my 1st one for syslog-ng, apologies if I screw it up! This all pertains to 3.3.5, on Solaris 10 x86. Per my original email, changes made between 3.3.4 and 3.3.5 have caused a situation wherein forwarded UDP packets, with spoofed source addressing, all revert to null addresses, the instant that a HUP is issued against syslog-ng (e.g. during log rotation). Also, as per that email, with Gergely's excellent (and educated) guess, I backed out the patch noted here: http://git.madhouse-project.org/debian/syslog-ng/patch/?id=a898014482f733e9…, and the result was that the null addressing issue was resolved. Obviously, as there were (no doubt) other reasons for that patch, I'm just wanting to be sure that this problem doesn't get lost in the shuffle, and that the patch gets reworked (to achieve its originally desired results, while managing to also resolve the null address problem that it caused with spoofing). Let me know if you need any further information regarding this problem. And, as always, I'm MORE than happy to be a test guinea pig, if need be. THANKS. -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 21

syslog-ng upgrade
by James McDonald 10 Aug '12

10 Aug '12

Does anyone know what needs to be done to successfully upgrade from 4.1 to 4.2 (F2)? I tried simply stopping the service and running 'rpm -ivh syslog-ng..rpm -force' but it did not complete successfully. I added the -force command because it was complaining of file conflicts. Balabit has requested I upgrade to this version, as they try to help me figure out my other problem I'm having with the syslog-ng service, but I was unable to find any documentation or get a response from Balabit yet. I thought I would see if anyone here had performed the upgrade and knew what needed to be done. Here is the output I got when I tried to run it: rpm -ivh syslog-ng-premium-edition-4.2.3-1.rhel5.i386.r pm --force warning: syslog-ng-premium-edition-4.2.3-1.rhel5.i386.rpm: Header V4 DSA signature: NOKEY, key ID 2aa28252 Preparing... ########################################### [100%] *IMPORTANT NOTICE - NEW LICENSE REQUIRED FOR MULTI-CPU OPERATION* Please note that the 4.2 version of syslog-ng Premium Edition includes multi-threaded processing with a 2 CPU core limit. If you are upgrading your syslog-ng server from version 4.1 to 4.2 and your syslog-ng deployment requires more than 2 CPU cores, you will need to obtain a new license file, free of charge, from the BalaBit sales department. Please contact the sales department at sales(a)balabit.com. Shutting down kernel logger: [FAILED] Shutting down system logger: [FAILED] Stopping syslog-ng: 1:syslog-ng-premium-editiwarning: /opt/syslog-ng/etc/syslog-ng.conf created as /opt/syslog-ng/etc/syslog-ng.conf.rpmnew ########################################### [100%] Starting syslog-ng: WARNING: window sizing for network sources has changed in syslog-ng 4.1. The window size is the value of the log_iw_size() option divided by the value of max_connections(). If log_iw_size()/max_connections() is lower than 100, it is automatically increased to 100. Adjust the log_fifo_size() option if needed to avoid message loss.; orig_window_size='1', new_window_size='100', min_log_fifo_size='100000' [ OK ] Thanks, James McDonald System Administrator RHCSA, Security+ jmcdonald(a)lce.com

3 2

Issue about using perl script in Receiving messages from external applications
by Jian Zhu 10 Aug '12

10 Aug '12

Hello, Could anyone please help me on this below issue? my syslog-ng.conf file look like below: The test.sh script work fine, I can see it's output in the test.sh.log file. But the test.pl doesn't work. the test.sh and test.pl have the same function of printing something to STDOUT. ------------------------------ ------------------------------------------------------------------------------------------------------------------- @version: 3.0 #Default configuration file for syslog-ng. # # For a description of syslog-ng configuration file directives, please read # the syslog-ng Administrator’s guide at: # # http://www.balabit.com/dl/html/syslog-ng-admin-guide_en.html/bk01-toc.html # options { # Number of syslog lines stored in memory before being written to files flush_lines (0); log_fifo_size (2048); create_dirs (yes); perm (0640); dir_perm (0750); }; source ppp { program("/root/perl/test.pl"); }; destination ppp { file ("/var/log/syslog-ng/$HOST/perl.pl.log" create_dirs(yes)); }; log { source(ppp); destination(ppp); }; source sss { program("/root/shell/test.sh"); }; destination sss { file ("/var/log/syslog-ng/$HOST/test.sh.log" create_dirs(yes)); }; log { source(sss); destination(sss); }; ------------------------------------------------------------------------------------------------------------------------------------------------- -- *Thanks & Best Regards* *James Zhu* *Mobile**:* *+86 18**6 8482 2818* *Email: *zhujian0805(a)gmail.com

1 0

Query On configuring Centralized Audit server with Auditd daemon
by Koresh... 09 Aug '12

09 Aug '12

Hi Folks, Need your help ! Want to configure a centralized Audit server (Currently the centralized server is running Octopussy Web interface, which receives logs from remote hosts by Rsyslog ). The challenge and confusion here is .. all my linux clients are configured with syslog-ng and the daemon is sending all the system logs and kernel logs like messages,secure,cron logs etc ... with out any trouble. The problem is the syslog-ng daemon is not able to send the auidtd logs (/var/log/audit.log) to the Rsyslog server, Hence request your help to guide me how to setup the syslog-ng to forward the audit.log to the remote Rsyslog server. It would be great if i can get client side and server side configuration guidelines. -- Thanks in Advance - Koresh

3 4

syslog-ng service not starting
by James McDonald 08 Aug '12

08 Aug '12

I returned from being out of the office for week and realized something was "off" with syslog-ng. I was unable to restart the service. I did a 'service syslog-ng status' and it says it's stopped. When I try 'service syslog-ng start' I get the following error: Error binding socket; addr='AF_INET(0.0.0.0:443)', error='Address already in use (98)' Error initializing source driver; source='inputs', id='inputs#3' Error initializing message pipeline; There is only instance of syslog-ng installed on the server but I did a reboot to try and clear any conflicting settings, but I still get the same error. Can someone help me clear this error and get the service started again? Thanks, James McDonald System Administrator RHCSA, Security+ jmcdonald(a)lce.com

5 8

query about dropped messages...
by Vijendra Agarwal 06 Aug '12

06 Aug '12

Hi, I am using syslog-ng 3.1.4 open source edition. my setup details are.. I have Log application running on MC1 with syslog-ng configured as client on it. My log application is communicating witg syslog-ng on unix-stream socket. I have machine M2 where my syslog-ng server is running. My syslog-ng client is coomunicating with syslog-ng server on TCP stream. It writes all the log messages received from my log application to the /var/log/messages file. My log application sends bulk of message around 600000 p/s , I have noticed from the syslog-statistics that the syslog-ng client dropped some log messages. I have a query here Can I store the dopped messages to some file by doing some configration setting? Or Is there a configyration setting by which I can avoide the dropping of messages by syslog-ng client? Kindly help me out to resolve the issue. Thanks Vijendra. -- Thanks Vij M No: 9960771025.

1 0

Syslog-ng OSE 3.3.5 - High CPU usage
by satish pattar 31 Jul '12

31 Jul '12

Hi All, I am using syslog-ng OSE & seeing a high CPU usage ? Could you please point out if this could be a config/rules issue or syslog-ng internal issue. CPU usage in sorted order : 0.0 11.9 16.6 22.2 25.0 25.0 25.0 25.0 33.3 33.3 50.0 ------------------------------------------ # syslog-ng --version syslog-ng 3.3.5 Installer-Version: 3.3.5 Revision: Compile-Date: Apr 20 2012 04:47:44 Default-Modules: afsocket,affile,afprog,afuser,basicfuncs,syslogformat,dbparser Available-Modules: syslogformat,dbparser,afuser,basicfuncs,afmongodb,convertfuncs,afprog,dummy,afsocket,confgen,affile,csvparser Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-IPv6: on Enable-Spoof-Source: off Enable-TCP-Wrapper: off Enable-Linux-Caps: off Enable-Pcre: off ----------- Here is the code snippet to send syslog messages to syslogger for(i=0; i<500;i++) { // this is the message format we are following syslog(LOG_INFO, "%s, %s, %d, %s, %s, %s %d, %s", "00:00:18","XYZ",getpid(),"Info","xyz.c", "Func1", 100, "WelCome to Syslog" ); } --------------------- # cat /proc/cpuinfo Processor : ARMv7 Processor rev 10 (v7l) processor : 0 BogoMIPS : 1988.29 processor : 1 BogoMIPS : 1988.29 Features : swp half thumb fastmult vfp edsp neon vfpv3 CPU implementer : 0x41 CPU architecture: 7 CPU variant : 0x2 CPU part : 0xc09 CPU revision : 10 ------------------------------ syslog-ng-confg: @version: 3.3 ###### # options options { # disable the chained hostname format in logs # (default is enabled) chain_hostnames(0); # the time to wait before a died connection is re-established # (default is 60) time_reopen(10); # the time to wait before an idle destination file is closed # (default is 60) time_reap(360); # the number of lines buffered before written to file # you might want to increase this if your disk isn't catching with # all the log messages you get or if you want less disk activity flush_lines(5); flush_timeout(500); # the number of lines fitting in the output queue log_fifo_size(10000); # enable or disable directory creation for destination files create_dirs(yes); # default owner, group, and permissions for log files # (defaults are 0, 0, 0600) #owner(root); group(root); perm(0640); frac_digits(4); # Override the global timestamp format #(set in the global ts_format() parameter) for the specific destination. ts_format(iso); # default owner, group, and permissions for created directories # (defaults are 0, 0, 0700) #dir_owner(root); dir_perm(0755); # enable or disable DNS usage # syslog-ng blocks on DNS queries, so enabling DNS may lead to # a Denial of Service attack # (default is yes) use_dns(no); # maximum length of message in bytes # this is only limited by the program listening on the /dev/log Unix # socket, glibc can handle arbitrary length log messages, but -- for # example -- syslogd accepts only 1024 bytes # (default is 2048) #log_msg_size(2048); # print statistic information only once every hour (it is not really needed) stats_freq(3600); }; ###### # sources # all known message sources source s_all { # message generated by Syslog-NG internal(); # standard Linux log source (this is the default place for the syslog() # function to send logs to) unix-dgram("/dev/log" program_override("mylogger: ")); # messages from the kernel file("/proc/kmsg" program_override("kernel: ")); # use the following line if you want to receive remote UDP logging messages # (this is equivalent to the "-r" syslogd flag) # udp(); }; ######## #templates #template t_default { template("${ISODATE} ${HOST} ${MSGHDR}${MSG}\n"); template_escape(no); }; #template t_logtty { template("${MSG}\n"); template_escape(no); }; template t_restmsg_notempty { template("${PROGRAM} ${date} ${cname}[${processId}] ${type} (${source}:${line}) ${Restmsg}\n"); }; template t_restmsg_empty { template("${PROGRAM} ${FACILITY} ${LEVEL} ${MSG}\n"); }; ###### # destinations destination d_log_notempty { file("/var/my_log" template(t_restmsg_notempty)); }; #destination d_log_notempty { file("/dev/console" template(t_restmsg_notempty)); }; destination d_log_empty { file("/var/my_log" template(t_restmsg_empty)); }; destination authlog { file("/var/log/auth.log"); }; destination debug { file("/var/log/debug"); }; destination messages { file("/var/log/messages"); }; destination _syslog { file("/var/log/syslog"); }; destination cron { file("/var/log/cron.log"); }; destination daemon { file("/var/log/daemon.log"); }; destination lpr { file("/var/log/lpr.log"); }; destination user { file("/var/log/user.log"); }; destination uucp { file("/var/log/uucp.log"); }; destination mail { file("/var/log/mail.log"); }; destination kern { file("/var/log/kern.log"); }; destination mailinfo { file("/var/log/mail.info"); }; destination mailwarn { file("/var/log/mail.warn"); }; destination mailerr { file("/var/log/mail.err"); }; destination newscrit { file("/var/log/news/news.crit"); }; destination newserr { file("/var/log/news/news.err"); }; destination newsnotice { file("/var/log/news/news.notice"); }; ###### # filters # all messages from the auth and authpriv facilities filter f_restmsg_notempty {match(".+" value("Restmsg"));}; filter f_restmsg_empty {not match(".+" value("Restmsg"));}; # all messages except from the auth and authpriv facilities filter f_auth { facility(auth); }; filter f_authpriv { not facility(auth, authpriv); }; #filter f_debug { level(notice) and not facility(auth, authpriv, news, mail); }; filter f_emergency { level(emerg); }; filter f_info { level(info); }; filter f_notice { level(notice); }; filter f_warn { level(warn); }; filter f_crit { level(crit); }; filter f_err { level(err); }; #filter f_syslog { not facility(authpriv, mail ); }; filter f_syslog { program(syslog-ng); }; filter f_cron { facility(cron); }; filter f_daemon { facility(daemon); }; filter f_kern { facility(kern); }; filter f_mail { facility(mail); }; filter f_user { facility(user); and level(error) }; filter f_uucp { facility(uucp); }; filter f_news { facility(news); }; filter f_messages { level(info..warn) and not facility(auth, authpriv, mail, news); }; filter f_all_info { not match("Info", value("type"), type("string")); }; filter f_all_generic{ not match("Generic", value("cname"), type("string")); }; parser p_db {db_parser( file("/etc/syslog-ng/rules/rules.xml") );}; log {source(s_all);parser(p_db);filter(f_restmsg_notempty);destination(d_log_notempty); }; #log { source(s_all); filter(f_authpriv); destination(authlog); }; log { source(s_all); filter(f_syslog); destination(_syslog); }; log { source(s_all); filter(f_cron); destination(cron); }; log { source(s_all); filter(f_daemon); destination(daemon); }; log { source(s_all); filter(f_kern); destination(kern); }; log { source(s_all); filter(f_mail); destination(mail); }; log { source(s_all); filter(f_user); destination(user); }; #log { source(s_all); filter(f_debug); destination(debug); }; ------------------------------------------------------------------------------------- rules.xml: <patterndb version='4' pub_date='2012-02-17'> <ruleset name='root' id='123456678'> <pattern>mylogger</pattern> <rules> <rule provider='test' id='182437592347598' class='system'> <patterns> <pattern>@ESTRING:date:,@ @ESTRING:cname:,@ @ESTRING:processId:,@ @ESTRING:type:,@ @ESTRING:source:,@ @ESTRING:line:,@ @ANYSTRING:Restmsg: @</pattern> </patterns> </rule> </rules> </ruleset> </patterndb> --------------------------------------------------------------------------------------------- Thanks, pattarsatish

2 1