- syslog-ng - lists.balabit.hu

Syslog-ng on Solaris 9 problem
by Elgin Lorenz 29 Oct '10

29 Oct '10

Hello experts, today we switched our syslog-ng server from linux (CentOS 5.5) to a Solaris 9 box. All went well, until I noticed that syslog-ng obviously changes some of the log entries. Here are the 2 examples I noticed: Message on the local host: Oct 18 14:19:15 xxx kernel: IN=eth1 OUT= MAC=xxx SRC=xxx DST=xxx LEN=235 TOS=0x00 PREC=0x00 TTL=128 ID=119 PROTO=UDP SPT=138 DPT=138 LEN=215 Message on the syslog-ng server: Oct 18 14:19:15 xxx kernel: kernel: IN=eth1 OUT= MAC=xxx SRC=xxx DST=xxx LEN=235 TOS=0x00 PREC=0x00 TTL=128 ID=119 PROTO=UDP SPT=138 DPT=138 LEN=215 (one "kernel: " added) Message on the local host: Oct 18 14:03:06 xxx last message repeated 2 times Message on the syslog-ng server: Oct 18 14:03:06 xxx last: message repeated 2 times (one ":" added) On Linux I never saw this behaviour. Until now I couldn't find the mistake. What am I doing wrong? Did anyone else see something like this? Thanks in advance for your help. Kind regards, Elgin Lorenz -- Elgin Lorenz BTU Cottbus Universitaetsrechenzentrum Tel. 0355 693573 E-Mail lorenz(a)tu-cottbus.de

3 8

Logging all message metadata
by Lars Kellogg-Stedman 28 Oct '10

28 Oct '10

As I work with the classification engine, I wish there were a way for me to log *all* metadata associated with a log message. That is, I would like to record any data parsed out of the message by the parsing engine, as well as all the default metadata that syslog-ng generates about a message. Is there any way to do this other than writing my own output driver? As far as I can tell, all of the available drivers -- including the sql() driver -- require me to explicitly list which attributes I want to log. While my ultimate target would be sending this data into a database, I would be happy if I were able to dump it all to a file in some sort of structured format that I could parse with my own tools (or maybe even feed back into syslog-ng). Thanks, -- Lars

3 8

Compilation problem in Debian (libsyslog-ng dependency)
by Jean F. Mousinho 28 Oct '10

28 Oct '10

Hello, I'm trying to manually build the package in Debian, but I'm getting an error when compiling: ... dh_shlibdeps: Compatibility levels before 4 are deprecated. dpkg-shlibdeps: warning: couldn't find library libdbparser.so needed by debian/syslog-ng/bin/pdbtool (its RPATH is '//lib://lib/syslog-ng'). Note: libraries are not searched in other binary packages that do not have any shlibs or symbols file. To help dpkg-shlibdeps find private libraries, you might need to set LD_LIBRARY_PATH. dpkg-shlibdeps: failure: no dependency information found for lib/.libs/libsyslog-ng.so.0 (used by debian/syslog-ng/bin/pdbtool). dh_shlibdeps: command returned error code 512 make: *** [binary-arch] Error 1 dpkg-buildpackage: failure: debian/rules binary gave error exit status 2 ... I've noticed that libdbparser.so is actually located in ./modules/dbparser/.libs/libdbparser.so but the problem seems to be that dpkg-shlibdeps doesn't find dependency information for libsyslog-ng? I don't know much how automake+dpkg-buildpackage work to know how to solve this, even tho I've tried to google for it. Thanks for your time. Jean-François Mousinho

3 3

Having hostname and sender's IP address in header
by Worsham, Michael 28 Oct '10

28 Oct '10

We are trying to streamline the syslog-ng v3.x data from the RHEL server-clients to the RHEL DCS/Syslog server while also providing another destination to a local Tripwire Log Center server. I previously had the version 3.0.8 options (as seen below), which recorded the data files correctly on the DCS server with the correct hostname attribute: options { chain_hostnames(no); time_reopen(10); time_reap(360); log_fifo_size(1000); group(adm); perm(0644); dir_perm(0755); use_dns(yes); use_fqdn(yes); create_dirs(yes); keep_hostname(yes); log_msg_size(32768); stats_freq(0); flush_lines(0); }; Destination (directory and log file structure): destination d_general { file ("/var/log/syslog/general/$HOST/$FULLHOST-$MONTH.$DAY.$YEAR.log" template("$DATE $HOST <$FACILITY.$PRIORITY> $MSGHDR $MSG\n") template_escape(no) ); }; Example: /var/log/syslog/general/CH33Test-WebCO/CH33Test-WebCO-10.25.2010.log However, we found today that Tripwire Log Center (which we are now required to use for log aggregation/reporting needs) doesn't have the ability to do hostname translation (i.e. regex the sender's hostname seen in the datastream and give it an IP address so that it can be tracked). So this means the data file that is being sent from the RHEL clients will need to keep the sender's IP address in the actual data stream. So if I do keep_hostname(no) and still enable the facility.priority template on the DCS server, it will show the IP address rather than the hostname of the reporting client: Oct 25 18:51:12 10.153.13.70 <syslog.err> syslog-ng[2820]: EOF occurred while idle; fd='9' Oct 25 18:51:12 10.153.13.70 <syslog.notice> syslog-ng[2820]: Syslog connection broken; fd='9', server='AF_INET(10.153.29.235:514)', time_reopen='10' Oct 25 18:51:22 10.153.13.70 <syslog.notice> syslog-ng[2820]: Syslog connection established; fd='9', server='AF_INET(10.153.29.235:514)', local='AF_INET(0.0.0.0:0)' However, this now causes a problem on the Syslog/DCS server with the local log file storage requirements: Old Version: /var/log/syslog/general/CH33Test-WebCO/CH33Test-WebCO-10.25.2010.log New Version: /var/log/syslog/general/10.153.13.70/10.153.13.70-10.25.2010.log NOTE: We don't have a local DNS that all of the servers can be queried against and that putting in nearly 100+ sender hosts in the /etc/hosts really isn't a viable option. How do I satisfy each requirement: 1) Allow the incoming data stream to be saved in a format based on the incoming sender's hostname for the directory structure (ex: "/var/log/syslog/general/$HOST/$FULLHOST-$MONTH.$DAY.$YEAR.log")? 2) Allow the IP address of the sender to be seen in the actual data file once saved to the DCS server (ex: "Oct 25 18:51:12 10.153.13.70 <syslog.err>...")? 3) Is there an option to have the incoming sender's data saved with both the hostname and IP address in the same header format (ex: "Oct 25 18:51:12 CH33Test-WebCO 10.153.13.70 <syslog.err>...")? -- Michael ________________________________ CONFIDENTIALITY NOTICE: This email and any attachments are intended solely for the use of the named recipient(s). This email may contain confidential and/or proprietary information of Scientific Research Corporation. If you are not a named recipient, you are prohibited from reviewing, copying, using, disclosing or distributing to others the information in this email and attachments. If you believe you have received this email in error, please notify the sender immediately and permanently delete the email, any attachments, and all copies thereof from any drives or storage media and destroy any printouts of the email or attachments. EXPORT COMPLIANCE NOTICE: This email and any attachments may contain technical data subject to U.S export restrictions under the International Traffic in Arms Regulations (ITAR) or the Export Administration Regulations (EAR). Export or transfer of this technical data and/or related information to any foreign person(s) or entity(ies), either within the U.S. or outside of the U.S., may require advance export authorization by the appropriate U.S. Government agency prior to export or transfer. In addition, technical data may not be exported or transferred to certain countries or specified designated nationals identified by U.S. embargo controls without prior export authorization. By accepting this email and any attachments, all recipients confirm that they understand and will comply with all applicable ITAR, EAR and embargo compliance requirements.

4 10

Error opening log file for writing
by Bill Snow 28 Oct '10

28 Oct '10

I have a openSUSE 11.3 system that I installed syslog-ng on through package management. The intention of the simple config is to get it to write log file messages from one program (pdns) to a separate log file outside of /var/log. No matter what I do with the file and directory permissions, I always get something like: syslog-ng[17864]: Error opening file for writing; filename='/opt/home/user/logs/pdns.log', error='Permission denied (13)' in the log file. syslog-ng is running as root, as follows: # ps xo pid,ppid,user,cmd | grep syslog 17863 1 root supervising syslog-ng 17864 17863 root /sbin/syslog-ng The special config lines in /etc/syslog-ng/syslog-ng.conf look like this: filter f_dyson { match('^pdns'); }; destination dyson { file("/opt/home/user/logs/pdns.log" perm(0644)); }; log { source(src); filter(f_dyson); destination(dyson); }; Googling around indicated a possible problem with capability management, but that is turned off in my version: # syslog-ng --version syslog-ng 3.1.1 Installer-Version: 3.1.1 Revision: ssh+git://bazsi@git.balabit//var/scm/git/syslog-ng/syslog-ng-ose--mainline--3.1#master#8747d74491eac3fdf5052194f47a68e659633ca9 Compile-Date: Jul 5 2010 13:47:02 Enable-Threads: off Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-Sun-STREAMS: off Enable-Sun-Door: off Enable-IPv6: on Enable-Spoof-Source: on Enable-TCP-Wrapper: on Enable-SSL: off Enable-SQL: off Enable-Linux-Caps: off Enable-Pcre: on I tried explicitly setting the user/group for the destination file in case syslog-ng was trying to write the file with a non-privileged owner in a directory owned by root.... and also tried setting the directory permissions to 777 just in case. Moving the destination to /var/log/pdns.log works. Of course that's not where I want the file. Anybody have an idea of what might be going on here? Thanks, Bill Snow

2 2

password mailing-list
by laugello 28 Oct '10

28 Oct '10

Hello Sorry of the o-topic I forgot password of the mailing-list How I can recover it tanks Luigi -- Augello Luigi Amministratore di Sistema Poli didattici di Agrigento, Caltanissetta e Trapani Università degli Studi di Palermo tel 093420928 VoIP 09123865802

2 1

AppArmor
by Peter Czanik 26 Oct '10

26 Oct '10

Hello, I wonder, if anyone here uses syslog-ng together with AppArmor. SCL calls external applications, or using program() in syslog-ng.conf also calls an external app. I spent quite some time with AppArmor documentation, tried logprof many times, asked on openSUSE mailing lists, but no luck. I could not execute external apps when AppArmor was enabled. Does anybody has a success story here? I'd be very interested in your AppArmor profile for syslog-ng! BTW: if you use openSUSE, you could test my 3.2beta1 packages. See my blog for details! Bye, -- Peter Czanik (CzP) <czanik(a)balabit.hu> BalaBit IT Security / syslog-ng upstream http://czanik.blogs.balabit.com/

1 0

Should configure error out if lex/flex is missing?
by Lars Kellogg-Stedman 19 Oct '10

19 Oct '10

After successfully running ./configure for syslog-ng 3.2beta1, the build immediately failed because I didn't have flex installed. I see that configure is already calling AM_PROG_LEX...so maybe this is just the expected behavior? After installing flex the build completed successfully.

2 3

[Bug 52] CAP_DAC_OVERRIDE is missing when capabilities are enabled, unable to write to files created with the owner() option.
by bugzilla＠bugzilla.balabit.com 19 Oct '10

19 Oct '10

https://bugzilla.balabit.com/show_bug.cgi?id=52 Balazs Scheidler <bazsi(a)balabit.hu> changed: What |Removed |Added ---------------------------------------------------------------------------- Resolution| |FIXED Status|NEW |RESOLVED --- Comment #4 from Balazs Scheidler <bazsi(a)balabit.hu> 2010-10-19 17:23:38 --- I've integrated this as well, thanks Zbigniew commit 967b1720c8487f3cbe49292c7e5ed3b871ab2de5 Author: Zbigniew Krzystolik <zbyniu(a)pld-linux.org> Date: Tue Oct 19 17:21:23 2010 +0200 fix capability in the file() destination fixes for settings like owner(), group(), perm() and analogous dir_* if create_dirs=yes for destination files - CAP_DAC_OVERRIDE force changes if parent dir has 000 perm, ie vservers (is inerhitted for dir_* too) - CAP_CHOWN - needed if dir_owner() or dir_group() are in use - CAP_FOWNER - to force chmod() for dirs with owner != root -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 0

Buffering AF_UNIX Destination, Batch Post Processing Messages
by Matthew Hall 19 Oct '10

19 Oct '10

Hello All, I want to configure an AF_UNIX SOCK_DGRAM syslog-ng destination which sends certain log messages to an external program for further processing and analysis. This program should batch up the messages into 60 second batches for processing. Currently I am running into an architectural challenge in how I should process the 60 second batch without slowing down the select which is collecting the messages from the destination. In the past when creating a similar kind of application in Java I handled this by creating a huge dynamic array to store objects creating from each incoming message, then passed the array reference to another background thread for processing, and began building a new array in the select thread. Currently I am trying to solve this same basic problem in Perl, which has poor threading support. I am investigating a few different options: * use threads anyway-- not recommended by more expert Perl devs I asked * prefork a process which listens to the AF_UNIX from syslog-ng, and writes to some kind of buffered non blocking pipe with a really big buffer-- not sure if such a pipe device actually exists, many pipes block * postfork a worker process which handles the 60 second batch-- problem here is that you want to have a whole lot of long term state data which is maintained between batches to help separate the needles from the haystacks, and you could get weird behavior on the duplicated FDs that are still being select()ed in the parent process which are copied into the child. * use some kind of message or job queue to copy things from a producer process to a consumer process-- gearman, theschwartz, beanstalk, rabbitmq, activemq, and poe::component::mq have been suggested-- this would probably cause a lot of context switching and unwanted buffer copies * see if there is something existing in syslog-ng that can help with this situation. can it somehow be convinced to buffer things internally for my process when my process is busy on a 60 second batch, or send in 60 second batch, etc. / whatever other clever people can dream up? Is this a problem other people have dealt with before? What did you do about this one? I want to get this right and avoid making a big mess or reinventing the wheel. Matthew.

4 18