October 2010 - syslog-ng - lists.balabit.hu

Having hostname and sender's IP address in header
by Worsham, Michael 28 Oct '10

28 Oct '10

We are trying to streamline the syslog-ng v3.x data from the RHEL server-clients to the RHEL DCS/Syslog server while also providing another destination to a local Tripwire Log Center server. I previously had the version 3.0.8 options (as seen below), which recorded the data files correctly on the DCS server with the correct hostname attribute: options { chain_hostnames(no); time_reopen(10); time_reap(360); log_fifo_size(1000); group(adm); perm(0644); dir_perm(0755); use_dns(yes); use_fqdn(yes); create_dirs(yes); keep_hostname(yes); log_msg_size(32768); stats_freq(0); flush_lines(0); }; Destination (directory and log file structure): destination d_general { file ("/var/log/syslog/general/$HOST/$FULLHOST-$MONTH.$DAY.$YEAR.log" template("$DATE $HOST <$FACILITY.$PRIORITY> $MSGHDR $MSG\n") template_escape(no) ); }; Example: /var/log/syslog/general/CH33Test-WebCO/CH33Test-WebCO-10.25.2010.log However, we found today that Tripwire Log Center (which we are now required to use for log aggregation/reporting needs) doesn't have the ability to do hostname translation (i.e. regex the sender's hostname seen in the datastream and give it an IP address so that it can be tracked). So this means the data file that is being sent from the RHEL clients will need to keep the sender's IP address in the actual data stream. So if I do keep_hostname(no) and still enable the facility.priority template on the DCS server, it will show the IP address rather than the hostname of the reporting client: Oct 25 18:51:12 10.153.13.70 <syslog.err> syslog-ng[2820]: EOF occurred while idle; fd='9' Oct 25 18:51:12 10.153.13.70 <syslog.notice> syslog-ng[2820]: Syslog connection broken; fd='9', server='AF_INET(10.153.29.235:514)', time_reopen='10' Oct 25 18:51:22 10.153.13.70 <syslog.notice> syslog-ng[2820]: Syslog connection established; fd='9', server='AF_INET(10.153.29.235:514)', local='AF_INET(0.0.0.0:0)' However, this now causes a problem on the Syslog/DCS server with the local log file storage requirements: Old Version: /var/log/syslog/general/CH33Test-WebCO/CH33Test-WebCO-10.25.2010.log New Version: /var/log/syslog/general/10.153.13.70/10.153.13.70-10.25.2010.log NOTE: We don't have a local DNS that all of the servers can be queried against and that putting in nearly 100+ sender hosts in the /etc/hosts really isn't a viable option. How do I satisfy each requirement: 1) Allow the incoming data stream to be saved in a format based on the incoming sender's hostname for the directory structure (ex: "/var/log/syslog/general/$HOST/$FULLHOST-$MONTH.$DAY.$YEAR.log")? 2) Allow the IP address of the sender to be seen in the actual data file once saved to the DCS server (ex: "Oct 25 18:51:12 10.153.13.70 <syslog.err>...")? 3) Is there an option to have the incoming sender's data saved with both the hostname and IP address in the same header format (ex: "Oct 25 18:51:12 CH33Test-WebCO 10.153.13.70 <syslog.err>...")? -- Michael ________________________________ CONFIDENTIALITY NOTICE: This email and any attachments are intended solely for the use of the named recipient(s). This email may contain confidential and/or proprietary information of Scientific Research Corporation. If you are not a named recipient, you are prohibited from reviewing, copying, using, disclosing or distributing to others the information in this email and attachments. If you believe you have received this email in error, please notify the sender immediately and permanently delete the email, any attachments, and all copies thereof from any drives or storage media and destroy any printouts of the email or attachments. EXPORT COMPLIANCE NOTICE: This email and any attachments may contain technical data subject to U.S export restrictions under the International Traffic in Arms Regulations (ITAR) or the Export Administration Regulations (EAR). Export or transfer of this technical data and/or related information to any foreign person(s) or entity(ies), either within the U.S. or outside of the U.S., may require advance export authorization by the appropriate U.S. Government agency prior to export or transfer. In addition, technical data may not be exported or transferred to certain countries or specified designated nationals identified by U.S. embargo controls without prior export authorization. By accepting this email and any attachments, all recipients confirm that they understand and will comply with all applicable ITAR, EAR and embargo compliance requirements.

4 10

Error opening log file for writing
by Bill Snow 28 Oct '10

28 Oct '10

I have a openSUSE 11.3 system that I installed syslog-ng on through package management. The intention of the simple config is to get it to write log file messages from one program (pdns) to a separate log file outside of /var/log. No matter what I do with the file and directory permissions, I always get something like: syslog-ng[17864]: Error opening file for writing; filename='/opt/home/user/logs/pdns.log', error='Permission denied (13)' in the log file. syslog-ng is running as root, as follows: # ps xo pid,ppid,user,cmd | grep syslog 17863 1 root supervising syslog-ng 17864 17863 root /sbin/syslog-ng The special config lines in /etc/syslog-ng/syslog-ng.conf look like this: filter f_dyson { match('^pdns'); }; destination dyson { file("/opt/home/user/logs/pdns.log" perm(0644)); }; log { source(src); filter(f_dyson); destination(dyson); }; Googling around indicated a possible problem with capability management, but that is turned off in my version: # syslog-ng --version syslog-ng 3.1.1 Installer-Version: 3.1.1 Revision: ssh+git://bazsi@git.balabit//var/scm/git/syslog-ng/syslog-ng-ose--mainline--3.1#master#8747d74491eac3fdf5052194f47a68e659633ca9 Compile-Date: Jul 5 2010 13:47:02 Enable-Threads: off Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-Sun-STREAMS: off Enable-Sun-Door: off Enable-IPv6: on Enable-Spoof-Source: on Enable-TCP-Wrapper: on Enable-SSL: off Enable-SQL: off Enable-Linux-Caps: off Enable-Pcre: on I tried explicitly setting the user/group for the destination file in case syslog-ng was trying to write the file with a non-privileged owner in a directory owned by root.... and also tried setting the directory permissions to 777 just in case. Moving the destination to /var/log/pdns.log works. Of course that's not where I want the file. Anybody have an idea of what might be going on here? Thanks, Bill Snow

2 2

password mailing-list
by laugello 28 Oct '10

28 Oct '10

Hello Sorry of the o-topic I forgot password of the mailing-list How I can recover it tanks Luigi -- Augello Luigi Amministratore di Sistema Poli didattici di Agrigento, Caltanissetta e Trapani Università degli Studi di Palermo tel 093420928 VoIP 09123865802

2 1

AppArmor
by Peter Czanik 26 Oct '10

26 Oct '10

Hello, I wonder, if anyone here uses syslog-ng together with AppArmor. SCL calls external applications, or using program() in syslog-ng.conf also calls an external app. I spent quite some time with AppArmor documentation, tried logprof many times, asked on openSUSE mailing lists, but no luck. I could not execute external apps when AppArmor was enabled. Does anybody has a success story here? I'd be very interested in your AppArmor profile for syslog-ng! BTW: if you use openSUSE, you could test my 3.2beta1 packages. See my blog for details! Bye, -- Peter Czanik (CzP) <czanik(a)balabit.hu> BalaBit IT Security / syslog-ng upstream http://czanik.blogs.balabit.com/

1 0

Should configure error out if lex/flex is missing?
by Lars Kellogg-Stedman 19 Oct '10

19 Oct '10

After successfully running ./configure for syslog-ng 3.2beta1, the build immediately failed because I didn't have flex installed. I see that configure is already calling AM_PROG_LEX...so maybe this is just the expected behavior? After installing flex the build completed successfully.

2 3

[Bug 52] CAP_DAC_OVERRIDE is missing when capabilities are enabled, unable to write to files created with the owner() option.
by bugzilla＠bugzilla.balabit.com 19 Oct '10

19 Oct '10

https://bugzilla.balabit.com/show_bug.cgi?id=52 Balazs Scheidler <bazsi(a)balabit.hu> changed: What |Removed |Added ---------------------------------------------------------------------------- Resolution| |FIXED Status|NEW |RESOLVED --- Comment #4 from Balazs Scheidler <bazsi(a)balabit.hu> 2010-10-19 17:23:38 --- I've integrated this as well, thanks Zbigniew commit 967b1720c8487f3cbe49292c7e5ed3b871ab2de5 Author: Zbigniew Krzystolik <zbyniu(a)pld-linux.org> Date: Tue Oct 19 17:21:23 2010 +0200 fix capability in the file() destination fixes for settings like owner(), group(), perm() and analogous dir_* if create_dirs=yes for destination files - CAP_DAC_OVERRIDE force changes if parent dir has 000 perm, ie vservers (is inerhitted for dir_* too) - CAP_CHOWN - needed if dir_owner() or dir_group() are in use - CAP_FOWNER - to force chmod() for dirs with owner != root -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 0

Buffering AF_UNIX Destination, Batch Post Processing Messages
by Matthew Hall 19 Oct '10

19 Oct '10

Hello All, I want to configure an AF_UNIX SOCK_DGRAM syslog-ng destination which sends certain log messages to an external program for further processing and analysis. This program should batch up the messages into 60 second batches for processing. Currently I am running into an architectural challenge in how I should process the 60 second batch without slowing down the select which is collecting the messages from the destination. In the past when creating a similar kind of application in Java I handled this by creating a huge dynamic array to store objects creating from each incoming message, then passed the array reference to another background thread for processing, and began building a new array in the select thread. Currently I am trying to solve this same basic problem in Perl, which has poor threading support. I am investigating a few different options: * use threads anyway-- not recommended by more expert Perl devs I asked * prefork a process which listens to the AF_UNIX from syslog-ng, and writes to some kind of buffered non blocking pipe with a really big buffer-- not sure if such a pipe device actually exists, many pipes block * postfork a worker process which handles the 60 second batch-- problem here is that you want to have a whole lot of long term state data which is maintained between batches to help separate the needles from the haystacks, and you could get weird behavior on the duplicated FDs that are still being select()ed in the parent process which are copied into the child. * use some kind of message or job queue to copy things from a producer process to a consumer process-- gearman, theschwartz, beanstalk, rabbitmq, activemq, and poe::component::mq have been suggested-- this would probably cause a lot of context switching and unwanted buffer copies * see if there is something existing in syslog-ng that can help with this situation. can it somehow be convinced to buffer things internally for my process when my process is busy on a 60 second batch, or send in 60 second batch, etc. / whatever other clever people can dream up? Is this a problem other people have dealt with before? What did you do about this one? I want to get this right and avoid making a big mess or reinventing the wheel. Matthew.

4 18

Re: [syslog-ng] [patch] fix linking issue in 3.2beta1
by Guillaume Rousse 19 Oct '10

19 Oct '10

Le 18/10/2010 10:18, Balazs Scheidler a écrit : > Hi, > > Thanks for the patch, but I it was my intent not to link the plugins > against GLIB & other dependencies. > > Do you have an error message that this patch actually fixes? Build failure, when -Wl,--no-undefined flag is used. > The reason that the plugins are not linked against GLIB and other is > that I have to support an almost static syslog-ng, in which case GLib is > statically linked into the libsyslog-ng.so shared object and plugins > pull it from there. I may indeed disable this flag here, it's primary target is shared library, not plugin. -- BOFH excuse #107: The keyboard isn't plugged in

2 1

pdbtool patternize update and my syslog-ng 3.2 branch
by Peter Gyongyosi 19 Oct '10

19 Oct '10

Hello, As the patterndb project is starting to gain some momentum I thought it'd be the right time to port my patternize tool to the new, plugin-based 3.2 codebase as the first step towards getting it integrated --- and to be able to use the fancy new pdbtool features along with patternize. To those who are unfamiliar with it, patternize is an addition to pdbtool that makes it possible to automatically generate a pattern database from raw logs using statistical data clustering methods: you can read more about it in this blog post: http://gyp.blogs.balabit.com/2010/01/introducing-pdbtool-patternize/ Besides the port to the new codebase, it's received some fixes and new features since my original post: * multiple small internal bugfixes to get rid of weird errors * added the option "/--named-parsers/" that names the found @ESTRING@s like "/.dict.string0,1,2,3.../" * Balint Kovacs has sent three contributions: added support for reading the logfile from the standard input, escaping special characters in the output and putting examples in the XML that can be used for self-testing. It can be found in my public syslog-ng 3.2 tree: http://git.balabit.hu/?p=gyp/syslog-ng-3.2.git;a=summary If you're already using it (I've received some feedback so I guess some of you do), please note that most probably this 3.2-based branch will get the fixes and new features from now on. It's only received a basic sanity check and the unit tests do pass, so as usual, handle it with care and all feedback is welcome. greets, Peter ps.: the branch also contains a patch that fixes a wrong section name in pdbtool's man page and I'll try to update the whole manpage a bit when adding a section for patternize soon -- Bazsi, you might want to pull those to the mainline.

6 24

decoding messages from sockd (SOCKS proxy)
by Matthew Hall 18 Oct '10

18 Oct '10

Hello all, I am running into some headaches with the poor formatting of sockd messages. How should I decode messages like this? Note I have not applied XML escapes to these yet as that's hard to read but I will do so when inserting them into a patterndb to prevent parse errors. Every message in this group begins with this string on one line: sockd[@NUMBER:pid:@]: @ESTRING:action::@ @ESTRING:phase::@ @IPv4:src:@.@NUMBER:srcport:@ -> Then there are a few different endings which happen in some messages that are giving me problems to decode. Here are three examples from my collected logs: smarthost.company.com.25 host.team.division.company.com.18050: invalid address: 0.0.0.0.18050 company.com.443: Connection reset by peer I am having a hard time figuring out how to break these up into domain name (src / dst as appropriate) and port (srcport / dstport). My best thought so far was to detect this and rewrite them using PCRE before applying patterndb matching. I could find the .[0-9]+ and replace with :\1, then I have the port delimited with ':' and I can pull it apart using: @ESTRING:src::@@NUMBER:srcport:@ Is it possible to do PCRE replacement using backreferences? Or is there another way to get this to work? Thanks, Matthew.

2 2