September 2010 - syslog-ng - lists.balabit.hu

[Bug 42] New: capabilities, chown, chmod
by bugzilla＠bugzilla.balabit.com 20 Dec '12

20 Dec '12

https://bugzilla.balabit.com/show_bug.cgi?id=42 Summary: capabilities, chown, chmod Product: syslog-ng Version: 3.0.x Platform: PC OS/Version: Linux Status: NEW Severity: normal Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: zbyniu(a)pld-linux.org Type of the Report: bug Estimated Hours: 0.0 Let's take a look at syslog-ng-3.0.1/src/affile.c lines 60-83 1. CAP_SYS_ADMIN is needed only for /proc/kmsg, it is added w/o check 2. CAP_DAC_READ_SEARCH should be added only if open fail with errno 13 2a. CAP_DAC_OVERRIDE should be added only if open fail with errno 13 and with CAP_DAC_READ_SEARCH set 3. fchown needs CAP_CHOWN unconditionaly 4. fchmod needs CAP_FOWNER if file owner != euid (root here) 5. all caps should be restored summary: - CAP_SYS_ADMIN and CAP_DAC_OVERRIDE are set always even if unnecessary, and permanently - owner, group and perm doesn't work -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

2 10

[Bug 14] New: Maintain symlinks to current and previous log output files...
by bugzilla＠bugzilla.balabit.com 28 Sep '11

28 Sep '11

https://bugzilla.balabit.com/show_bug.cgi?id=14 Summary: Maintain symlinks to current and previous log output files... Product: syslog-ng Version: 2.0.x Platform: PC URL: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=218688 OS/Version: Linux Status: NEW Severity: minor Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: sasa(a)debian.org Type of the Report: enhancement Estimated Hours: 0.0 >From bugs.debian.org: I'm hoping that upstream might consider adding functionality to syslog-ng to allow it to track current and previous log files and maintain symlinks to them, as a user option. This replicates one of the options that I (and others) find useful in cronolog. This has been previously discussed on the syslog-ng mailing list, though apparently nothing fully came of it. The archive of that discussion is at: https://lists.balabit.hu/pipermail/syslog-ng/2002-June/003452.html Obviously, it's probably somewhat of a low priority, but it would be nice. -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 1

filter/parse based on certificate attributes?
by Christian Tramnitz 22 Feb '11

22 Feb '11

With syslog 3.0 and ssl support using mutual certificate authentication is it possible to parse/filter/tag based on attributes in the certificate? I'd like to have a common listener and then relay or append messages based on client-cert attribute values... (Sorry if this message is arriving more than once, I seem to have problems with posting through Gmane.) Regards, Christian

4 4

[Bug 92] New: Allow user to specify SSL/TLS cipher suite for connections
by bugzilla＠bugzilla.balabit.com 11 Jan '11

11 Jan '11

https://bugzilla.balabit.com/show_bug.cgi?id=92 Summary: Allow user to specify SSL/TLS cipher suite for connections Product: syslog-ng Version: 3.1.x Platform: PC OS/Version: Linux Status: NEW Severity: minor Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: brett.edgar(a)truedigitalsecurity.com Type of the Report: enhancement Estimated Hours: 0.0 Created an attachment (id=22) --> (https://bugzilla.balabit.com/attachment.cgi?id=22) Patch to allow user to specify acceptable TLS cipher suites. Some compliance regimens (e.g., the PCI DSS) require SSL/TLS connections to disable weak (<128 bit key length) ciphers and old protocols like SSLv2. Syslog-ng already disables SSLv2 by default, but there is no way to disable the weak ciphers without modifying the source with every new release. I have developed a very small patch that adds a configuration file line with the tls { } stanza to specify which TLS ciphers can be used. Example: source tlssrc { tcp(ip(1.2.3.4) port(514) max-connections(32) tls( key_file("/etc/syslog-ng/server.key") cert_file("/etc/syslog-ng/server.crt") cipher_suite("ALL:!aNULL:!ADH:!eNULL:!LOW:!EXP:RC4+RSA:+HIGH:+MEDIUM:!SSLv2") peer_verify(optional-untrusted)) ); }; -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 3

Buffering AF_UNIX Destination, Batch Post Processing Messages
by Matthew Hall 19 Oct '10

19 Oct '10

Hello All, I want to configure an AF_UNIX SOCK_DGRAM syslog-ng destination which sends certain log messages to an external program for further processing and analysis. This program should batch up the messages into 60 second batches for processing. Currently I am running into an architectural challenge in how I should process the 60 second batch without slowing down the select which is collecting the messages from the destination. In the past when creating a similar kind of application in Java I handled this by creating a huge dynamic array to store objects creating from each incoming message, then passed the array reference to another background thread for processing, and began building a new array in the select thread. Currently I am trying to solve this same basic problem in Perl, which has poor threading support. I am investigating a few different options: * use threads anyway-- not recommended by more expert Perl devs I asked * prefork a process which listens to the AF_UNIX from syslog-ng, and writes to some kind of buffered non blocking pipe with a really big buffer-- not sure if such a pipe device actually exists, many pipes block * postfork a worker process which handles the 60 second batch-- problem here is that you want to have a whole lot of long term state data which is maintained between batches to help separate the needles from the haystacks, and you could get weird behavior on the duplicated FDs that are still being select()ed in the parent process which are copied into the child. * use some kind of message or job queue to copy things from a producer process to a consumer process-- gearman, theschwartz, beanstalk, rabbitmq, activemq, and poe::component::mq have been suggested-- this would probably cause a lot of context switching and unwanted buffer copies * see if there is something existing in syslog-ng that can help with this situation. can it somehow be convinced to buffer things internally for my process when my process is busy on a 60 second batch, or send in 60 second batch, etc. / whatever other clever people can dream up? Is this a problem other people have dealt with before? What did you do about this one? I want to get this right and avoid making a big mess or reinventing the wheel. Matthew.

4 18

pdbtool patternize update and my syslog-ng 3.2 branch
by Peter Gyongyosi 19 Oct '10

19 Oct '10

Hello, As the patterndb project is starting to gain some momentum I thought it'd be the right time to port my patternize tool to the new, plugin-based 3.2 codebase as the first step towards getting it integrated --- and to be able to use the fancy new pdbtool features along with patternize. To those who are unfamiliar with it, patternize is an addition to pdbtool that makes it possible to automatically generate a pattern database from raw logs using statistical data clustering methods: you can read more about it in this blog post: http://gyp.blogs.balabit.com/2010/01/introducing-pdbtool-patternize/ Besides the port to the new codebase, it's received some fixes and new features since my original post: * multiple small internal bugfixes to get rid of weird errors * added the option "/--named-parsers/" that names the found @ESTRING@s like "/.dict.string0,1,2,3.../" * Balint Kovacs has sent three contributions: added support for reading the logfile from the standard input, escaping special characters in the output and putting examples in the XML that can be used for self-testing. It can be found in my public syslog-ng 3.2 tree: http://git.balabit.hu/?p=gyp/syslog-ng-3.2.git;a=summary If you're already using it (I've received some feedback so I guess some of you do), please note that most probably this 3.2-based branch will get the fixes and new features from now on. It's only received a basic sanity check and the unit tests do pass, so as usual, handle it with care and all feedback is welcome. greets, Peter ps.: the branch also contains a patch that fixes a wrong section name in pdbtool's man page and I'll try to update the whole manpage a bit when adding a section for patternize soon -- Bazsi, you might want to pull those to the mainline.

6 24

[Bug 95] New: Missing capabilities support for unix-stream() source
by bugzilla＠bugzilla.balabit.com 18 Oct '10

18 Oct '10

https://bugzilla.balabit.com/show_bug.cgi?id=95 Summary: Missing capabilities support for unix-stream() source Product: syslog-ng Version: 3.0.x Platform: PC OS/Version: Linux Status: NEW Severity: normal Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: zbyniu(a)pld-linux.org Type of the Report: --- Estimated Hours: 0.0 Operations on unix-stream() sources need capabilities: - CAP_CHOWN - needed if owner() or group() are in use - CAP_FOWNER - to force chmod() for sockets with owner != root - CAP_DAC_OVERRIDE - force changes if some parent dir has 000 perm, ie vservers Attached patch fixes all these problems. -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 6

Bazsi's blog: Syslog-ng correllation
by Balazs Scheidler 18 Oct '10

18 Oct '10

Syslog-ng correllation I think we’ve reached an important milestone with syslog-ng: log message correllation was added to db-parser(). As you probably know dbparser and its sister project patterndb is able to transform unstructured syslog messages into a normalized format: the human readable string content becomes a set of name-value pairs. The problem is that in a lot of cases messages miss one or two details that would really be needed to understand them and this information usually comes in a followup message. For example: one message in postfix logs contain the sender address and while the recipient information comes in the next message. It is trivial to understand that in reality most cases you want the information in sender,recipient pairs. Another example is sshd, where the authentication failure comes in one and the exact reason for the failure comes in the next. Currently what you can do with syslog-ng is to put the separate messages into two SQL tables and join them at query time. This gets ugly quite fast: increased storage needs, the hassle with managing two tables instead of one and not to mention the increase of the time needed to query the database. Sometimes the sole reason for creating SQL tables in this case is to perform the correllation, otherwise you’d be happier with a CSV file. And that’s what became possible now with the latest git commit of syslog-ng 3.2. The idea is simple: when a patterndb rule matches, you can tell syslog-ng to remember that message by adding it to a correllation state. This state is identified with information extracted from the message making it a unique session identifier. When the next line comes in you can reference the information stored earlier. Basically the correllation state is a list of log messages associated with a session id. To add a new message to this state, you need a store rule: <rule id=”…”> <patterns> <pattern>foo session: @STRING:sessionid@, param: @STRING:param@</pattern> </patterns> <store id=”$sessionid” timeout=”60″/> </rule> The id attribute of the store element specifies a template containing any syslog-ng name-value pairs, probably extracted from the current message itself. When the final information comes in you can use the join attribute of the values tag: <rule id=”…”> <patterns> <pattern>bar session: @STRING:sessionid@</pattern> </patterns> <values join=”$sessionid”> <value name=”param”>${param}@1</value> </values> </rule> here the join attribute specifies the session to look up (which must match in the two messages), and if there’s a match all messages stored in the correllation state becomes available when evaluating the name-value pairs associated with the current message. The key here is the new syntax in the template string “@1″ appended to a name-value pair reference. After the “@” character, you can reference a message in the correllation state by specifying the index backward from the current message. This way @0 is the current message, @1 is the one prior to the current one, @2 is before that and so on. There are more complex ways to use/query the contents of the correllation state, but those will appear in a followup post. Stay tuned!

2 8

anushri kannu wants to chat
by anushri kannu 16 Oct '10

16 Oct '10

----------------------------------------------------------------------- anushri kannu wants to stay in better touch using some of Google's coolest new products. If you already have Gmail or Google Talk, visit: http://mail.google.com/mail/b-959daaa790-e05881a890-2yjbBnthaRYR7q040x9uEKf… You'll need to click this link to be able to chat with anushri kannu. To get Gmail - a free email account from Google with over 2,800 megabytes of storage - and chat with anushri kannu, visit: http://mail.google.com/mail/a-959daaa790-e05881a890-2yjbBnthaRYR7q040x9uEKf… Gmail offers: - Instant messaging right inside Gmail - Powerful spam protection - Built-in search for finding your messages and a helpful way of organizing emails into "conversations" - No pop-up ads or untargeted banners - just text ads and related information that are relevant to the content of your messages All this, and its yours for free. But wait, there's more! By opening a Gmail account, you also get access to Google Talk, Google's instant messaging service: http://www.google.com/talk/ Google Talk offers: - Web-based chat that you can use anywhere, without a download - A contact list that's synchronized with your Gmail account - Free, high quality PC-to-PC voice calls when you download the Google Talk client We're working hard to add new features and make improvements, so we might also ask for your comments and suggestions periodically. We appreciate your help in making our products even better! Thanks, The Google Team To learn more about Gmail and Google Talk, visit: http://mail.google.com/mail/help/about.html http://www.google.com/talk/about.html (If clicking the URLs in this message does not work, copy and paste them into the address bar of your browser).

4 4

syslog-ng issue
by anushri kannu 06 Oct '10

06 Oct '10

Hi Every one, I am new to concept of syslog-ng configuration. Already syslog-ng configured in linux server We have 6 syslog-ng server 4 location syslog-ng server receives logs from all the syslog client .. working fine 1 centralized syslog-ng (server receives log from 4 locations ....... working fine 1 we have tcim syslog-ng server receives logs from centralized syslog-ng server... it was working before for both solaris and linux . Now suddenly not collecting logs only for linux. No changes were made. Centalized syslog-ng configuration file options { log_fifo_size(8192); create_dirs(yes); group(sysgrp); dir_group(sysgrp); dir_perm(0750); perm(0440); chain_hostnames(no); keep_hostname(yes); stats(3600); use_fqdn(yes); use_time_recvd(yes); }; Standard filters # Level Filters filter f_emerg { level (emerg); }; filter f_alert { level (alert .. emerg); }; filter f_crit { level (crit .. emerg); }; filter f_err { level (err .. emerg); }; filter f_warning { level (warning .. emerg); }; filter f_notice { level (notice .. emerg); }; filter f_info { level (info .. emerg); }; filter f_debug { level (debug .. emerg); }; # Facility Filters filter f_kern { facility (kern); }; filter f_user { facility (user); }; filter f_mail { facility (mail); }; filter f_daemon { facility (daemon); }; filter f_auth { facility (auth); }; filter f_authpriv { facility (authpriv); }; filter f_syslog { facility (syslog); }; filter f_lpr { facility (lpr); }; filter f_news { facility (news); }; filter f_uucp { facility (uucp); }; filter f_os_unix { not program(EvntSLog) and not program(NetScreen) and not match ("NetScreen device_id") and not match ("%AAA-") and not match ("%AUTH-") and not match ("%AUTHPRIV-") and not match ("%CALLHOME-") and not match ("%CDP-") and not match ("%EARL-") and not match ("%FILESYS-") and not match ("%IMAGE_DNLD-SLOT") and not match ("%IP-") and not match ("%KERN-") and not match ("%LICMGR-") and not match ("%LINEPROTO-") and not match ("%LINK-") and not match ("%MCAST-") and not match ("%MODULE-") and not match ("%OSPF-") and not match ("%PLATFORM-") and not match ("%PRUNING-") and not match ("%PORT-") and not match ("%SPANTREE-") and not match ("%SYS-") and not match ("%UDLD-") and not match ("%VSHD-") source s_local { unix-stream("/dev/log"); udp(ip(0.0.0.0) port(514)); tcp(ip(0.0.0.0) port(5149) max-connections(333)); internal(); pipe("/proc/kmsg"); }; destination dl_hosts-unix { file("/var/log/syslog-ng/hosts-unix/$HOST/$YEAR/$MONTH/$DAY/$FACILITY.$LEVEL"); }; log { source(s_local); filter(f_os_unix); ###not filter(f_os_switch); destination(dl_hosts-unix); }; destination dl_tcim { udp("10.230.148.18" port(514) template("<$PRI> $DATE $HOST $MESSAGE\r\n")); }; log { source(s_local); destination(dl_tcim); }; tcim server configurarion file. options { sync (0); time_reopen (10); log_fifo_size (1000); long_hostnames (off); # use_dns (no); use_dns (yes); use_fqdn (no); create_dirs (no); keep_hostname (yes); }; source src { udp(); tcp(port(514) keep-alive(yes)); filter f_lnx_hosts { host("amex") or host("green") or host("sa") or host("yellow") or host("urinf01") or etc..; .. .. . }; destination d_lnx { file("/var/log/tcim/$HOST/syslog-$YEAR-$MONTH-$DAY.log" template("<$PRI>$DATE $HOST $MSG\n") create_dirs(yes) owner(svc-tcim) group(users) perm(0660) dir_owner(svc-tcim) dir_group(users) dir_perm(0770) ); }; log { source(src); filter(f_lnx_hosts); destination(d_lnx); }; I did try below command in TCIM server to check the comunication between centralized syslog-ng serer and tcim server tcpdump -nn -tp -port 514.. IP 10.180.40.83.59535 > 10.230.148.18.514: UDP, length 375 IP 10.180.40.83.59535 > 10.230.148.18.514: UDP, length 193 IP 10.180.40.83.59535 > 10.230.148.18.514: UDP, length 638 IP 10.180.40.83.59535 > 10.230.148.18.514: UDP, length 638 1740 packets captured 1740 packets received by filter 0 packets dropped by kernel Packets are getting from centralised log server. Do not know where the mistake is. Please help to resolve this issue.

2 2