February 2013 - syslog-ng - lists.balabit.hu

how to archive logs efficiently
by Daniel Neubacher 05 Apr '13

05 Apr '13

Hey there, my syslog-ng has gotten quite big with 50k logs per second and the server seems to hit the io limit at night. While a few month ago I could run a gzip with ionice over all old logs the server doesn't like it anymore and quite a lot of logs are storing while the compression lasts. I'm using the ose so I've got no logstore. And for a second I've thought about writing the logs a compressed fuse fs but... fuse :P So how are you guys doing it? -- Daniel Neubacher, Network Administrator daniel.neubacher(a)xing.com<mailto:daniel.neubacher@xing.com> XING AG Gaensemarkt 43, 20354 Hamburg, Germany Tel. +49 40 419131-28, Fax +49 40 419131-11 Commercial Reg. (Registergericht): Amtsgericht Hamburg, HRB 98807 Exec. Board (Vorstand): Dr. Stefan Groß-Selbeck (Vorsitzender), Dr. Thomas Vollmoeller, Ingo Chu, Dr. Helmut Becker, Jens Pape Chairman of the Supervisory Board (Aufsichtsratsvorsitzender): Dr. Neil Sunderland This e-mail may contain confidential and/or privileged information. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and destroy this e-mail. Any unauthorised copying, disclosure or distribution of the material in this e-mail is strictly forbidden and may be unlawful.

5 7

strange error message: Invalid byte sequence or other error while converting input
by Lucas, Sascha 01 Mar '13

01 Mar '13

Hi, I'm running a self-compiled syslog-ng-3.4.1 on SLES11-SP2. Compilation is mostly identical to the rpm-spec from OBS home:czanik:syslog-ng34, expect spoof-source is disabled. I have the following configuration enabled, to avoid non-UTF8 chars in mongodb: source s_network { tcp( max-connections(10) port(2012) encoding(UTF-8)); udp( port(2012) encoding(UTF-8)); }; Feb 27 12:54:13 gisu1000 syslog-ng[3335]: Syslog connection accepted; fd='11', client='AF_INET(x.x.x.x:50928)', local='AF_INET(0.0.0.0:2012)' Feb 27 12:54:13 gisu1000 syslog-ng[3335]: Invalid byte sequence or other error while converting input, skipping character; encoding='UTF-8', char='0xd2' Feb 27 12:54:13 gisu1000 syslog-ng[3335]: Syslog connection closed; fd='11', client='AF_INET(x.x.x.x:50928)', local='AF_INET(0.0.0.0:2012)' Can someone point me to what this means and how to avoid this? Thanks, Sascha. Vorsitzender des Aufsichtsrates: Ralf Hiltenkamp Geschäftsführung: Michael Krüger (Sprecher), Stephan Drescher Sitz der Gesellschaft: Halle/Saale Registergericht: Amtsgericht Stendal | Handelsregister-Nr. HRB 208414 UST-ID-Nr. DE 158253683 Diese E-Mail enthält vertrauliche und/oder rechtlich geschützte Informationen. Wenn Sie nicht der richtige Empfänger sind oder diese E-Mail irrtümlich erhalten haben, informieren Sie bitte sofort den Absender und vernichten Sie diese Mail. Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser Mail oder des Inhalts dieser Mail sind nicht gestattet. Diese Kommunikation per E-Mail ist nicht gegen den Zugriff durch Dritte geschützt. Die GISA GmbH haftet ausdrücklich nicht für den Inhalt und die Vollständigkeit von E-Mails und den gegebenenfalls daraus entstehenden Schaden. Sollte trotz der bestehenden Viren-Schutzprogramme durch diese E-Mail ein Virus in Ihr System gelangen, so haftet die GISA GmbH - soweit gesetzlich zulässig - nicht für die hieraus entstehenden Schäden.

1 1

credit card masking in log messages
by Marton ILLES 01 Mar '13

01 Mar '13

Hello Loggers! I was playing around with the latest 3.4.1 release and created two rewrite configuration blocks to mask and hash credit card numbers in log messages. I think it would be a good idea to add these config blocks to SCL so anyone could use it latter easily. These are fairly simple two blocks: block rewrite hash_cc(value("MESSAGE")) { subst( "(?P<1>:4[0-9]{12}(?:[0-9]{3})?|5[1-5][0-9]{14}|6(?:011|5[0-9][0-9])[0-9]{12}|3[47][0-9]{13}|3(?:0[0-5]|[68][0-9])[0-9]{11}|(?:2131|1800|35\d{3})\d{11})", "$(sha1 --length 16 $1)", value(`value`), flags(global, store-matches), type(pcre)); }; block rewrite mask_cc(value("MESSAGE")) { subst( "(?P<1>:4[0-9]{12}(?:[0-9]{3})?|5[1-5][0-9]{14}|6(?:011|5[0-9][0-9])[0-9]{12}|3[47][0-9]{13}|3(?:0[0-5]|[68][0-9])[0-9]{11}|(?:2131|1800|35\d{3})\d{11})", "$(substr $1 0 6)******$(substr $1 -4 4)" value(`value`), flags(global, store-matches), type(pcre)); }; You can use it in your rewrite statements easily: log { source(s_local); rewrite { mask_cc(); }; #rewrite { hash_cc(value(".sdata.my-cc")); }; destination(d_local); }; And it turns a message like this: Transaction approved; checkout-id='46255763', amount='38.43', cc='5542043004559005' Into this, in case of masking: Transaction approved; checkout-id='46255763', amount='38.43', cc='554204******9005' And in case of hashing: Transaction approved; checkout-id='46255763', amount='38.43', cc='986d97fc95435b22' For anyone interested in more details on credit card number masking/hashing/tokenization I also wrote a blogpost about this: http://marci.blogs.balabit.com/2013/02/masking-credit-card-numbers-in-log-m… The regexps could be enhanced further, but I guess it is good for a start. Bazsi, Algernon could you please add this to SCL? Sorry, I was lazy to generate a git pull request. :( Happy Masking! cheers, Marton

2 2

[Bug 220] New: Version 3. 4 logical configuration incompatible with previous versions
by bugzilla＠bugzilla.balabit.com 01 Mar '13

01 Mar '13

https://bugzilla.balabit.com/show_bug.cgi?id=220 Summary: Version 3.4 logical configuration incompatible with previous versions Product: syslog-ng Version: 3.4.x Platform: PC OS/Version: Linux Status: NEW Severity: major Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: erempel(a)uvic.ca Type of the Report: --- Estimated Hours: 0.0 In version 3.3 the top level configuration structure treated all log statements as if they were inside a single block. If there was a log statement with a flags(final) then no other log statements were processed. This allowed @include statements to include snippets that included logs statements with flags(final). In version 3.4 the top level configuration structure treats all log statements independently. If there was a log statement with a flags(final) it only affects processing within that log statement and processing does continue with then next log statement. Including files with log statements and flags(final) can'r really be done and the @include can not be used inside of log statements. -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 9

storing an array in mongodb
by Lucas, Sascha 27 Feb '13

27 Feb '13

Hi, I'm experimenting with patterndb and the classifier (syslog-ng 3.4.1). ATM the classifier is stored in the TAGS key as a comma separated string. For searching on different classes it would be nice to store it in an array. This can be indexed and searched (per element). Is this possible? TIA, Sascha. Vorsitzender des Aufsichtsrates: Ralf Hiltenkamp Geschäftsführung: Michael Krüger (Sprecher), Stephan Drescher Sitz der Gesellschaft: Halle/Saale Registergericht: Amtsgericht Stendal | Handelsregister-Nr. HRB 208414 UST-ID-Nr. DE 158253683 Diese E-Mail enthält vertrauliche und/oder rechtlich geschützte Informationen. Wenn Sie nicht der richtige Empfänger sind oder diese E-Mail irrtümlich erhalten haben, informieren Sie bitte sofort den Absender und vernichten Sie diese Mail. Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser Mail oder des Inhalts dieser Mail sind nicht gestattet. Diese Kommunikation per E-Mail ist nicht gegen den Zugriff durch Dritte geschützt. Die GISA GmbH haftet ausdrücklich nicht für den Inhalt und die Vollständigkeit von E-Mails und den gegebenenfalls daraus entstehenden Schaden. Sollte trotz der bestehenden Viren-Schutzprogramme durch diese E-Mail ein Virus in Ihr System gelangen, so haftet die GISA GmbH - soweit gesetzlich zulässig - nicht für die hieraus entstehenden Schäden.

3 2

No "-n" switch for syslog-ng?
by P Levine 27 Feb '13

27 Feb '13

I've been getting many "syslog-ng.service start request repeated too quickly..." issues in systemd journal . The syslog-ng service would always fail. I noticed there seems to be a bug in which systemd executes syslog-ng with the "-n" flag. I recently posted the bug here: https://bugs.freedesktop.org/show_bug.cgi?id=61324 After modifying lib/gprocess.c to accept "-n" in place of "-F", recompiling, and reinstalling, the problem has disappeared and the syslog-ng process remains up. Is this a systemd bug? I have been googling for 2 days to see if it relates to any other bugs and have found nothing. I just started using systemd 4 days ago and its my impression that syslog-ng has supported systemd for some time. So, could there be something on my end causing the issue?

4 15

syslog-ng OSE 3.4 doc updates
by Fekete Robert 26 Feb '13

26 Feb '13

Hi, I am currently working on updating the 3.4 adminguide, and releasing updated drafts every few days. The current version already has two short sections about junctions and channels (and a bunch of smaller updates as well). http://www.balabit.com/sites/default/files/documents/syslog-ng-ose-3.4-guid… http://www.balabit.com/sites/default/files/documents/syslog-ng-ose-3.4-guid… Any feedback, comment, clarification request, or real-life use-case is most welcome. I hope to release more stuff on Friday, and also to write a list of the new features that are already documented. Kind Regards, Robert

5 17

syslog-ng ruleset confusion or bug?
by David Stainton 26 Feb '13

26 Feb '13

Greetings, I am running syslog-ng-3.3.5-1 on CentOS release 6.3. It seems like there is a subtle bug in syslog-ng or more likely a bug in my syslog-ng configuration. I am unable to get log message to appear in the other destinations besides /var/log/messages... I've tried testing like this: logger info.local3 howdy logger info.local5 howdy logger info.local6 howdy logger emerg.local6 howdy However the log messages only show up in /var/log/messages. I wonder why! Any help, observations or suggestions would be much appreciated. Thanks! David PS Here is my simple config: @version: 3.3 options { create_dirs (yes); dir_perm(0755); keep_hostname (yes); perm(0644); flush_lines (0); time_reopen (10); use_dns (yes); use_fqdn (no); }; source s_sys { file ("/proc/kmsg" program_override("kernel: ")); unix-stream ("/dev/log"); internal(); unix-stream ("/var/named/chroot/dev/log" optional(yes)); }; destination d_mesg { file("/var/log/messages"); }; destination d_app { file("/var/log/philotic/$PROGRAM.log"); }; destination d_app_event { file("/var/log/philotic/$PROGRAM-event.log"); }; destination d_philotic { file("/var/log/philotic/$PROGRAM.log"); }; filter f_info { level(info..emerg) }; filter f_default { level(info..emerg) and not ( facility(mail) or facility(authpriv) or facility(cron) or facility(local6) or facility(local5) or facility(14) ); }; filter f_app_event { facility(local3); }; filter f_app { facility(local6); }; filter f_philotic { facility(local5); }; log { source(s_sys); filter(f_default); destination(d_mesg); }; log { source(s_sys); filter(f_info); filter(f_app); destination(d_app); }; log { source(s_sys); filter(f_info); filter(f_app_event); destination(d_app_event); }; log { source(s_sys); filter(f_info); filter(f_philotic); destination(d_philotic); };

2 2

syslog-ng OSE 3.3 and 3.4 documentation updates
by Fekete Robert 25 Feb '13

25 Feb '13

Hi, I have released updated version of The syslog-ng Open Source Edition 3.4 Administrator Guide. This release should document every feature of OSE 3.4 - if something is missing or unclear, please let me know. Comments and bug reports are welcome. I have also released an updated version of the The syslog-ng Open Source Edition 3.3 Administrator Guide - this is mostly a bugfix release, and is intended to be the last release of the 3.3 guide. Please note that because of some website reorganization, the URL of the docs has changed, so old bookmarks and browser history will not work. You can find the docs at http://www.balabit.com/support/documentation , just select _syslog-ng_ and the version number (3.3 or 3.4). The documentation of older releases has been moved to http://www.balabit.com/downloads/files/archived_documents Regards, Robert

1 0

Log file sending to syslog
by a bv 25 Feb '13

25 Feb '13

Hi, I have log files which i like to send these logs as real time events to the syslog-ng running and mysql running which collects the real time logs. How can i do this? Regards

3 2