- syslog-ng - lists.balabit.hu

Collect log from beginning incase central server's log file gets deleted.
by sagar naravane 27 Oct '12

27 Oct '12

Hi, Below is my setup (see the config details). Issue i am facing is : Lets say on central servers apache logs are collected till 2:00 PM for current date. Incase if i delete the log file on central server, a new file gets created and starts collecting the log after the delete time ie 2:00 PM. What i want to achieve is that in such an event where the log files gets deleted.. the new log-file will collect the log from the beginning of the day ie 12:00 AM. Is this possible ? *************** client config *********************** @include "scl.conf" options { keep_hostname(yes); log_fifo_size(100000) }; rewrite r_setfilename { set( "$FILE_NAME", value(".SDATA.file(a)18372.4.name") ); }; rewrite r_apache_basepath_www-dev01 { subst( "/apps/log/apache/www-dev01/", "", value(".SDATA.file(a)18372.4.name") type("string") flags("prefix") ); }; source s_apache_www-dev01 { file("/apps/log/apache/www-dev01/*.log" program_override("apache/www/dev01") flags(no-parse) recursive(yes) ); }; destination d_server { syslog("x.x.x.x" transport("tcp") port(514)); }; log { source(s_apache_www-dev01); rewrite(r_setfilename); rewrite(r_apache_basepath_www-dev01); destination(d_server); }; *************** Server Config ************ @include "scl.conf" options { create_dirs (yes); dir_perm(0755); dir_group(60030); dir_owner(wwwtools); perm(0755); owner(wwwtools); group(60030); keep_hostname(yes); threaded(yes); log_msg_size(5000000); }; source s_network { syslog(ip(0.0.0.0) port(514) transport("tcp") max-connections(200)); }; filter f_apache_www-dev01 { program('apache/www/dev01'); }; destination d_central { file("/apps/log/syslog-ng/${PROGRAM}/${YEAR}${MONTH}${DAY}/${HOST}/${. SDATA.file(a)18372.4.name}" template("${MSGONLY}\n") ); }; log { source(s_network); filter(f_apache_www-dev01); destination(d_central); }; -- Regards, Sagar Naravane

3 3

[Bug 194] New: Can't open libsyslog-ng-crypto. so during module loading on Fedora
by bugzilla＠bugzilla.balabit.com 27 Oct '12

27 Oct '12

https://bugzilla.balabit.com/show_bug.cgi?id=194 Summary: Can't open libsyslog-ng-crypto.so during module loading on Fedora Product: syslog-ng Version: 3.3.x Platform: All OS/Version: Linux Status: ASSIGNED Severity: normal Priority: unspecified Component: syslog-ng AssignedTo: algernon(a)balabit.hu ReportedBy: algernon(a)balabit.hu CC: algernon(a)balabit.hu Type of the Report: --- Estimated Hours: 0.0 This clone is to track the library path issue separately. +++ This bug was initially created as a clone of Bug #188 +++ Linux distro: Fedora 17 x86_64 Syslog-ng source tarball: syslog-ng-3.3.5.90-20120810-v3.3.5-43-g19f6c.tar.gz we have encountered a couple of problems when running syslog-ng-3.3.5.90-20120810 under Fedora 17 x86_64: 1) ld.so library path 2) undefined symbol: tls_context_setup_session Steps to reproduce the problem: 1) just run "/usr/sbin/syslog-ng --version" ---------- # /usr/sbin/syslog-ng --version syslog-ng 3.3.5.90 Installer-Version: 3.3.5.90 Revision: Compile-Date: Aug 17 2012 22:09:52 Default-Modules: affile,afprog,afsocket,afuser,basicfuncs,csvparser,dbparser,syslogformat,afsql Error opening plugin module; module='afsocket', error='libsyslog-ng-crypto.so: cannot open shared object file: No such file or directory' Error opening plugin module; module='afsocket-notls', error='//usr/lib64/syslog-ng/libafsocket-notls.so: undefined symbol: tls_context_setup_session' Error opening plugin module; module='afsocket-tls', error='libsyslog-ng-crypto.so: cannot open shared object file: No such file or directory' Available-Modules: dummy,afsql,tfjson,dbparser,csvparser,convertfuncs,afuser,basicfuncs,afmongodb,confgen,syslogformat,affile,afprog Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-IPv6: on Enable-Spoof-Source: on Enable-TCP-Wrapper: on Enable-Linux-Caps: on Enable-Pcre: on ---------- The library path problem can be solved by creating the file "/etc/ld.so.conf.d/syslog-ng.conf" containing the syslog-ng module directory ( /usr/lib64/syslog-ng in this particular case ). Running syslog-ng under strace without touching the ld.so library paths produces: ---------- # # strace -e trace=file /usr/sbin/syslog-ng --version ... access("//usr/lib64/syslog-ng/libafsocket.so", F_OK) = 0 stat("//usr/lib64/syslog-ng/libafsocket.so", {st_mode=S_IFREG|0755, st_size=103560, ...}) = 0 open("//usr/lib64/syslog-ng/libafsocket.so", O_RDONLY|O_CLOEXEC) = 4 open("/lib64/tls/libsyslog-ng-crypto.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory) open("/lib64/libsyslog-ng-crypto.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory) open("/usr/lib64/tls/x86_64/libsyslog-ng-crypto.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory) stat("/usr/lib64/tls/x86_64", 0x7fff0f81fcb0) = -1 ENOENT (No such file or directory) open("/usr/lib64/tls/libsyslog-ng-crypto.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory) stat("/usr/lib64/tls", {st_mode=S_IFDIR|0555, st_size=4096, ...}) = 0 open("/usr/lib64/x86_64/libsyslog-ng-crypto.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory) stat("/usr/lib64/x86_64", 0x7fff0f81fcb0) = -1 ENOENT (No such file or directory) open("/usr/lib64/libsyslog-ng-crypto.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory) stat("/usr/lib64", {st_mode=S_IFDIR|0555, st_size=49152, ...}) = 0 Error opening plugin module; module='afsocket', error='libsyslog-ng-crypto.so: cannot open shared object file: No such file or directory' ... ----------- Note that all syslog-ng modules with the exception of libsyslog-ng-crypto.so are loaded from the syslog-ng modules directory. Output of "syslog-ng --version" after creating the file "/etc/ld.so.conf.d/syslog-ng.conf" ---------- # /usr/sbin/syslog-ng --version syslog-ng 3.3.5.90 Installer-Version: 3.3.5.90 Revision: Compile-Date: Aug 17 2012 22:09:52 Default-Modules: affile,afprog,afsocket,afuser,basicfuncs,csvparser,dbparser,syslogformat,afsql Error opening plugin module; module='afsocket-notls', error='//usr/lib64/syslog-ng/libafsocket-notls.so: undefined symbol: tls_context_setup_session' Available-Modules: dummy,afsocket,afsql,tfjson,dbparser,csvparser,convertfuncs,afuser,basicfuncs,afmongodb,afsocket-tls,confgen,syslogformat,affile,afprog Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-IPv6: on Enable-Spoof-Source: on Enable-TCP-Wrapper: on Enable-Linux-Caps: on Enable-Pcre: on ---------- /jpo -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 6

Syslog-ng OSE multiline messages support
by Daniel Josue Rodriguez Paiz 25 Oct '12

25 Oct '12

Hello, I have one question, does syslog-ng OSE support multiline parsing logs? i have one applications that send multiline messages and syslog-ng save the log of the first line only. Thanks for your support in adavnce, Daniel

3 4

[Bug 207] New: HOST macro contains different data inside db_parser than in the config
by bugzilla＠bugzilla.balabit.com 23 Oct '12

23 Oct '12

https://bugzilla.balabit.com/show_bug.cgi?id=207 Summary: HOST macro contains different data inside db_parser than in the config Product: syslog-ng Version: 3.3.x Platform: PC OS/Version: Linux Status: NEW Severity: normal Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: erempel(a)uvic.ca Type of the Report: --- Estimated Hours: 0.0 When using the $HOST macro in the config file of syslog-ng the macro contains source@hostname type of data. When using the $HOST macro inside the db_parser the macro contains just the hostname. This is confusing. I am not how fix this. I would not like to see the removal of the "source@" portion, but I do want to have the raw hostname available. Perhaps having a macro of $SOURCE that would have "source@hostname" and make the $HOST have just the hostname. The FULLHOST will still have the chain of hosts, including the $SOURCE, just like it has now. -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

3 5

Please help: Can not figure out why syslog-ng ose 3.2.2 server doesn't receive syslog client messages over udp/11514 on 64-bit RHEL 5
by Mehdi Mortezai 19 Oct '12

19 Oct '12

Hello, I am new to syslog-ng. I have set up a syslog-ng ose 3.2.2 server and configured syslog on 64-bit RHEL 5 client servers to send all messages to the syslog-ng server over udp/11514 (client's syslog.conf: *.* @syslog-ngIP:11514) but no packet is received. When I change the port on the syslog-ng server and on the client's syslog from udp/11514 to udp/514 all works. Verified that firewalls passing packets for both udp/514 and 11514 on the server and RHEL client and the port is not used by any other app. I cannot figure out why the logs are not happening when changing the port to udp/11514 from default? Is there any global options I should set or any change to my source statements? Here are the global options and sources definitions: *options* { time_reopen (10); long_hostnames (off); owner("root"); group("root"); perm(0600); dir_perm(0755); use_dns (yes); use_fqdn (no); create_dirs (yes); keep_hostname (no); }; *source:* ########### Define Sources *### Internal * source s_local_sys { file ("/proc/kmsg" program_override("kernel: ")); unix-stream ("/dev/log"); internal();}; *### External or remote:* # For Server Groups source s_RHEL5_udp { udp(port(11514));}; *# For all other single hosts:* source s_remote_udp { udp(port(514));}; Thanks in advance for your help. Medi

1 0

GeoIP template function
by Csaba Major 19 Oct '12

19 Oct '12

Hi, I have a system where I have a lot of logs with IP addresses (typically firewall logs), and the requirement was to also include the country name in the logs. I found that a geoip based template function would be an easy way for this, so I created a PoC for that. It works pretty well, however some improvements and performance measurement would be necessary. The created patch is attached (based on 3.3.5), some may find it useful :) (It requires libgeoip, and a geoip database, of course) Regards, Csaba

4 6

Can't get kernel messages on FreeBSD
by Tomoki BENIYA 19 Oct '12

19 Oct '12

Hi all. I installed syslog-ng3.3.6_3 using ports on FreeBSD9.0-RELEASE(i386). I can't get kernel messages by default configuration. ex. em0: link state changed to DOWN em0: link state changed to UP But, I can see them using 'cat /dev/klog' after stoppedsyslog-ng. So, I think syslog-ng can't get kermel messages from /dev/klog. How to get kernel messages using syslog-ng? Please give me a hint or something! * /etc/rc.conf syslogd_enable="NO" syslog_ng_enable="YES" * /usr/local/etc/syslog-ng.conf source s_kern { file("/dev/klog"); }; filter f_kern { facility(kern); }; destination d_kern { file("/var/log/kern.log"); }; log { source(s_kern); filter(f_kern); destination(d_kern); }; -- Tomoki BENIYA <beniya(a)bit-isle.co.jp>

2 2

syslog-ng 3.05 receive but not write on filesystem
by FIORENZI ALESSANDRO 17 Oct '12

17 Oct '12

Hi, I am experiencing e problem that havent have before. Are 3 years that all seems to work fine with syslog-ng 3.05. recently I have find that some of my servers sends syslog messages to syslog server, Syslog server interface recive it, syslog-ng elaborate it but do not write on file system. I have configured syslog-ng server to log message in the following manner /opt/logs/$YEAR/$MONTH/$DAY/SERVER/$HOST/messages What I have done: - Listen on eth4 to look for udp syslog messages coming form server A - Try to access via ssh to server A - I have seen udp packet from A to syslog-server. Than I have investigate on syslog-ng - I have get the pid of syslog-ng and make a strace -p pid -o prova.txt - Try to access via ssh to server A (to let server a send udp syslog message to syslog server) - Stop strace - Look on prova.txt for IP of server A and find It! But I cannot do more, can someone help me? Alessandro Fiorenzi Prima di stampare, pensa all'ambiente ** Think about the environment before printing ________________________________ Il presente messaggio, inclusi gli eventuali allegati, ha natura aziendale e potrebbe contenere informazioni confidenziali e/o riservate. Chiunque lo ricevesse per errore, ? pregato di avvisare tempestivamente il mittente e di cancellarlo. E' strettamente vietata qualsiasi forma di utilizzo, riproduzione o diffusione non autorizzata del contenuto di questo messaggio o di parte di esso. Pur essendo state assunte le dovute precauzioni per ridurre al minimo il rischio di trasmissione di virus, si suggerisce di effettuare gli opportuni controlli sui documenti allegati al presente messaggio. Non si assume alcuna responsabilit? per eventuali danni o perdite derivanti dalla presenza di virus. Per lo svolgimento delle attivit? di investimento nel Regno Unito, la societ? ? autorizzata da Banca d'Italia ed ? soggetta alla vigilanza limitata della Financial Services Authority. Maggiori informazioni in merito ai poteri di vigilanza della Financial Services Authority sono a disposizione previa richiesta.. Nel Regno Unito Intesa Sanpaolo S.p.A. opera attraverso la filiale di Londra, sita in 90 Queen Street, London EC4N 1SA, registrata in Inghilterra & Galles sotto No.FC016201, Branch No.BR000036 In osservanza dei requisito imposti dal Internal Revenue Service (Agenzia delle Entrate degli Stati Uniti), qualunque discussione relativa a temi di natura fiscale contenuta in questo messaggio o nei suoi allegati non e' intesa ne' e' stata scritta per essere utilizzata, ne' puo' essere utilizata per (i) evitare l'imposizione di gravami fiscali secondo il codice tributario vigente negli Stati Uniti o (ii) per promuovere, sollecitare o raccomandare una operazione finanziaria o altra transazione indirizzata ad un altro destinatario. *** ________________________________ This email (including any attachment) is a corporate message and may contain confidential and/or privileged and/or proprietary information. If you have received this email in error, please notify the sender immediately, do not use or share it and destroy this email. Any unauthorised use, copying or disclosure of the material in this email or of parts hereof (including reliance thereon) is strictly forbidden. We have taken precautions to minimize the risk of transmitting software viruses but nevertheless advise you to carry out your own virus checks on any attachment of this message. We accept no liability for loss or damage caused by software viruses. For the conduct of investment business in the UK, the Company is authorised by Banca d'Italia and subject to limited regulation in the UK by the Financial Services Authority. Details about the extent of our regulation by the Financial Services Authority are available from us on request. In the UK Intesa Sanpaolo S.p.A. operates through its London Branch, located at 90 Queen Street, London EC4N 1SA. Registered in England & Wales under No.FC016201, Branch No.BR000036 To comply with requirements imposed by the IRS, we inform you that any discussion of U.S. federal tax issues contained herein (including any attachments) was not intended or written to be used, and cannot be used by you, for the purpose of (i) avoiding penalties under the Internal Revenue Code or (ii) promoting, marketing or recommending any transaction or matter addressed herein to another party.

2 1

Debian package news for syslog-ng 3.3 and 3.4
by Gergely Nagy 16 Oct '12

16 Oct '12

Hi! I've been providing syslog-ng 3.3 packages for a while[1], more than a year, actually! And since I announced my intention to package syslog-ng 3.4 in January[2], I believe it's time to give an update on where things stand, and what to expect from the repository in the coming weeks, months. There are two important news I want to share: - Ubuntu Natty (11.04) support will soon be discontinued. Official support for the platform ends this month, the repository will not see any further updates. I plan to completely remove it by December, but the most recent package already built (3.3.7RC2) will be available until then. (Though, since building for Natty is easy, I might end up uploading 3.3.7 final, before the nail hits the coffin.) - syslog-ng 3.4 packages are coming, built from my sandbox branch, which is a collection of everything that's intended to reach the master branch, and are in already usable state. These include: an amqp destination, indented-multiline and linux 3.5+ /dev/kmsg support, along with a host of bugfixes. The location of the repository will be: deb http://packages.madhouse-project.org/$VENDOR $RELEASE syslog-ng-devel Where $VENDOR is debian or ubuntu, and release is the particular OS release (wheezy, precise, etc). Initially packages will be provided for Ubuntu Oneiric, Precise and Quantal, along with Debian Squeeze, Wheezy and unstable. Ubuntu Lucid and Natty were dropped from the list compared to 3.3. Natty due to reaching its end of life, and Lucid because backporting all the required bits and pieces to it took up more time than all the rest combined. If there's a significant interest in syslog-ng 3.4 packages for Ubuntu Lucid, please let me know. A mass of demanding users can always convince me that the pain of backporting to lucid is worth it. The syslog-ng 3.3.7RC2 packages are already in the repositories, 3.4.0alpha3 + patches is building as I write this, and it's very likely they'll hit my archive tomorrow latest. The syslog-ng-devel component is already created, and I'll soon start populating it with the required dependencies on platforms that need them. I've yet to figure out how to change my repository page[3] to allow convenient selection of the various bits and pieces, so the information there may lag behind for a little while. [1]: http://article.gmane.org/gmane.comp.syslog-ng/11900 [2]: http://article.gmane.org/gmane.comp.syslog-ng/12990 [3]: http://asylum.madhouse-project.org/projects/debian/ Finally, I'd like to share a few interesting statistics, a result of a few greps through my server logs: Since August 20, 2011, there have been 119 unique downloaders of the Debian packages, 614 downloads in total (spanning all versions from 3.3.0beta2 up until 3.3.6.91). For Ubuntu, these numbers are 452 unique downloaders, and 1563 downloads in total. I'm quite happy about those numbers, these debs are by far the most downloaded things I have created in the past decade (not counting one particular package in the Debian archive I try to forget). :) So a huge "Thank you!" goes out to everyone who uses these packages! -- |8]

1 1

syslog-ng 3.3.7 RC2 (3.3.6.91) released
by Gergely Nagy 15 Oct '12

15 Oct '12

Hi! While the 3.3.7 RC1 looked promising, a few more issues turned up meanwhile, so I am releasing an RC2 aswell. With the credits section of the NEWS file being longer than the list of fixes, my hope is that the 3.3.7 final release (due october 31) will not be much more than a version bump. As usual, testing this release candidate and reporting any and all issues found with it - be those regressions or anything else - would be greatly appreciated! The current state of the NEWS file is included below, and if anyone wants to track how the release is progressing and what needs to be fixed still before the final release, see the 3.3.7 target milestone on bugzilla: https://bugzilla.balabit.com/buglist.cgi?query_format=advanced&target_miles… The release is available from the following sources: GitHub ------ The release is tagged as v3.3.6.91 on the official repository at git://github.com/balabit/syslog-ng-3.3.git A detailed list of commits & changes is available via the compare view: https://github.com/balabit/syslog-ng-3.3/compare/v3.3.6.90...v3.3.6.91 Tarball ------- A make dist tarball built from the same tag is available from my package repository: http://packages.madhouse-project.org/syslog-ng/3.3/3.3.6.91/syslog-ng-3.3.6… And finally, the NEWS file from the release: 3.3.6.91 (3.3.7 RC2) Mon, 08 Oct 2012 13:00:00 +0200 Bugfixes ======== * The bundled ivykis has been updated, the new version contains a workaround for an issue specific to FreeBSD 9 only. * The file source was corrected to properly handle character devices. This, and the ivykis update fixes a CPU spinning issue on FreeBSD. [#201] * The sun-streams module had a file descriptor leak, which has been corrected too. [#151] * Various minor fixes around the build system. Features ======== * The stats will now list the filename of unix domain sockets. [#195] Credits ======= syslog-ng is developed as a community project, and as such it relies on volunteers to do the work necessarily to produce syslog-ng. Reporting bugs, testing changes, writing code or simply providing feedback are all important contributions, so please if you are a user of syslog-ng, contribute. These people have helped in this release: Attila Nagy <bra(a)fsn.hu> Balazs Scheidler <bazsi(a)balabit.hu> Evan Rempel <erempel(a)uvic.ca> Gergely Nagy <algernon(a)balabit.hu> Lennert Buytenhek <buytenh(a)wantstofly.org> Michael Hocke <michael.hocke(a)nyu.edu> Peter Czanik <czanik(a)balabit.hu> -- |8]

3 6