October 2012 - syslog-ng - lists.balabit.hu

syslog-ng 3.05 receive but not write on filesystem
by FIORENZI ALESSANDRO 17 Oct '12

17 Oct '12

Hi, I am experiencing e problem that havent have before. Are 3 years that all seems to work fine with syslog-ng 3.05. recently I have find that some of my servers sends syslog messages to syslog server, Syslog server interface recive it, syslog-ng elaborate it but do not write on file system. I have configured syslog-ng server to log message in the following manner /opt/logs/$YEAR/$MONTH/$DAY/SERVER/$HOST/messages What I have done: - Listen on eth4 to look for udp syslog messages coming form server A - Try to access via ssh to server A - I have seen udp packet from A to syslog-server. Than I have investigate on syslog-ng - I have get the pid of syslog-ng and make a strace -p pid -o prova.txt - Try to access via ssh to server A (to let server a send udp syslog message to syslog server) - Stop strace - Look on prova.txt for IP of server A and find It! But I cannot do more, can someone help me? Alessandro Fiorenzi Prima di stampare, pensa all'ambiente ** Think about the environment before printing ________________________________ Il presente messaggio, inclusi gli eventuali allegati, ha natura aziendale e potrebbe contenere informazioni confidenziali e/o riservate. Chiunque lo ricevesse per errore, ? pregato di avvisare tempestivamente il mittente e di cancellarlo. E' strettamente vietata qualsiasi forma di utilizzo, riproduzione o diffusione non autorizzata del contenuto di questo messaggio o di parte di esso. Pur essendo state assunte le dovute precauzioni per ridurre al minimo il rischio di trasmissione di virus, si suggerisce di effettuare gli opportuni controlli sui documenti allegati al presente messaggio. Non si assume alcuna responsabilit? per eventuali danni o perdite derivanti dalla presenza di virus. Per lo svolgimento delle attivit? di investimento nel Regno Unito, la societ? ? autorizzata da Banca d'Italia ed ? soggetta alla vigilanza limitata della Financial Services Authority. Maggiori informazioni in merito ai poteri di vigilanza della Financial Services Authority sono a disposizione previa richiesta.. Nel Regno Unito Intesa Sanpaolo S.p.A. opera attraverso la filiale di Londra, sita in 90 Queen Street, London EC4N 1SA, registrata in Inghilterra & Galles sotto No.FC016201, Branch No.BR000036 In osservanza dei requisito imposti dal Internal Revenue Service (Agenzia delle Entrate degli Stati Uniti), qualunque discussione relativa a temi di natura fiscale contenuta in questo messaggio o nei suoi allegati non e' intesa ne' e' stata scritta per essere utilizzata, ne' puo' essere utilizata per (i) evitare l'imposizione di gravami fiscali secondo il codice tributario vigente negli Stati Uniti o (ii) per promuovere, sollecitare o raccomandare una operazione finanziaria o altra transazione indirizzata ad un altro destinatario. *** ________________________________ This email (including any attachment) is a corporate message and may contain confidential and/or privileged and/or proprietary information. If you have received this email in error, please notify the sender immediately, do not use or share it and destroy this email. Any unauthorised use, copying or disclosure of the material in this email or of parts hereof (including reliance thereon) is strictly forbidden. We have taken precautions to minimize the risk of transmitting software viruses but nevertheless advise you to carry out your own virus checks on any attachment of this message. We accept no liability for loss or damage caused by software viruses. For the conduct of investment business in the UK, the Company is authorised by Banca d'Italia and subject to limited regulation in the UK by the Financial Services Authority. Details about the extent of our regulation by the Financial Services Authority are available from us on request. In the UK Intesa Sanpaolo S.p.A. operates through its London Branch, located at 90 Queen Street, London EC4N 1SA. Registered in England & Wales under No.FC016201, Branch No.BR000036 To comply with requirements imposed by the IRS, we inform you that any discussion of U.S. federal tax issues contained herein (including any attachments) was not intended or written to be used, and cannot be used by you, for the purpose of (i) avoiding penalties under the Internal Revenue Code or (ii) promoting, marketing or recommending any transaction or matter addressed herein to another party.

2 1

Debian package news for syslog-ng 3.3 and 3.4
by Gergely Nagy 16 Oct '12

16 Oct '12

Hi! I've been providing syslog-ng 3.3 packages for a while[1], more than a year, actually! And since I announced my intention to package syslog-ng 3.4 in January[2], I believe it's time to give an update on where things stand, and what to expect from the repository in the coming weeks, months. There are two important news I want to share: - Ubuntu Natty (11.04) support will soon be discontinued. Official support for the platform ends this month, the repository will not see any further updates. I plan to completely remove it by December, but the most recent package already built (3.3.7RC2) will be available until then. (Though, since building for Natty is easy, I might end up uploading 3.3.7 final, before the nail hits the coffin.) - syslog-ng 3.4 packages are coming, built from my sandbox branch, which is a collection of everything that's intended to reach the master branch, and are in already usable state. These include: an amqp destination, indented-multiline and linux 3.5+ /dev/kmsg support, along with a host of bugfixes. The location of the repository will be: deb http://packages.madhouse-project.org/$VENDOR $RELEASE syslog-ng-devel Where $VENDOR is debian or ubuntu, and release is the particular OS release (wheezy, precise, etc). Initially packages will be provided for Ubuntu Oneiric, Precise and Quantal, along with Debian Squeeze, Wheezy and unstable. Ubuntu Lucid and Natty were dropped from the list compared to 3.3. Natty due to reaching its end of life, and Lucid because backporting all the required bits and pieces to it took up more time than all the rest combined. If there's a significant interest in syslog-ng 3.4 packages for Ubuntu Lucid, please let me know. A mass of demanding users can always convince me that the pain of backporting to lucid is worth it. The syslog-ng 3.3.7RC2 packages are already in the repositories, 3.4.0alpha3 + patches is building as I write this, and it's very likely they'll hit my archive tomorrow latest. The syslog-ng-devel component is already created, and I'll soon start populating it with the required dependencies on platforms that need them. I've yet to figure out how to change my repository page[3] to allow convenient selection of the various bits and pieces, so the information there may lag behind for a little while. [1]: http://article.gmane.org/gmane.comp.syslog-ng/11900 [2]: http://article.gmane.org/gmane.comp.syslog-ng/12990 [3]: http://asylum.madhouse-project.org/projects/debian/ Finally, I'd like to share a few interesting statistics, a result of a few greps through my server logs: Since August 20, 2011, there have been 119 unique downloaders of the Debian packages, 614 downloads in total (spanning all versions from 3.3.0beta2 up until 3.3.6.91). For Ubuntu, these numbers are 452 unique downloaders, and 1563 downloads in total. I'm quite happy about those numbers, these debs are by far the most downloaded things I have created in the past decade (not counting one particular package in the Debian archive I try to forget). :) So a huge "Thank you!" goes out to everyone who uses these packages! -- |8]

1 1

syslog-ng 3.3.7 RC2 (3.3.6.91) released
by Gergely Nagy 15 Oct '12

15 Oct '12

Hi! While the 3.3.7 RC1 looked promising, a few more issues turned up meanwhile, so I am releasing an RC2 aswell. With the credits section of the NEWS file being longer than the list of fixes, my hope is that the 3.3.7 final release (due october 31) will not be much more than a version bump. As usual, testing this release candidate and reporting any and all issues found with it - be those regressions or anything else - would be greatly appreciated! The current state of the NEWS file is included below, and if anyone wants to track how the release is progressing and what needs to be fixed still before the final release, see the 3.3.7 target milestone on bugzilla: https://bugzilla.balabit.com/buglist.cgi?query_format=advanced&target_miles… The release is available from the following sources: GitHub ------ The release is tagged as v3.3.6.91 on the official repository at git://github.com/balabit/syslog-ng-3.3.git A detailed list of commits & changes is available via the compare view: https://github.com/balabit/syslog-ng-3.3/compare/v3.3.6.90...v3.3.6.91 Tarball ------- A make dist tarball built from the same tag is available from my package repository: http://packages.madhouse-project.org/syslog-ng/3.3/3.3.6.91/syslog-ng-3.3.6… And finally, the NEWS file from the release: 3.3.6.91 (3.3.7 RC2) Mon, 08 Oct 2012 13:00:00 +0200 Bugfixes ======== * The bundled ivykis has been updated, the new version contains a workaround for an issue specific to FreeBSD 9 only. * The file source was corrected to properly handle character devices. This, and the ivykis update fixes a CPU spinning issue on FreeBSD. [#201] * The sun-streams module had a file descriptor leak, which has been corrected too. [#151] * Various minor fixes around the build system. Features ======== * The stats will now list the filename of unix domain sockets. [#195] Credits ======= syslog-ng is developed as a community project, and as such it relies on volunteers to do the work necessarily to produce syslog-ng. Reporting bugs, testing changes, writing code or simply providing feedback are all important contributions, so please if you are a user of syslog-ng, contribute. These people have helped in this release: Attila Nagy <bra(a)fsn.hu> Balazs Scheidler <bazsi(a)balabit.hu> Evan Rempel <erempel(a)uvic.ca> Gergely Nagy <algernon(a)balabit.hu> Lennert Buytenhek <buytenh(a)wantstofly.org> Michael Hocke <michael.hocke(a)nyu.edu> Peter Czanik <czanik(a)balabit.hu> -- |8]

3 6

[PATCH (3.4)] json: The test should not depend on the environment's timezone
by Gergely Nagy 13 Oct '12

13 Oct '12

The $(format-json) test was failing when a timezone was set in the environment that did not match the tests expectations. To fix this, explicitly set the timezone to UTC. Signed-off-by: Gergely Nagy <algernon(a)balabit.hu> --- modules/json/tests/test_json.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/modules/json/tests/test_json.c b/modules/json/tests/test_json.c index f188b7f..15d038b 100644 --- a/modules/json/tests/test_json.c +++ b/modules/json/tests/test_json.c @@ -7,7 +7,7 @@ test_format_json(void) { assert_template_format("$(format-json MSG=$MSG)", "{\"MSG\":\"árvíztűrőtükörfúrógép\"}"); assert_template_format_with_context("$(format-json MSG=$MSG)", "{\"MSG\":\"árvíztűrőtükörfúrógép\"}{\"MSG\":\"árvíztűrőtükörfúrógép\"}"); - assert_template_format("$(format-json --scope rfc3164)", "{\"PROGRAM\":\"syslog-ng\",\"PRIORITY\":\"err\",\"PID\":\"23323\",\"MESSAGE\":\"árvíztűrőtükörfúrógép\",\"HOST\":\"bzorp\",\"FACILITY\":\"local3\",\"DATE\":\"Feb 11 19:58:35\"}"); + assert_template_format("$(format-json --scope rfc3164)", "{\"PROGRAM\":\"syslog-ng\",\"PRIORITY\":\"err\",\"PID\":\"23323\",\"MESSAGE\":\"árvíztűrőtükörfúrógép\",\"HOST\":\"bzorp\",\"FACILITY\":\"local3\",\"DATE\":\"Feb 11 18:58:35\"}"); assert_template_format("$(format-json msg.text=$MSG msg.id=42 host=bzorp)", "{\"msg\":{\"text\":\"árvíztűrőtükörfúrógép\",\"id\":\"42\"},\"host\":\"bzorp\"}"); assert_template_format("$(format-json msg.text.str=$MSG msg.text.len=42 msg.id=42 host=bzorp)", "{\"msg\":{\"text\":{\"str\":\"árvíztűrőtükörfúrógép\",\"len\":\"42\"},\"id\":\"42\"},\"host\":\"bzorp\"}"); @@ -17,6 +17,8 @@ int main(int argc G_GNUC_UNUSED, char *argv[] G_GNUC_UNUSED) { app_startup(); + putenv("TZ=UTC"); + tzset(); init_template_tests(); plugin_load_module("json-plugin", configuration, NULL); -- 1.7.10.4

1 0

[PATCH (3.4)] json-format: Unbreak $(format-json)
by Gergely Nagy 13 Oct '12

13 Oct '12

The $(format-json) template function was misbehaving under certain conditions, namely where we had a sub-object, a simple value, and another object starting right after: in this case, no comma was placed between the simple value and the name of the new sub-object. The fix changes how comma placement is done completely: it is no longer a mix of prefix_data & user_data magic, but all state is kept in the user data, with the _obj_start(), _obj_end() and _value() functions setting it as appropriate: - When starting a new object, if we need a comma, we'll put one, and turn the comma flag off (the first item within an object won't need a comma). - When ending an object, we never add a comma, but the next item will need the comma flag set. - When adding a value, if the comma flag is set, we add one, and in the end, set the flag ourselves, as the next one will need a comma too. This is much simpler than the previous hackery, and also results in correct JSON in every situation. Signed-off-by: Gergely Nagy <algernon(a)balabit.hu> --- modules/json/format-json.c | 27 +++++---------------------- modules/json/tests/test_json.c | 2 ++ 2 files changed, 7 insertions(+), 22 deletions(-) diff --git a/modules/json/format-json.c b/modules/json/format-json.c index 4f0f01c..e17bcb0 100644 --- a/modules/json/format-json.c +++ b/modules/json/format-json.c @@ -118,14 +118,8 @@ tf_json_obj_start(const gchar *name, gpointer user_data) { json_state_t *state = (json_state_t *)user_data; - gboolean need_comma = FALSE; - if (prefix_data) - need_comma = GPOINTER_TO_INT(*prefix_data); - else - need_comma = state->need_comma; - - if (need_comma) + if (state->need_comma) g_string_append_c(state->buffer, ','); if (name) @@ -133,13 +127,11 @@ tf_json_obj_start(const gchar *name, g_string_append_c(state->buffer, '"'); g_string_append_escaped(state->buffer, name); g_string_append(state->buffer, "\":{"); - state->need_comma = TRUE; } else g_string_append_c(state->buffer, '{'); - if (prefix_data) - *prefix_data=GINT_TO_POINTER(0); + state->need_comma = FALSE; return FALSE; } @@ -152,11 +144,10 @@ tf_json_obj_end(const gchar *name, { json_state_t *state = (json_state_t *)user_data; - if (prev_data) - *prev_data = GINT_TO_POINTER(1); - g_string_append_c(state->buffer, '}'); + state->need_comma = TRUE; + return FALSE; } @@ -164,18 +155,10 @@ static gboolean tf_json_value(const gchar *name, const gchar *prefix, const gchar *value, gpointer *prefix_data, gpointer user_data) { - gboolean need_comma = FALSE; json_state_t *state = (json_state_t *)user_data; - if (prefix_data) - need_comma = GPOINTER_TO_INT(*prefix_data); - else - need_comma = state->need_comma; - - if (need_comma) + if (state->need_comma) g_string_append_c(state->buffer, ','); - else if (prefix_data) - *prefix_data = GINT_TO_POINTER(1); g_string_append_c(state->buffer, '"'); g_string_append_escaped(state->buffer, name); diff --git a/modules/json/tests/test_json.c b/modules/json/tests/test_json.c index f188b7f..2a88c63 100644 --- a/modules/json/tests/test_json.c +++ b/modules/json/tests/test_json.c @@ -11,6 +11,8 @@ test_format_json(void) assert_template_format("$(format-json msg.text=$MSG msg.id=42 host=bzorp)", "{\"msg\":{\"text\":\"árvíztűrőtükörfúrógép\",\"id\":\"42\"},\"host\":\"bzorp\"}"); assert_template_format("$(format-json msg.text.str=$MSG msg.text.len=42 msg.id=42 host=bzorp)", "{\"msg\":{\"text\":{\"str\":\"árvíztűrőtükörfúrógép\",\"len\":\"42\"},\"id\":\"42\"},\"host\":\"bzorp\"}"); + assert_template_format("$(format-json kernel.SUBSYSTEM=pci kernel.DEVICE.type=pci kernel.DEVICE.name=0000:02:00.0 MSGID=801 MESSAGE=test)", + "{\"kernel\":{\"SUBSYSTEM\":\"pci\",\"DEVICE\":{\"type\":\"pci\",\"name\":\"0000:02:00.0\"}},\"MSGID\":\"801\",\"MESSAGE\":\"test\"}"); } int -- 1.7.10.4

1 0

[RFC (3.4) 0/5]: multiline & linux 3.5+ /dev/kmsg support
by Gergely Nagy 13 Oct '12

13 Oct '12

The next few patches gradually introduce two - in my opinion - interesting features for syslog-ng 3.4: support for one kind of multiline messages, where continuation lines start with whitespace; and support for the /dev/kmsg format introduced in linux 3.5. Neither implementation is complete yet, there are subtle issues with both (see later), but they're at a stage where taking them to a test drive would be useful, and code review wouldn't hurt either. The patches are also available on the feature/3.4/indented-multiline[1] branch of my git repository[2]. The branch may get rebased in the future, I do not make any promise of keeping it fast-forwardable. [1]: https://github.com/algernon/syslog-ng/commits/feature/3.4/indented-multiline [2]: git://github.com/algernon/syslog-ng.git linux 3.5 /dev/kmsg =================== The linux 3.5+ /dev/kmsg support is the easier to test: you only need a 3.5+ kernel, and when using the system() source, things will magically work! This includes parsing the relative timestamps in kmsg messages and also parsing any additional key-value pairs. This means that if we had an input line like this: ,---- | 6,802,65338577;ATL1E 0000:02:00.0: eth0: NIC Link is Up <100 Mbps Full Duplex> | SUBSYSTEM=pci | DEVICE=+pci:0000:02:00.0 `---- Then we'll get back something like this: ,---- | { | "FILE_NAME" : "/dev/kmsg", | "MSGID" : "802", | "HOST" : "luthien", | "DATE" : "Oct 13 16:53:44", | "FACILITY" : "kern", | "SOURCEIP" : "127.0.0.1", | "TAGS" : ".source.s_kmsg", | "kernel" : { | "SUBSYSTEM" : "pci", | "DEVICE" : { | "name" : "0000:02:00.0", | "type" : "pci" | } | }, | "PRIORITY" : "info", | "HOST_FROM" : "luthien", | "MESSAGE" : "ATL1E 0000:02:00.0: eth0: NIC Link is Up <100 Mbps Full Duplex>", | "PROGRAM" : "kernel" | } `---- For the above, the following template was used: $(format-json --scope selected-macros --scope nv-pairs --key .kernel.* --shift 1) (Mind you, program_override() is currently broken in 3.4, but that is unrelated to the /dev/kmsg support) Any name-value pairs the kernel supplies will be put into variables of the same name, prefixed with ".kernel." (so SUBSYSTEM becomes .kernel.SUBSYSTEM), and the value of DEVICE= will be further parsed, based on rules written down in the appropriate kernel docs. Indented multiline ================== The indented multiline support enhances the file() and tcp() sources with the ability to read multiline records (if the indented-multiline flag is set), where continuation lines start with whitespace. Such output is produced by the $(indent-multi-line) template function. Usage: source s_ml { tcp(port(12345) flags(indented-multiline, no-parse)); }; Then, you can test it like this: (cat <<EOF This is the first line - second - third EOF ) | nc localhost 12345 Using a JSON output, this would result in: {"MESSAGE":"This is the first line\n - second\n - third"} A record can be terminated in two ways: by another line that does not begin with whitespace, or by reacing EOF. This means that if netcat didn't close the connection, we wouldn't get the test record until another line arrives. Unfortunately, there's not much I can do about this limitation, but if anyone has a good idea how to work around or fix it, please let me know! Known issues ============ The /dev/kmsg handling is not reload/restart-safe: if syslog-ng gets reloaded or restarted, it will read the whole lot of /dev/kmsg over again, for example. There are possibly other subtle issues too, but all of those will be ironed out before I submit the patches for merging. The indented-multiline support has a more serious limitation: if data needs to be split (which happens in some circumstances when the incoming data is bigger than the internal message size (8192 by default)), it is not correctly reassembled, making indented-multiline less than useful for some tasks.

1 5

[Bug 202] New: pdbtool dump does not recognize the -p option
by bugzilla＠bugzilla.balabit.com 13 Oct '12

13 Oct '12

https://bugzilla.balabit.com/show_bug.cgi?id=202 Summary: pdbtool dump does not recognize the -p option Product: syslog-ng Version: 3.3.x Platform: PC OS/Version: Linux Status: NEW Severity: normal Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: erempel(a)uvic.ca Type of the Report: --- Estimated Hours: 0.0 Man page states dump [options] Display the RADIX tree built from the patterns. This shows how are the patterns represented in syslog-ng and it might also help to track down pattern-matching problems. The dump utility can dump the tree used for matching the PROGRAM or the MSG parts. --debug or -d Enable debug/diagnostic messages on stderr. --pdb or -p Name of the pattern database file to use. --program or -P Displays the RADIX tree built from the patterns belonging to the $PROGRAM application. --program-tree or -T Display the $PROGRAM tree. --verbose or -v Enable verbose messages on stderr. Example and sample output: But the command does localhost% pdbtool dump -P fscheck -p patterndb.xml Error parsing command line arguments: Unknown option -p localhost% pdbtool dump -P fscheck --pdb patterndb.xml Error parsing command line arguments: Unknown option --pdb -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 2

[Bug 201] New: system() does not work on FreeBSD: /dev/ klog is pretty much ignored
by bugzilla＠bugzilla.balabit.com 13 Oct '12

13 Oct '12

https://bugzilla.balabit.com/show_bug.cgi?id=201 Summary: system() does not work on FreeBSD: /dev/klog is pretty much ignored Product: syslog-ng Version: 3.3.x Platform: All OS/Version: FreeBSD Status: ASSIGNED Severity: normal Priority: unspecified Component: syslog-ng AssignedTo: algernon(a)balabit.hu ReportedBy: algernon(a)balabit.hu Type of the Report: --- Estimated Hours: 0.0 While debugging another problem, I found a couple of issues with system() on FreeBSD: * follow-freq(0) doesn't appear to work, so /dev/klog gets ignored outright * Even with follow-freq(1), we never read from /dev/klog, because stat() will report its size to be 0, so we never end up reading from it. The second is easy to fix, adding an || !S_ISREG(st.st_mode) to the appropriate condition in log_reader_io_follow_file() does the right thing. -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 4

[Bug 200] New: logging does not work on FreeBSD
by bugzilla＠bugzilla.balabit.com 13 Oct '12

13 Oct '12

https://bugzilla.balabit.com/show_bug.cgi?id=200 Summary: logging does not work on FreeBSD Product: syslog-ng Version: 3.4.x Platform: PC OS/Version: FreeBSD Status: NEW Severity: normal Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: czanik(a)balabit.hu Type of the Report: --- Estimated Hours: 0.0 Using the latest syslog-ng 3.4 git, with Algernon sandbox patches applied (to be able to generate a tar.gz dist file), logging does not seem to work. Without command line arguments, nothing gets logged, except for version information at startup. When using "-vdF", logs coming from the network are logged. Tested on FreeBSD 9.0 with a slightly modified sysutils/syslog-ng-rc port (removed utmp magic from ports Makefile). -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 6

cisco rewrite code
by Evan Rempel 12 Oct '12

12 Oct '12

Does anyone have a pre build set of patterns/rewrite rule to rewrite all cisco logs into something that is a little more compliant? We are trying to use a master pattern database to identify/classify messages, but the cisco logs don't have usable "program names" so the pattern database can't even get started :-( Thanks for any pointers. -- Evan

4 3