May 2011 - syslog-ng - lists.balabit.hu

Running script for checking syslog-ng status
by Henry Xu 26 May '11

26 May '11

Hi All, Does syslog-ng have a running script that check the syslog-ng running status? If it crashed or stopped running,this script can make sure to start a new syslog-ng process. This morning,the syslog-ng on my server crashed,and somehow resulted in the server halt. Thanks. Best, Henry

4 5

[PATCH] tfjson: Support either json-c or json-glib.
by Gergely Nagy 25 May '11

25 May '11

Add the ability to build the module with either json-c, or json-glib (version 0.12 or later). If both are present, we'll prefer json-c. Requested-by: Peter Czanik <czanik(a)balabit.hu> Signed-off-by: Gergely Nagy <algernon(a)balabit.hu> --- configure.in | 19 ++++++++++-- modules/tfjson/tfjson.c | 73 ++++++++++++++++++++++++++++++++++++++--------- 2 files changed, 75 insertions(+), 17 deletions(-) diff --git a/configure.in b/configure.in index 9c03b65..87de97f 100644 --- a/configure.in +++ b/configure.in @@ -25,7 +25,8 @@ EVTLOG_MIN_VERSION="0.2.12" OPENSSL_MIN_VERSION="0.9.8" LIBDBI_MIN_VERSION="0.8.0" IVYKIS_MIN_VERSION="0.18" -JSON_MIN_VERSION="0.7" +JSON_C_MIN_VERSION="0.7" +JSON_GLIB_MIN_VERSION="0.12" PCRE_MIN_VERSION="6.1" dnl *************************************************************************** @@ -563,7 +564,10 @@ fi dnl *************************************************************************** dnl json headers/libraries dnl *************************************************************************** -PKG_CHECK_MODULES(JSON, json >= $JSON_MIN_VERSION,, JSON_LIBS="") +PKG_CHECK_MODULES(JSON, json >= $JSON_C_MIN_VERSION, JSON_TYPE="C", JSON_LIBS="") +if test "x$JSON_TYPE" = "x"; then + PKG_CHECK_MODULES(JSON, json-glib-1.0 >= $JSON_GLIB_MIN_VERSION, JSON_TYPE="glib", JSON_LIBS="") +fi dnl *************************************************************************** dnl pcre headers/libraries @@ -813,7 +817,7 @@ if test "x$enable_json" = "xauto"; then enable_json="yes" fi elif test "x$enable_json" = "xyes" -a -z "$JSON_LIBS"; then - AC_MSG_ERROR(Cannot find json-c version >= $JSON_MIN_VERSION: is pkg-config in path?) + AC_MSG_ERROR(Cannot find json-c version >= $JSON_C_MIN_VERSION or json-glib-1.0 >= $JSON_GLIB_MIN_VERSION: is pkg-config in path?) fi if test "x$enable_systemd" = "xauto"; then @@ -964,6 +968,15 @@ AC_DEFINE_UNQUOTED(ENABLE_PCRE, `enable_value $enable_pcre`, [Enable PCRE suppor AC_DEFINE_UNQUOTED(ENABLE_ENV_WRAPPER, `enable_value $enable_env_wrapper`, [Enable environment wrapper support]) AC_DEFINE_UNQUOTED(ENABLE_SYSTEMD, `enable_value $enable_systemd`, [Enable systemd support]) +case "$JSON_TYPE" in + "C") + AC_DEFINE_UNQUOTED(HAVE_JSON_C, 1, [Have json-c]) + ;; + "glib") + AC_DEFINE_UNQUOTED(HAVE_JSON_GLIB, 1, [Have json-glib]) + ;; +esac + AM_CONDITIONAL(ENABLE_ENV_WRAPPER, [test "$enable_env_wrapper" = "yes"]) AM_CONDITIONAL(ENABLE_SYSTEMD, [test "$enable_systemd" = "yes"]) AM_CONDITIONAL(ENABLE_SSL, [test "$enable_ssl" = "yes"]) diff --git a/modules/tfjson/tfjson.c b/modules/tfjson/tfjson.c index 07fc76e..7155f15 100644 --- a/modules/tfjson/tfjson.c +++ b/modules/tfjson/tfjson.c @@ -25,7 +25,15 @@ #include "cfg.h" #include "value-pairs.h" +#include "config.h" + +#ifdef HAVE_JSON_C #include <json.h> +#endif + +#ifdef HAVE_JSON_GLIB +#include <json-glib/json-glib.h> +#endif static gboolean tf_json_prepare(LogTemplateFunction *self, LogTemplate *parent, @@ -35,6 +43,8 @@ tf_json_prepare(LogTemplateFunction *self, LogTemplate *parent, { ValuePairs *vp; + g_type_init (); + vp = value_pairs_new_from_cmdline (parent->cfg, argc, argv, error); if (!vp) return FALSE; @@ -45,6 +55,7 @@ tf_json_prepare(LogTemplateFunction *self, LogTemplate *parent, return TRUE; } +#if HAVE_JSON_C static gboolean tf_json_foreach (const gchar *name, const gchar *value, gpointer user_data) { @@ -57,16 +68,57 @@ tf_json_foreach (const gchar *name, const gchar *value, gpointer user_data) return FALSE; } -static struct json_object * -tf_json_format_message(ValuePairs *vp, LogMessage *msg) +static void +tf_json_append(GString *result, ValuePairs *vp, LogMessage *msg) +{ + struct json_object *json; + + json = json_object_new_object(); + + value_pairs_foreach(vp, tf_json_foreach, msg, 0, json); + + g_string_append(result, json_object_to_json_string (json)); + json_object_put(json); +} +#endif + +#if HAVE_JSON_GLIB +static gboolean +tf_json_foreach (const gchar *name, const gchar *value, gpointer user_data) { - struct json_object *root; + JsonBuilder *builder = (JsonBuilder *)user_data; + + json_builder_set_member_name(builder, name); + json_builder_add_string_value(builder, value); + + return FALSE; +} + +static void +tf_json_append(GString *result, ValuePairs *vp, LogMessage *msg) +{ + JsonBuilder *builder; + JsonGenerator *gen; + gchar *str; + + builder = json_builder_new(); + json_builder_begin_object(builder); + + value_pairs_foreach(vp, tf_json_foreach, msg, 0, builder); + + json_builder_end_object(builder); + + gen = json_generator_new(); + json_generator_set_root(gen, json_builder_get_root(builder)); + str = json_generator_to_data(gen, NULL); - root = json_object_new_object(); - value_pairs_foreach (vp, tf_json_foreach, msg, 0, root); + g_object_unref(gen); + g_object_unref(builder); - return root; + g_string_append(result, str); + g_free(str); } +#endif static void tf_json_call(LogTemplateFunction *self, gpointer state, GPtrArray *arg_bufs, @@ -77,14 +129,7 @@ tf_json_call(LogTemplateFunction *self, gpointer state, GPtrArray *arg_bufs, ValuePairs *vp = (ValuePairs *)state; for (i = 0; i < num_messages; i++) - { - LogMessage *msg = messages[i]; - struct json_object *json; - - json = tf_json_format_message(vp, msg); - g_string_append(result, json_object_to_json_string (json)); - json_object_put(json); - } + tf_json_append(result, vp, messages[i]); } static void -- 1.7.2.5

4 4

dropped udp packets and help with config
by Zeek Anow 24 May '11

24 May '11

Something really wrong with syslog-ng or my config. I'm dropping way too many packets. I will admit that my configuration is probably really a large part of the problem and would appreciate it if someone could take a look at it and offer some suggestions. There is another thread going about a similar problem on a similar platform. We recently upgraded to Solaris 10 from Solaris 9 and I don't recall us dropping that many packets before. And we also upgraded from a very older Sylog-ng version to 3.1.2. I am basing the dropped packets on the udp stats, not syslog-ng stats. Syslog-ng stats has NO dropped packets. UDP udpInDatagrams -4599313 udpInErrors - 0 udpOutDatagrams - 3421 udpOutErrors - 0 tcpInErrs - 0 udpNoPorts -2587612 udpInCksumErrs - 0 udpInOverflows -95806254 The above is a 3 hour sample and it is from our syslog server that does not get that much traffic. ____________________________ Here is the current version info: Solaris 10, syslog-ng 3.1.2 Installer-Version: 3.1.2 Revision: ssh+git://bazsi@git.balabit //var/scm/git/syslog-ng/syslog-ng-ose--mainline--3.1#master#8bf13c304b6ab5fc1a372b49d55c78370efe14ca Compile-Date: Oct 25 2010 23:56:18 Enable-Threads: off Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-Sun-STREAMS: on Enable-Sun-Door: on Enable-IPv6: on Enable-Spoof-Source: on Enable-TCP-Wrapper: off Enable-SSL: on Enable-SQL: off Enable-Linux-Caps: off Enable-Pcre: on _____________________________ Below is a very small sampling of our syslog-ng.conf. We are filtering on about 1400 devices most of which are Firewalls and routers. The IPs in the following sample have been made up. One of my questions is "Does the number of devices we are filtering on make a difference? (1400)" We have several sites and use just one version of the syslogng.conf file. It is a lot easier to maintain one copy:)) Also notice the format: ("^10\.123\.10\.133$") for the filters. All 1400 are in that format. I was hoping this would help a little but don't really know for sure:)) The source statementbelow "...external_Future_tcp" has not yet been implemented. Since we are dropping so many packets, I was going to try configuring the devices to log TCP instead of UDP. @version: 3.0 # Created: 01 March 2011 #----------------[ GLOBAL OPTIONS ]------------------------- options { create_dirs(yes); use_dns(no); time_reopen(10); time_reap(360); keep_timestamp(yes); }; #---------------------[ SOURCES ]--------------------------- source s_local { sun-stream("/dev/log" door("/etc/.syslog_door")); internal(); }; source s_external { udp(); }; source s_external_tcp { tcp(max-connections(50) port(514)); }; source s_external_Future_tcp { tcp(max-connections(1400) port(1470)); }; #---------------------[ DESTINATION ]--------------------------- destination d_local { file("/var/adm/messages" perm(0655) dir_perm(0655)); }; destination d_network_file { file("/logs/$YEAR/$MONTH/$DAY/network.log" perm(0655) dir_perm(0655)); }; destination d_bacsit { udp("10.11.13.114" port(2514) spoof-source(yes)); }; destination d_network_syslogd { udp("10.11.13.116" port(1514) spoof-source(yes)); }; destination d_firewall_file { file("/logs/$YEAR/$MONTH/$DAY/firewall/$HOST.log" perm(0655) dir_perm(0655)); }; destination d_mrv_file { file("/logs/$YEAR/$MONTH/$DAY/mrv.log" perm(0655) dir_perm(0655)); }; destination d_mail_file { file("/logs/$YEAR/$MONTH/$DAY/mail/$HOST.log" perm(0655) dir_perm(0655)); }; destination d_f567_file { file("/logs/$YEAR/$MONTH/$DAY/f5s/$HOST.log" perm(0655) dir_perm(0655)); }; #---------------------[ FILTERS ]--------------------------- filter f_f567 { host("^10\.123\.10\.133$") or # Host B host("^10\.100\.10\.200$") or # Host A host("^10\.115\.10\.246$") or # Host C host("^10\.121\.10\.102$") or # Host D host("^10\.117\.10\.99$"); # Host F }; filter f_mrv { host("^10\.68\.69\.100$") or # host("^10\.100\166\.10$") or # }; . . . and so on #---------------------[ LOGS ]--------------------------- log { source(s_local); destination(d_local); }; log { source(s_external); filter(f_f567); destination(d_f5_file); }; log { source(s_external); source(s_external_tcp); filter(f_firewall); destination(d_bacsit); }; log { source(s_external); filter(f_network); destination(d_bacsit); }; log { source(s_external); source(s_external_tcp); filter(f_firewall); destination(d_combo_file); }; log { source(s_external); filter(f_mail); destination(d_mail_file); }; ....and so on I'm grateful for all help and suggestions. Thanks!!

5 5

[ANN] syslog-ng module collection
by Gergely Nagy 23 May '11

23 May '11

Hi! A while ago, there was talk about creating a repository for third party and not-yet-merged syslog-ng modules, and today, I'm happy to say that I finally found a little time to make it happen! Before going much further, I'd like to tell where it is available from: http://git.madhouse-project.org/syslog-ng/module-collection/ And git://git.madhouse-project.org/syslog-ng/module-collection.git, respectively. A mirror on git.balabit.hu will be set up at a later point. I plan to make Debian packages available too, once syslog-ng 3.3 becomes available in Debian (experimental, or otherwise). The collection at the moment consists of my SMTP destination driver, ported to the latest syslog-ng 3.3 API (and should work with 3.4 too, although I didn't have time to test with that yet). With that out, let me explain why this 'collection' was created: * It is easier for users to use. No need to compile the whole of syslog-ng, perhaps with patches applied, merged and whatsoever. Just download the module collection sources, ./configure && make && make install, and you're done! Though, one still needs syslog-ng 3.3 and associated libraries (ivykis [installed by syslog-ng 3.3 by default], glib and eventlog), along with whatever the module collection needs (libesmtp at the moment). * It is easier for me. Because I don't have to rebase my sources to the latest syslog-ng sources, I don't have to care about stuff outside of the module itself. I can run any compatible version of syslog-ng, and the collection will just work. This makes it a lot easier for me to develop. Not to mention that this way, I can compile the module for syslog-ng 3.3 and 3.4 from the same source! * We can collect other modules easier. In case other third-party modules surface, it's easy to wire them into the collection, and voila, we'll have a nice distribution of unofficial modules! -- |8]

1 0

Re: [syslog-ng] Help: Meeting problems when writing to Oracle using syslog-ng 3.2.4
by Henry Xu 22 May '11

22 May '11

Hi, Anyone can help resolve this? I changed oracle database version from 10g to 11g.There's no such error message appeared in the message log file,but no data was write into the syslog table(the table can be created). Is there something wrong with my configuration? Thanks very much. Best, Henry From: Henry Xu [mailto:bxu@italkbb.com] Sent: Saturday, May 14, 2011 10:45 PM To: 'syslog-ng(a)lists.balabit.hu' Subject: Help: Meeting problems when writing to Oracle using syslog-ng 3.2.4 Hi All, I am trying to use syslog-ng writing to Oracle directly. However,I encountered the following errors: Error running SQL query; type='oracle', host='208.77.1.99', port='', user='service0', database='XE', error='3113: OR A-03113: end-of-file on communication channel\x0a', query='CREATE TABLE syslog (datetime varchar2(16), hostname varchar2(32), app varchar2(32), pid varchar2( 8), message varchar2(4000))' But when I checked the Oracle database,the table syslog already created.I don't know why the syslog-ng gave such error message and why it didn't write to Oracle using INSERT SQL. Here are my syslog-ng version info: syslog-ng 3.2.4 Installer-Version: 3.2.4 Revision: ssh+git://bazsi@git.balabit//var/scm/git/syslog-ng/syslog-ng-ose--mainline-- 3.2#master#ef7b91e4a1b1f9628c66138b4ae83de7e4c697c6 Compile-Date: May 13 2011 15:42:29 Enable-Threads: on Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-Sun-STREAMS: off Enable-IPv6: on Enable-Spoof-Source: on Enable-TCP-Wrapper: on Enable-SSL: on Enable-SQL: on Enable-Linux-Caps: off Enable-Pcre: on Enable-Pacct: off Here is my syslog-ng.conf content: ############################################################################ # # Default syslog-ng.conf file which collects all local logs into a # single file called /var/log/messages. # @version: 3.2 @include "scl.conf" options { stats_freq(3600); keep_hostname(yes); flush_lines(16); log_fifo_size(16384); }; source s_local { system(); internal(); }; source s_network { udp(); }; destination d_local { file("/var/log/messages"); }; destination d_net { file("/var/log/syslog-ng_network"); }; destination d_oracle_syslog { sql(type(oracle) host("208.77.1.99") username("service0") password("acstest") database("XE") table("syslog") columns("datetime varchar2(16)", "hostname varchar2(32)", "app varchar2(32)", "pid varchar2(8)", "message varchar2(4000)") values("$R_DATE", "$HOST", "$PROGRAM", "$PID", "$MSG") indexes("datetime", "hostname", "app", "pid", "message") ); }; log { source(s_network); destination(d_net); }; log { source(s_local); # uncomment this line to open port 514 to receive messages #source(s_network); destination(d_local); }; #log { # source(s_network); # destination(d_oracle_syslog); #}; source s_src { unix-stream("/dev/log"); file("/proc/kmsg"); internal(); }; log { source(s_src); destination(d_oracle_syslog); }; ============================================================================ =============================== Anyone can give some suggestions? Really appreciate your help! Best, Henry

2 1

value-pairs and sdata
by Balazs Scheidler 22 May '11

22 May '11

hi, we had a conversation with Algernon a couple of weeks ago how mongodb should handle value collisions which arise when both the SDATA and the .SDATA.<id>.<param> values are to be added to the document. As a reminder, mongodb strips the initial dot, and subsequent dots are used to break down values as subdocuments. This means that at the top level both SDATA as value and SDATA as a document are present. I found out the following solution, which is a change in how mongodb currently works. I'd like to do this before releasing 3.3beta. Feedback is welcome. * the SDATA macro is not included in rfc5424, selected-macros, but can be explicitly specified by key(), pair() and it's still included in all-macros * a new "sdata" scope is introduced which expands to the .SDATA.<id>.<param> values, but the SDATA is not included either. * nv-pairs is split: nv-pairs contains values which have no leading dot, dot-nv-pairs only contains the ones that have leading dots. all-nv-pairs contain both. This way, SDATA macro is only included if someone really wants it, thus collision is much less likely. Also, I'd like to replace the initial dot in mongodb to initial underscore, this way eliminating the possibility of collisions completely. I've a half baked patch, but nothing complete yet. However I'd really like to push the beta version out ASAP, and this one is keeping it back. Thanks for any comments. -- Bazsi

2 7

[git] syslog-ng git tree is rebased
by Balazs Scheidler 22 May '11

22 May '11

Hi, Just thought I'd send out a notification, that because of the last couple of mongodb/libmongo-client changes, I've removed a couple of commits, that I've already pushed out, and thus I've rebased the master branch. If you have a local git clone, you can always get back to track by rebasing your tree against the new master, or if you have no local patches, simply: $ git fetch $ git reset --hard origin/master Sorry for the trouble. -- Bazsi

1 0

Logging only if phrase found
by Sebastian Deißner 20 May '11

20 May '11

Hi @all, i am using syslog-ng for logging into mysql. **************************** options { chain_hostnames(off); sync(0); stats(43200); }; source src { unix-stream("/dev/log"); internal(); pipe("/proc/kmsg"); }; destination d_mysql { program("/usr/bin/mysql --user=syslogfeeder --password=SuperPassWord syslog" template("INSERT INTO logs (host, facility, priority, level, tag, datetime, program, msg) VALUES ( '$HOST', '$FACILITY', '$PRIORITY', '$LEVEL', '$TAG', '$YEAR-$MONTH-$DAY $HOUR:$MIN:$SEC','$PROGRAM', '$MSG' );\n") template-escape(yes)); }; destination messages { file("/var/log/messages"); }; log { source(src);destination(messages); destination(d_mysql); }; **************************** I want to have only the entries in my log database, that contain a specific phrase, for example all the entries with "...connect..." Is this possible with syslog-ng? tnx Sebastian

3 4

problem with windows agent and windows 2008
by Giovanni Mancuso 12 May '11

12 May '11

Hi, I have a problem with syslog-ng windows agent. I installed the client in a Windows 2003 machine and it works very well. I try to install the same client in a windows 2008 SP2 machine, but it send ONLY the startup/shutdown of the agent. It doesn't send other log (for ezample the windows event logs). I try to run the agent in debug mode (with /E) but i don't find useful information. Any ideas? Thank

2 1

syslog-ng Insider - May 2011
by Peter Czanik 12 May '11

12 May '11

Dear syslog-ng users, This is the 3rd issue of the syslog-ng Insider, a monthly newsletter that brings you syslog-ng related news. Your feedback and news tips about the next issue is welcome at documentation(a)balabit.com <mailto:documentation@balabit.com> FEATURED NEWS syslog-ng OSE 3.3 development ----------------------------- The previous month did not see a new syslog-ng 3.3 alpha or beta release, but development still went on. There were many important bugfixes and performance improvements. Also, most of the patches from the community were merged, including value-pairs(), mongo-db updates and JSON support. There is now a feature freeze in effect, so new features go now into the freshly opened syslog-ng 3.4 repository. The current version can be checked out out from git, which has regular updates at http://git.balabit.hu/?p=bazsi/syslog-ng-3.3.git;a=summary The mailing list archive with some more patches: http://lists.balabit.hu/pipermail/syslog-ng/ Blog about the feature freeze: http://bazsi.blogs.balabit.com/2011/05/syslog-ng-3-3-feature-freeze-3-4-bra… Blog series on web GUIs for syslog-ng ------------------------------------- Web based GUIs for syslog-ng is a hot topic recently, so a new blog series was started. This month I'll cover a new challenger: ELSA, which has patterndb at its heart.If you have any suggestions what else to cover, please let us know! ELSA: https://czanik.blogs.balabit.com/2011/05/elsa-web-interface-for-syslog-ng-a… Support the syslog-ng project with your testimonial! ---------------------------------------------------- We send syslog-ng T-Shirts for the best use-cases and testimonials. We are especially interested in how you use syslog-ng: your environment, performance metrics, the UNIX platforms that you use and anything that you deem interesting about your setup. This is your possibility to influence the syslog-ng development process and to help the syslog-ng project. http://www.balabit.com/testimonials/send OTHER SHORT NEWS * Maintenance policy update for syslog-ng OSE: http://bazsi.blogs.balabit.com/2011/05/maintenance-policy-update-for-syslog… * Algernon's syslog-ng status update: http://algernon.blogs.balabit.com/2011/05/veni-vidi-vici/ NEW RELEASES * new 3.2 OSE bugfix release: http://lists.balabit.hu/pipermail/syslog-ng/2011-May/016576.html RECENT WHITEPAPERS * How to establish a distributed syslog architecture in heterogeneous environment with syslog-ng Premium Edition? http://www.balabit.com/support/documentation/syslog-ng-v4.0-whitepaper-dist… * Get familiar with the advantages of using syslog-ng Premium Edition to collect system log (syslog) messages into an SQL database, such as Oracle or MySQL. http://www.balabit.com/support/documentation/syslog-ng-v4.0-whitepaper-sysl… ARCHIVE http://insider.blogs.balabit.com/ -- Peter Czanik (CzP) <czanik(a)balabit.hu> BalaBit IT Security / syslog-ng upstream http://czanik.blogs.balabit.com/

1 0