April 2011 - syslog-ng - lists.balabit.hu

[PATCH] afmongodb: Make the collection name a simple string.
by Gergely Nagy 30 Apr '11

30 Apr '11

Instead of allowing templates in the collection name, make it a simple string instead, to allow for easily implementing bulk inserts in the future. Signed-off-by: Gergely Nagy <algernon(a)balabit.hu> --- modules/afmongodb/afmongodb.c | 30 ++++++++++-------------------- 1 files changed, 10 insertions(+), 20 deletions(-) diff --git a/modules/afmongodb/afmongodb.c b/modules/afmongodb/afmongodb.c index 284066a..929d35b 100644 --- a/modules/afmongodb/afmongodb.c +++ b/modules/afmongodb/afmongodb.c @@ -47,7 +47,7 @@ typedef struct /* Shared between main/writer; only read by the writer, never written */ gchar *db; - LogTemplate *coll; + gchar *coll; gchar *host; gint port; @@ -84,11 +84,9 @@ typedef struct mongo_connection *conn; gint32 seq_num; - GString *current_namespace; - gint ns_prefix_len; + gchar *ns; GString *current_value; - bson *bson_sel, *bson_upd, *bson_set; } MongoDBDestDriver; @@ -145,8 +143,8 @@ afmongodb_dd_set_collection(LogDriver *d, const gchar *collection) { MongoDBDestDriver *self = (MongoDBDestDriver *)d; - log_template_unref(self->coll); - self->coll = log_template_new(log_pipe_get_config(&d->super), NULL, collection); + g_free(self->coll); + self->coll = g_strdup(collection); } void @@ -177,7 +175,7 @@ afmongodb_dd_format_stats_instance(MongoDBDestDriver *self) static gchar persist_name[1024]; g_snprintf(persist_name, sizeof(persist_name), - "mongodb,%s,%u,%s,%s", self->host, self->port, self->db, self->coll->template); + "mongodb,%s,%u,%s,%s", self->host, self->port, self->db, self->coll); return persist_name; } @@ -187,7 +185,7 @@ afmongodb_dd_format_persist_name(MongoDBDestDriver *self) static gchar persist_name[1024]; g_snprintf(persist_name, sizeof(persist_name), - "afmongodb(%s,%u,%s,%s)", self->host, self->port, self->db, self->coll->template); + "afmongodb(%s,%u,%s,%s)", self->host, self->port, self->db, self->coll); return persist_name; } @@ -274,10 +272,6 @@ afmongodb_worker_insert (MongoDBDestDriver *self) g_free (oid); bson_finish (self->bson_sel); - g_string_truncate(self->current_namespace, self->ns_prefix_len); - log_template_append_format(self->coll, msg, NULL, LTZ_LOCAL, - self->seq_num, NULL, self->current_namespace); - for (i = 0; i < self->num_fields; i++) { log_template_format(self->fields[i].value, msg, NULL, LTZ_SEND, @@ -292,7 +286,7 @@ afmongodb_worker_insert (MongoDBDestDriver *self) bson_append_document (self->bson_upd, "$set", self->bson_set); bson_finish (self->bson_upd); - p = mongo_wire_cmd_update (1, self->current_namespace->str, 1, + p = mongo_wire_cmd_update (1, self->ns, 1, self->bson_sel, self->bson_upd); if (!mongo_packet_send (self->conn, p)) @@ -335,12 +329,8 @@ afmongodb_worker_thread (gpointer arg) NULL); success = afmongodb_dd_connect(self, FALSE); - self->current_namespace = g_string_sized_new(64); - self->ns_prefix_len = strlen (self->db) + 1; - self->current_namespace = - g_string_append_c (g_string_assign (self->current_namespace, - self->db), '.'); + self->ns = g_string_concat (self->db, ".", self->coll, NULL); self->current_value = g_string_sized_new(256); @@ -383,7 +373,7 @@ afmongodb_worker_thread (gpointer arg) afmongodb_dd_disconnect(self); - g_string_free (self->current_namespace, TRUE); + g_free (self->ns); g_string_free (self->current_value, TRUE); bson_free (self->bson_sel); @@ -516,7 +506,7 @@ afmongodb_dd_free(LogPipe *d) g_free(self->fields); g_free(self->db); - log_template_unref(self->coll); + g_free(self->coll); g_free(self->user); g_free(self->password); g_free(self->host); -- 1.7.2.5

2 2

Formatting syslogs using syslog-ng
by Pramod Pillai 30 Apr '11

30 Apr '11

Hi I need to format the below syslog message. Apr 13 07:37:28 host-mgr dhcpd: [ID 702911 local7.error] DHCPDISCOVER to Apr 13 03:37:15 host-mgr 23 3 DHCPDISCOVER My destination config looks like this. destination sol_dest { file("/var/log/sol_syslog" perm(0644) template("$S_DATE $HOST $FACILITY_NUM $LEVEL_NUM $MSGONLY\n") template_escape(no) ); }; With the template mentioned above I am getting output as shown below. I need to remove " [ID 702911 local7.error] " . How to do it. Apr 13 03:37:15 mgr-t 23 3 [ID 702911 local7.error] DHCPDISCOVER I am running this on Solaris machine with syslong-ng 3.1.2 Regards Pramod

2 1

Not all messages getting to program destination
by Bill Anderson 30 Apr '11

30 Apr '11

Setup: v3.1 program destination: python script secondary file destination in the _same_ log block (so it should be getting everything the program destination gets), no final flag. stats turned on, 60s interval. Problem: In the script I am tallying lines parsed per minute. The lines parsed per-minute is not matching a grep -c on the file destination. stats indicate no dropped or queued messages to either destination. If I cat the file to the same destination script on the command line, I get the same counts as I do grepping the file directly. I would expect that the script would return the same results from the shell as when running as a destination in syslog. Where else can I look/what else can I turn on, to see why the discrepancy exists? Cheers, Bill -- Bill Anderson, RHCE Linux Systems Engineer bill.anderson(a)bodybuilding.com

4 7

Question about syntax for source
by Phusion 30 Apr '11

30 Apr '11

In syslog-ng 2.x versions under OpenBSD I used to use the following line for a source. source local { internal(); pipe("/dev/klog" log_prefix("kernel: ")); unix-dgram("/dev/log"); }; When trying to convert this for syslog-ng 3.x under OpenBSD. I have an error. When I use syslog-ng -s -f ../syslog-ng.conf it works, but when I run syslog-ng -p /var/run/syslog-ng.pid, I get the following error. # /usr/local/sbin/syslog-ng -p /var/run/syslog-ng.pid WARNING: you are using the pipe driver, underlying file is not a FIFO, it should be used by file(); filename='/dev/klog' Here is the current source line using syslog-ng 3.0 under OpenBSD 4.8. source local { internal(); pipe("/dev/klog" program_override("kernel: ")); unix-dgram("/dev/log"); }; Please advise. Phusion

6 12

log message difference between syslog and syslog-ng?
by Evan Baer 30 Apr '11

30 Apr '11

Hello, We switched one of our last remaining syslog servers to syslog-ng a few days ago, with no major surprises, except for one small difference in log message formatting between syslog and syslog-ng for messages from some of our network devices. The difference is in the added ':' in the 5th column, after the hostname: Apr 27 12:48:52 10.26.13.65 oob1-switch, System: Set fan speed to MED (75%) Apr 28 16:59:53 10.26.13.65 oob1-switch,: System: Set fan speed to LOW (50%) (the first line is from syslog, the second is from syslog-ng3-3.1.1, running on freebsd) For what it is worth, this formatting difference also occurs with other devices of the same manufacturer and model which have valid reverse ip entries in dns. We have a few support scripts which read the logs and are picky about the line formatting, so I'm keen to tweak the output lines in syslog-ng.conf if possible. Thoughts? Thanks in advance, -- Evan

2 1

syslog-ng and systemd
by Matthias Runge 30 Apr '11

30 Apr '11

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hello users, Fedora 15 (and later) uses systemd as init-replacement. Currently, I'm building a fedora package for syslog-ng version 3.2.x. It is based on a git-snapshot dated April 24th. Is there a way to signal syslog-ng to reload it's configuration other than HUPing its pid? The documentation of systemd[1] says: ExecReload= should be specified for all services supporting reload. It is highly recommended to add code here that synchronously reloads the configuration file here (i.e. /bin/kill -HUP $MAINPID is usually a poor choice, due to its asynchronous nature). Omit this option if your service does not support reloading. I would highly appreciate not to maintain a pidfile. Newer linux kernels support so called cgroups. Those can be used to group processes. Systemd creates a cgroup for each service and starts in our case syslog-ng in this cgroup. When no process runs in syslog-ng cgroup, the process exited and must be restarted. systemd takes care of this. Again, is there any other way to signal syslog-ng to reload its config? Thanks, Matthias [1] https://fedoraproject.org/wiki/Systemd_Packaging_Draft#Writing_unit_files -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org/ iQEcBAEBAgAGBQJNuSg1AAoJEOnz8qQwcaIWb6oIAKMjeWwi5C/qZFeWgSL0/X5Y lsMjjWQ/YvhdgjHxNLEHFpTsJT6nrFo1akLN9nL4I8j23vekEgtDrWKAeOk/A+S9 LVUmgcGYnXJC3Oshu2Xg8MLcBQIIrXVrboTeG5GHGBG3gWmRuzL0yH9PjSKbNcd0 PtHSxqEBwhHN9vOc59a5UvclElGgmSzqmYdS5Qm84fStii+j7GNwkqRQvoEGrPlw RrOSFsIPGStXnxk+SzRn8JCKrk7ueZM7l8QoJPIfghdumFqyjPhZRd2BQUlorr6z J3u3GEpXaIOyj+qyix+KTTofjiJh/TC4/SVnqmNTQgsqKLNy1hDCWHXz2X1v7TY= =M50d -----END PGP SIGNATURE-----

3 2

gmane posting problem
by Micah Anderson 30 Apr '11

30 Apr '11

My previous message that I sent to this list I tried to send three different times via GMANE, which is subscribed to this list. Usually I have no issue posting to lists through this method, but for some reason, my message was not getting posted. Typically what happens is you post to the list via gmane, the gmane autoresponder sends you an email to confirm that you are a legit person and not a spammer, and then you reply and then you are allowed to post to the list. I did all that, but my messages were not coming through. Perhaps the list is configured in a strange way where these messages are getting rejected? thanks for looking into this, micah --

2 1

Maintenance Policy update for syslog-ng Open Source Editions
by Balazs Scheidler 30 Apr '11

30 Apr '11

Dear syslog-ng users, As discussed on the syslog-ng mailing list, the current versioning policies regarding syslog-ng Open Source Edition is confusing, and with the proliferation of syslog-ng versions, their maintenance is an increasing burden on the syslog-ng project. Currently three major versions are supported (3.0, 3.1 and 3.2) and a fourth one (3.3) is in active development. A decision was made that the distinction between "feature" and "stable" releases for syslog-ng OSE releases will cease to exist: all releases will have the same status support-wise: * they will be supported for a year, or * until the next stable release is made * whichever is longer Also, the versioning of Open Source and Premium editions will become completely independent, and it is not possible to compare their functionality on the version number alone. The Premium Edition will always be based on a specific Open Source release, and provide additional functionality compared to the base version. OSE releases published after the PE release may provide additional functionality, not yet present in a PE release. The changes above cause some of our currently supported versions to be deprecated. In order to provide a time window for migration to a newer release, the following EOL dates were set: syslog-ng 3.0: 30th, June 2011. syslog-ng 3.1: 30th, June 2011. Everyone running these or earlier releases should upgrade to the latest 3.2.x release, which is currently at 3.2.2. For more information: * support periods for syslog-ng OSE versions: http://www.balabit.com/network-security/syslog-ng/opensource-logging-system… * to download the latest syslog-ng OSE sources: http://www.balabit.com/network-security/syslog-ng/opensource-logging-system… The syslog-ng project, on its own doesn't provide syslog-ng binaries, except for a limited number of Linux distributions. It is expected that users compile syslog-ng on their own, or use the binary provided by the OS supplier. This change doesn't affect you, if * you run a Premium Edition of syslog-ng, * you have a support contract in place, that says otherwise Happy logging, -- Bazsi

1 0

syslog-ng to elasticsearch ?
by Daniel Maher 28 Apr '11

28 Apr '11

Hello, I am curious to know if anybody has tried (or even better, succeeded) in sending logs from Syslog-NG direct to Elasticsearch ? I see that there is already a driver for MongoDB, which is along the same lines... My first idea was to send all the incoming logs to a named pipe, then have a small script read from the pipe and trigger calls to ES for each line (not unlike the many SQL howtos out there). I suspect that this would work, but if anybody has any ideas which are more elegant, I would love to hear about it. Thanks, and have a great day ! -- Daniel Maher « can't talk, too busy calculating computrons. »

3 4

FW: Solaris 10_x86 server.
by sramesh.kumar＠wipro.com 27 Apr '11

27 Apr '11

<sramesh.kumar(a)wipro.com> writes: > Thanks to your mail… But I am getting below error….in solaris10 os. > Please suggest. > bash-3.00# /usr/local/sbin/syslog-ng -d -f > /etc/syslog-ng/syslog-ng.conf > > Error opening file for reading; filename='/proc/kmsg', error='No such > file or directory (2)' > > Error initializing source driver; source='s_sys' > > \bash-3.00# ls -l /proc/kmsg > > /proc/kmsg: No such file or directory > > bash-3.00# That's because /proc/kmsg is not available on Solaris: it's a Linux thing. You can safely remove it from the configuration file. I'd strongly suggest reading the documentation available at http://www.balabit.com/sites/default/files/documents/syslog-ng-v2.0-guide-a… (Also, upgrading to a more recent version of syslog-ng would perhaps be a good idea, 2.0.5 is ancient) -- |8] ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html Please do not print this email unless it is absolutely necessary. The information contained in this electronic message and any attachments to this message are intended for the exclusive use of the addressee(s) and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately and destroy all copies of this message and any attachments. WARNING: Computer viruses can be transmitted via email. The recipient should check this email and any attachments for the presence of viruses. The company accepts no liability for any damage caused by any virus transmitted by this email. www.wipro.com

1 0