- syslog-ng - lists.balabit.hu

stunnel through elb.. need packets sent semi-frequently
by Matt Wise 08 Apr '11

08 Apr '11

I'm using stunnel to pass some data through an Amazon ELB. Unfortunately, there are times when the session times out due to packets not passing for 60 seconds (an amazon ELB timeout). I'm wondering whether theres some way I can configure stunnel itself to pass some whitespaces or something back and forth every few seconds to make sure the session stays open and live? (and yes, I'm posing the same question to the app developer who's ap we're passing through.. but I'm looking at both avenues.. )

2 2

diskqueue framework patch
by Nagy Attila 08 Apr '11

08 Apr '11

Hi all, due to the need for disk queue usage in PE 4.1 we had to extend the syslog-ng sources, in order to be able write plugins that implement this feature. This patch contains these modifications. Basically it extends the 'class' hierarchy (and 'virtual' functions, see the new LogDestDriver) so that plugins can define their own queue mechanisms (as some do this in PE modules) and these special queues can be assigned to destinations. -Attila NAGY (syslog-ng-dev)

3 5

[Bug 118] New: syslog-ng -s errors with catchall flag (1.6 to 3.1)
by bugzilla＠bugzilla.balabit.com 08 Apr '11

08 Apr '11

https://bugzilla.balabit.com/show_bug.cgi?id=118 Summary: syslog-ng -s errors with catchall flag (1.6 to 3.1) Product: syslog-ng Version: unspecified Platform: Other OS/Version: Solaris Status: NEW Severity: normal Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: michael.mishou(a)csirc.irs.gov Type of the Report: bug Estimated Hours: 0.0 This is a bug report that is pretty much copied from the mailing list, verified by Gergely Nagy. I used this snapshot to find the bug: http://git.balabit.hu/?p=bazsi/syslog-ng-3.1.git;a=snapshot;h=8bf13c304b6ab… But Gergely reports that the problem exists in multiple 3.x version (and also likely 2.x versions as well). I was getting the error Unknown log statement flag; flag='catchall' when doing a 'syslog-ng -s -f FILE' on a 2.x configuration file using version 3.1.2 of syslog-ng. All the documentation (2.x and 3.x) refers to log path flags: catchall, fallback, final, flow-control. But in center.c the actual strcmp functions are looking for: catch-all, fallback, final, flow_control OR flow-control gint log_connection_lookup_flag(const gchar *flag) { if (strcmp(flag, "catch-all") == 0) return LC_CATCHALL; else if (strcmp(flag, "fallback") == 0) return LC_FALLBACK; else if (strcmp(flag, "final") == 0) return LC_FINAL; else if (strcmp(flag, "flow_control") == 0 || strcmp(flag, "flow-control") == 0) return LC_FLOW_CONTROL; msg_error("Unknown log statement flag", evt_tag_str("flag", flag), NULL); return 0; } I'm guessing it should probably be something like this instead: gint log_connection_lookup_flag(const gchar *flag) { if (strcmp(flag, "catch-all") == 0 || strcmp(flag, "catchall") == 0) return LC_CATCHALL; else if (strcmp(flag, "fallback") == 0) return LC_FALLBACK; else if (strcmp(flag, "final") == 0) return LC_FINAL; else if (strcmp(flag, "flow_control") == 0 || strcmp(flag, "flow-control") == 0) return LC_FLOW_CONTROL; msg_error("Unknown log statement flag", evt_tag_str("flag", flag), NULL); return 0; } Gergely's notes/recommendations for fix: It's the same in 3.2 and 3.3 too - good catch! Whether this is a documentation or code bug, if the docs say it's 'catchall' ever since 2.0, then the code should support that too, along with 'catch-all'. I'd say this is a bug in both: the documentation should be updated to say 'catch-all', which has been working in past versions, and the code should be updated to support what the documentation has been advertising in the past years. -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 1

Convert UTC to local time zone using syslog-ng
by Pramod Pillai 07 Apr '11

07 Apr '11

Hi I receive syslog from remote note with timezone set as UTC. I need to convert these timestamp to local time zone Can some one suggest how can i achive it. Regards Pramod

4 5

[RFC]: afsql improvement plans
by Gergely Nagy 03 Apr '11

03 Apr '11

Hi! I'm about to blow up^W^Wrefactor the afsql destination to increase both performance and functionality. To do this properly, and to satisfy most needs, I'd like to ask the greater community (especially those of you who have more experience with logging to SQL than I do!) about the plans laid out on the paragraphs below. Those of you who do log into SQL one way or the other, but aren't interested in technical mumbo-jumbo, please skip to the end of this e-mail, I'd have a few questions, for which all responses would be most appreciated! Where, when, how?? ================== There are multiple goals to accomplish, which I believe are not exclusive: we want to keep the sql() statement in the configuration file, and maintain backwards compatibility aswell. On the other hand, we want to improve performance too, and that is best accomplished by using the native database libraries. To this end, I propose we introduce new database types: "mysql-native", "postgresql-native", "sqlite-native", and so on and so forth. The first step will be to rearrange the current afsql driver, so that it has a generic configuration interface, but can have multiple backends (dbi - the only existing backend right now; and one for each native db client). I'd change the AFSqlDestDriver structure to only contain the configurable data and nothing else. The backends would subclass this type (much like how AFSqlDestDriver subclasses LogDriver) and add their own. The reason I want to do this, making the AFSqlDestDriver a config-only interface instead of keeping larger chunks of code there (for example, the threading code) is because I do not want to limit my possibilities: I can imagine an SQL driver utilizing multiple writer threads easily. However, neither model works for all databases (concurrent writes to SQLite are usually not a good idea, for example). And that's about the generic interface! Lets dive into the more interesting parts: the native drivers. Since MySQL is the most used server out there, that will be the first I tackle. My goal with that is to implement a driver that is not only faster than our current destinations, but one that is also faster than the perl scripts people have been using until now, when they wanted top performance. All that while maintaining reasonable flexibility. An ambitious plan, but hopefully not impossible. I've been reading the MySQL documentation for the past few hours to explore my options, and found a few ways to increase our insert speed: mysql_list_tables() ------------------- Instead of doing a full select to verify that a table exists, we can just try to list the table. This is probably not a significant increase, but every bit counts. Prepared statements ------------------- We'll use prepared statements wherever possible - I expect this alone will result in a measurable performance increase, since the server will not have to parse each and every insert - only once, we'll bind variables thereafter. LOCK TABLES / transactions -------------------------- Currently, syslog-ng can do bulk-inserts, wrapped in a transaction. But that only works if the MySQL table we insert to is on a storage engine that supports transactions (InnoDB). However, MyISAM has been the default for quite a while, and that does not support transactions to the best of my knowledge. In this case, insert performance can be increased by locking the tables for a few hundred inserts at a time, and unlocking after: poor man's transactions, to some extent. This, however, is more of a workaround than a real improvement. Insertion plans =============== There's a couple of alternative ways to improve insert speed by changing how we actually do them. All have their pros and cons, which I will try to enumerate below. LOAD DATA INFILE ---------------- The fastest way by far is LOAD DATA INFILE, which is what people use in the various perl scripts, as far as I remember. The downside is that - as far as I saw so far - this does need a temporary file, which has its downsides, like writing to disk (unless put on tmpfs or similar). An option which I haven't tried would be to use a pipe(), so that the data never hits the disk. But nevertheless, the mysql-native db type will have an option to use LOAD DATA INFILE. * Pros: + The fastest way to insert * Cons: - Needs a temporary file, with all its drawbacks, some of which can be worked around. - Serialisation is costlier than simple INSERTs. - No fine-grained error checking: if something goes wrong, the whole LOAD will have to be dropped (unless we add code that splits bulks up into smaller chunks, and retries - but that's something I want to avoid if possible) For LOAD DATA to work, we will have to format the data to insert into a suitable format, and if there's an error in the serialisation somewhere, we'll have a hard time to figure out where the problem is, and syslog-ng will just give up. Also, beating our data into a suitable format isn't exactly cheap, either (think escaping: processing large amounts of string data, possibly changing strings - there's both processing and memory ramifications involved). The need for a file is also a (minor) issue. I suppose named pipes would work, that way data never hits the disk between syslog-ng and mysql. However, that still involves copying data around. We'll see if these affect LOAD DATA's performance badly enough (they probably don't). INSERT INTO ... VALUES (...), (...), ... ---------------------------------------- This is considerably slower than LOAD DATA, but still faster than using INSERTs (even if we use transactions / locking for those). The downside is that - like with LOAD DATA - serialisation is still costlier than INSERTs. * Pros: + Reasonably fast + Does not require neither a temporary file, nor gross hacks * Cons: - Slower than LOAD DATA - Serialisation can be costly. - No fine-grained error checking Serialisation is a little less costly than in LOAD DATA's case, but this can be worked around a little with prepared statements (with a fallback to parsed ones, in case we have to insert less data than originally planned). The same issues with error handling persist, though. INSERT ... DELAYED ------------------ The DELAYED strategy works best if we have multiple writer threads, as in this case, INSERT becomes a fire & forget call, and we can return to our business far sooner. This does not work all that well with bulk inserts, though. * Pros: + Reasonably fast with multiple writers + Simple & fast serialisation * Cons: - No error checking at all (except for connection and syntax errors) - Probably still slower than LOAD DATA The advantage of this is that we can get the data from syslog-ng to the mysql server faster than any of the other methods, and then the work is off-loaded to the server. This has the potential of being faster than any of the above, but with the cost of hardly any possibilities to recover from an error, as we get no feedback after inserting: did it succeed? did it not? We'll have no idea. The Plan ======== The plan is to implement all the strategies above, and measure them against each other and external scripts. This will take a little time, because I want to get the architecture right, and figure out whether we want a single or multiple writer threads (per-destination, of course), or if I want to make that a configurable option aswell. Once done, and people are happy with it, I'll move on to the next database (PostgreSQL, to scratch my own itch too). The Questions ============= What I'd like to know most, is, that would YOU expect from a native mysql destination? Apart from speed, what would you want from it, that the current driver does not support, but which is possible via external scripts? If you happened to read through my wall of text above, did I miss anything important? Cheers, -- |8]

4 10

More on httpry_logger
by Martin Holste 03 Apr '11

03 Apr '11

I Just thought I'd mention to the list that I posted a write-up on analyzing the effects of the lizamoon SQL injection campaign by looking at logs from httpry_logger through ELSA (both available at http://code.google.com/p/enterprise-log-search-and-archive/) on my blog at ossectools.blogspot.com. httpry_logger can be used stand-alone with whatever log system you're using now. In addition to syslog, it outputs to a number of other formats including basic DBI handle, MongoDB, and flat file. --Martin

1 0

Proper configuration for Syslog-ng PE v4.0 disk-based buffering feature is required...
by Prashant Badhe 02 Apr '11

02 Apr '11

Hi, I am evaluating Syslog-ng PE v4.0 disk-based buffering feature on Linux, but it doesn’t seems to be working. Below is the configuration I added to my sysylog-ng.conf file: destination d_syslog { tcp("myserver" port(1468) log_disk_fifo_size(102400) ); file("/var/log/messages"); }; When the connection to the server is up then I receive the forwarded syslogs properly. When the network connection goes down, syslog-ng stores these messages at location /opt/syslog-ng/var, (.qf files). But when the network connection reestablishes, these stored messages never sent to the server. Can anybody please guide me if there is some missing configuration or wrong configuration I have done to make this feature work? Thanks in advance, Prashant Badhe

2 1

Blog post on ELSA
by Martin Holste 01 Apr '11

01 Apr '11

I just put up an entry on my blog (http://ossectools.blogspot.com) describing the Enterprise Log Search and Archive Project (http://code.google.com/p/enterprise-log-search-and-archive) I've been working on which uses Syslog-NG >= 3.1 and pattern-db at its core. There are a lot of other open-source log collection frameworks out there that are easier to install, such as Logzilla (php-syslog-ng), but if you're trying to log > 1k messages/sec (common in large orgs) and need something GPL licensed, installing ELSA will probably be worth your while. We're using it to index 15k messages/sec with basic hardware. It's currently storing tens of billions logs, and full-text, ad-hoc queries complete in about 1/2 to 2 seconds, including group-by queries on arbitrary fields for reporting. I put a few screenshots and a feature list in the post. The documentation is pretty basic right now, but I'm happy to assist if you run into issues. ELSA is also open to plugin creation, so if you find ELSA useful and create plugins, please let me know and I can add them to the project. Also, patterns for the pattern-db are more than welcome! I've included patterns for Cisco FWSM connections and denies, Snort logs, Windows logs from Eventlog-to-Syslog as well as Snare, and URL's from my httpry wrapper, which is available on the project site as well as in the tarball/source code. Comments and feedback are welcome! Thanks, Martin

6 8

how to push sulogs/utmpx/wtmpx to central logserver
by Tony MacDoodle 31 Mar '11

31 Mar '11

Hello All, I would like to push only the following log files to a central syslog-ng server. All clients are Solaris 10 and the syslog-ng server is also Solaris 10. IS it possible to push only this information? wtmpx utmpx sulog Thanks

4 3

Make Errors on Solaris 10
by Tony MacDoodle 31 Mar '11

31 Mar '11

Hello, I am trying to compile syslog-ng 3.2.2 for Solaris 10 Sparc and getting the following errors when I run a make: ./configure --enable-tcp-wrapper syslog-ng Open Source Edition 3.2.2 configured Compiler options: compiler : gcc -std=gnu99 compiler options : -g -O2 -Wall -D_REENTRANT -D_PTHREADS -I/usr/local/include/glib-2.0 -I/usr/local/lib/glib-2.0/include -I/usr/local/include/eventlog -D_GNU_SOURCE -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 linker flags : -lpthread prefix : /usr/local linking mode : dynamic Features: Sun STREAMS support (module): yes Debug symbols : no GCC profiling : no Memtrace : no IPV6 support : yes spoof-source support : no tcp-wrapper support : yes SSL support (module) : no SQL support (module) : no Linux capability support : no PCRE support : no Env wrapper support : no PACCT module (EXPERIMENTAL) : no root:~/syslog/syslog-ng-3.2.2#* make* make all-recursive Making all in lib make: Fatal error in reader: Makefile, line 637: Unexpected end of line seen Current working directory /root/syslog/syslog-ng-3.2.2/lib *** Error code 1 The following command caused the error: fail= failcom='exit 1'; \ for f in x $MAKEFLAGS; do \ case $f in \ *=* | --[!k]*);; \ *k*) failcom='fail=yes';; \ esac; \ done; \ dot_seen=no; \ target=`echo all-recursive | sed s/-recursive//`; \ list='lib modules syslog-ng scripts tests doc contrib scl debian tgz2build'; for subdir in $list; do \ echo "Making $target in $subdir"; \ if test "$subdir" = "."; then \ dot_seen=yes; \ local_target="$target-am"; \ else \ local_target="$target"; \ fi; \ (CDPATH="${ZSH_VERSION+.}:" && cd $subdir && make $local_target) \ || eval $failcom; \ done; \ if test "$dot_seen" = "no"; then \ make "$target-am" || exit 1; \ fi; test -z "$fail" make: Fatal error: Command failed for target `all-recursive' Current working directory /root/syslog/syslog-ng-3.2.2 *** Error code 1 make: Fatal error: Command failed for target `all' Thanks

2 1