- syslog-ng - lists.balabit.hu

[PATCH (3.3)] doc: Fix EXTRA_DIST
by Gergely Nagy 21 Feb '12

21 Feb '12

There is a \ missing from the end of the second line of EXTRA_DIST, which results in a couple of files being left out of make dist, which in turn makes distcheck and building from dist tarballs fail. This patch adds the missing \. Signed-off-by: Gergely Nagy <algernon(a)balabit.hu> --- doc/Makefile.am | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/doc/Makefile.am b/doc/Makefile.am index 33b113c..6c6c1c2 100644 --- a/doc/Makefile.am +++ b/doc/Makefile.am @@ -1,7 +1,7 @@ man_MANS = man/syslog-ng.8 man/syslog-ng.conf.5 man/pdbtool.1 man/loggen.1 man/syslog-ng-ctl.1 EXTRA_DIST = man/syslog-ng.8.xml man/syslog-ng.conf.5.xml man/pdbtool.1.xml man/loggen.1.xml man/syslog-ng-ctl.1.xml \ - $(man_MANS) + $(man_MANS) \ security/bof-2002-09-27.txt \ security/dos-2000-11-22.txt \ xsd/patterndb-1.xsd \ -- 1.7.9

1 0

[Bug 152] New: Vers 3.3.3 pipe source does not parse host from input line
by bugzilla＠bugzilla.balabit.com 14 Feb '12

14 Feb '12

https://bugzilla.balabit.com/show_bug.cgi?id=152 Summary: Vers 3.3.3 pipe source does not parse host from input line Product: syslog-ng Version: 3.3.x Platform: PC OS/Version: Linux Status: NEW Severity: normal Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: erempel(a)uvic.ca Type of the Report: --- Estimated Hours: 0.0 Try as I might, I can not get a pipe source to maintain the hostname. Syslog-ng includes the hostname as the $MSGHDR (or the $PROGRAM). my config # ----------------------------------- source patterndb { pipe("/var/log/syslog.pipes/patterndb" log_fetch_limit(500) log_iw_size(100000) ); }; source int { internal(); }; template t_standardfile { template("$S_ISODATE $FULLHOST $FACILITY.$LEVEL $MSGHDR][$PROGRAM][$MESSAGE\n"); template_escape(no); }; destination d_var_patterndb { file("/var/log/patterndb.$R_YEAR$R_MONTH$R_DAY.000000" owner("root") group("syslogs") perm(0640) template(t_standardfile)); }; log { source(patterndb); destination(d_var_patterndb); }; # ----------------------------------- data I write tothe pipe # ----------------------------------- <12>2011-11-25T00:00:30-08:00 somehost.uvic.ca mmfs: Fri Nov 25 00:00:29.618 2011: Accepted and connected to 172.20.102.38 hermes0080 <c0n350> <12>2011-11-25T00:00:30-08:00 somehost.uvic.ca mmfs: Fri Nov 25 00:00:29.620 2011: Connecting to 172.20.107.23 nestor0167 <c0n200> <12>2011-11-25T00:00:30-08:00 somehost.uvic.ca mmfs: Fri Nov 25 00:00:29.621 2011: Connected to 172.20.107.23 nestor0167 <c0n200> # ----------------------------------- The output file # ----------------------------------- 2011-11-25T00:00:30-08:00 patterndb(a)catamount.comp.uvic.ca user.warning somehost.uvic.ca ][somehost.uvic.ca][mmfs: Fri Nov 25 00:00:29.618 2011: Accepted and connected to 172.20.102.38 hermes0080 <c0n350> 2011-11-25T00:00:30-08:00 patterndb(a)catamount.comp.uvic.ca user.warning somehost.uvic.ca ][somehost.uvic.ca][mmfs: Fri Nov 25 00:00:29.620 2011: Connecting to 172.20.107.23 nestor0167 <c0n200> 2011-11-25T00:00:30-08:00 patterndb(a)catamount.comp.uvic.ca user.warning somehost.uvic.ca ][somehost.uvic.ca][mmfs: Fri Nov 25 00:00:29.621 2011: Connected to 172.20.107.23 nestor0167 <c0n200> # ----------------------------------- I would expect the output file to read # ----------------------------------- 2011-11-25T00:00:30-08:00 somehost.uvic.ca user.warning mmfs: ][mmfs][Fri Nov 25 00:00:29.618 2011: Accepted and connected to 172.20.102.38 hermes0080 <c0n350> 2011-11-25T00:00:30-08:00 somehost.uvic.ca user.warning mmfs: ][mmfs][Fri Nov 25 00:00:29.620 2011: Connecting to 172.20.107.23 nestor0167 <c0n200> 2011-11-25T00:00:30-08:00 somehost.uvic.ca user.warning mmfs: ][mmfs][Fri Nov 25 00:00:29.621 2011: Connected to 172.20.107.23 nestor0167 <c0n200> # ----------------------------------- -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 2

Network compression
by Rory Toma 14 Feb '12

14 Feb '12

I notice that network compression is listed as a feature for syslog-ng. I can't find any documentation on how to enable this anywhere, however. How do I tell a syslog-ng receiver to accept compressed traffic?

2 3

Documentation questions
by Chris Johnson 13 Feb '12

13 Feb '12

I was reading through the documentation, and noticed that in some examples the log statements are named: log demo_filteredlog { source(s1); filter(demo_filter); destination(d1);}; When I tried using this method in my config file, syslog-ng fails to start. Is this method supported or is this just an oversight with the documentation? Also, the links to the sample pattern database (http://www.balabit.com/downloads/files/patterndb-snapshot/?utm_source=syslo…) apprears to be incorrect. Is this the same pattern database at /downloads/files?path=/patterndb-snapshot? or is there a more recent one available to download? Thanks Chris

2 1

Cant get pattern matching to work
by Chris Johnson 12 Feb '12

12 Feb '12

Sorry for bombarding the mailing list with questions. I have been working through the documentation but just seem to keep hitting obstacles. I have compared this to the examples in the manual and some samples I was able to track down, but Im not seeing where I am going wrong. For this question, I am working with the following data: "MESSAGE" : "RT_FLOW_SESSION_CLOSE: session closed unset: 192.168.199.253/55189->8.8.8.8/53 junos-dns-udp X.X.X.X/60836->8.8.8.8/53 source-nat-rule None 17 trust-to-untrust trust untrust 30455 1(83) 1(83) 2 N/A N/A N/A(N/A) vlan.0", "PROGRAM" : "RT_FLOW" Based on this , I have generated the following pattern database and config just to see if I am heading down the right path. Unfortunately, what I thought would generate a pattern match on the logs and generate the appropriate macros, are not retuning anything. I have also tried taking out the $PROGRAM patter, but that did not seem to help. I have also validated this with pdbtool and it succeeds. Any help is greatly appreciated Chris patterndb.xml <patterndb version='3' pub_date='2011-02-11'> <ruleset name='session_close' id='123456678'> <rules> <rule provider='cj' id='182437592347598' class='session'> <patterns> <pattern> @ESTRING:TEST1:::@ @ANYSTRING:TEST2@ </pattern> </patterns> </rule> </rules> </ruleset> </patterndb> syslog-ng.conf #####Destinations##### destination d_mongodb { mongodb( value-pairs( key("TEST1") key("TEST2") scope("base") ) ); }; #####Parser##### parser pattern_db { db_parser( file("/usr/local/etc/patterndb.xml") ); }; #####Log##### log { source(s_network); parser(pattern_db); destination(d_mongodb); }; Unfortunately, what I thought would pattern match and generate a macro for

2 6

syslog-ng hangs with high message volume to sqlite database
by Patrick Hemmer 10 Feb '12

10 Feb '12

I seem to have run into a situation where syslog-ng (3.3.4) will hang when sending a high message volume to an sqlite3 database. When I run the `loggen` utility with a high message rate, syslog-ng writes out the following error: Jan 28 22:58:12 storm.feystorm.net syslog-ng[22839]: Error running SQL query; type='sqlite3', host='', port='', user='syslog-ng', database='/syslogtest/logs.sql3', error='5: database is locked', query='COMMIT' Jan 28 22:58:12 storm.feystorm.net syslog-ng[22839]: SQL transaction commit failed, rewinding backlog and starting again; Syslog-ng will sit stuck like this for a minute and then log the error again. Sometimes it'll write a few more entries to the database before it errors, but its never more than a few. I'm not sure if this issue is a problem with libdbi or syslog-ng, which is why I'm here and not the bug tracker. I came across this issue trying to track down another issue where syslog-ng seems to hang completely and will never recover (wont respond to anything other than a `kill -9`). Whether this is the same issue or not, I do not know. I was able to trigger this with the following loggen command: loggen -r 60 -I 1 --active-connections=2 -x ./socket This server is a fairly low power server (its a central log collector for my home network), so it you might have to significantly bump up the message rate to duplicate the issue. Config: @version: 3.3 options { keep_hostname(yes); chain_hostnames(no); normalize_hostnames(yes); use_dns(no); use_fqdn(yes); stats_freq(0); mark_freq(0); }; source s_socket { unix-stream("/syslogtest/socket"); }; source s_internal { internal(); }; destination d_internal { file("/syslogtest/internal.log"); }; log { source(s_internal); destination(d_internal); }; destination d_sqlite { sql( type('sqlite3') database("/syslogtest/logs.sql3") table("logs") columns("time", "time_r", "host", "facility", "priority", "program", "pid", "tag", "message") values("$S_UNIXTIME", "$R_UNIXTIME", "$FULLHOST", "$FACILITY_NUM", "$LEVEL_NUM", "$PROGRAM", "$PID", "$DBTAG", "$MSG") null("") flags(explicit-commits) flush_lines(10) flush_timeout(200) ); }; filter f_test1 { message('^prg00000') or program('syslog-ng'); }; filter f_test2 { message('^prg00001') or program('syslog-ng'); }; log { source(s_internal); source(s_socket); destination(d_sqlite); }; #log { source(s_internal); source(s_socket); filter(f_test1); destination(d_sqlite); }; #log { source(s_internal); source(s_socket); filter(f_test2); destination(d_sqlite); }; Database schema: CREATE TABLE logs (id integer primary key autoincrement, time unsigned integer not null, time_r unsigned integer not null, host text not null, facility unsigned integer not null, priority unsigned integer not null, program text not null, pid unsigned integer, tag text, message text not null); CREATE INDEX host on logs (host, time); CREATE INDEX program on logs (program, time);

2 6

AMQP destination driver
by Thomas Wollner 08 Feb '12

08 Feb '12

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi List, some days ago I found logstash and I`m very impressed about the capabilities. It took me half an hour to have a running system and with patternized messages it is real fun to search and find logs. My plans are to classify messages with patterndb and deliver json-format to logstash via amqp for further indexing and searching. I think the options to the driver should be the same like monogdb destination driver, only the delivery is different. do not insert into database, but write it to an amqp enabled message queue like rabbitmq. are there any plans to create an AMQP destination driver? any thoughts on that? best regards, Tom -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.2 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iD8DBQFPMYmRTCCRT+dccOYRAjKmAJ9nfAtCo4aIumsK97ebmXaRM9TO/gCeO+D7 JiHk0SOdi/E3AkzyBo462/0= =Xt93 -----END PGP SIGNATURE-----

3 3

Re: [syslog-ng] log file size limit
by Gene Mats 08 Feb '12

08 Feb '12

Previously there were some discussion to add size_limit option to the file destination driver. > options { file_size_limit(123456); }; > destination log1 { file("/var/log/log1.log" size_limit(456789)); }; I've just compiled syslog-ng 3.3.4, and received a syntax error when trying to use these options. Are there any plans to add this option to syslog-ng in the future? Can the patch that was previously attached, be applied to 3.3.4 source? -Thank You, Gene Mat > On 2011-07-25 20:03, Gergely Nagy wrote: >> On Mon, Jul 25, 2011 at 19:58, Sergei Zhirikov<sfzhi at yahoo.com> wrote: >>> >>> I have a patch that does something of that kind. When the log file grows above certain size syslog-ng switches to a new one and renames the old one. Further processing (such as compression) should be done externally. >>> >>> I posted that patch in this list quite some time ago, but as far as I can tell, it went unnoticed... >> >> Can you please post it again? I'd love to have a look. >> > > Attached. > The patch is against syslog-ng-3.1.4, but the idea is pretty straight > forward, so I don't expect it to be difficult to port to a newer version. > > The patch introduces two new configuration options: > 'file_size_limit' to be used inside global 'options' and 'size_limit' to > be used inside 'file' destination. Each option specifies log file size > limit in bytes. If the global option is set to a value greater than zero > it applies to all 'file' destinations. A particular file destination can > remove the limit by setting it to zero. For example: > > # set the global file size limit > options { file_size_limit(123456); }; > > # set a different size limit for a particular file destination > destination log1 { file("/var/log/log1.log" size_limit(456789)); }; > > # remove size limit for a particular file destination > # (useful only if there is a global size limit set) > destination log2 { file("/var/log/log2.log" size_limit(0)); }; > > The file size is checked after writing each log message and if the file > has grown up to or above the size limit the file is renamed and a new > empty file is created to continue logging to. The name format the > "overgrown" log file is renamed to is "<p>-<s>.<m>-<r>", where <p> - the > full path and name of the original log file, <s> - current time in UNIX > format (seconds since Jan 1, 1970), <m> - fractional part of the current > time (microseconds, 6 digits), <r> - a random number (10 digits). > > The intended use is to have incrond or another similar mechanism to > detect when there is a new "renamed" log file and to process it in > whatever way necessary (gzip it, parse it, send it my email, etc.). > > -- > Sergei. >

2 1

Rewriting Cisco messages
by Martin Holste 07 Feb '12

07 Feb '12

I spent some time yesterday working out the proper regexes to handle the many ways Cisco sends its timestamps, depending on how the device is configured. However, I feel like my solution can be improved upon, so I'd like to see if there's a better way. Here's what I've got so far that seems to be working, though there seems to be a 10-15% CPU penalty at the moment: #4w4d: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/15, changed state to down #1y46w: %CDP-4-NATIVE_VLAN_MISMATCH: Native VLAN mismatch discovered on GigabitEthernet0/1 (1), with S-COR-02 GigabitEthernet2/15 (40). filter f_rewrite_cisco_program_3 { match('^\d+[ywdh]\d+[ywdh]: (%[^:]+): ([^\n]+)' value("MSGONLY") type("pcre") flags("store-matches" "nobackref")); }; #Feb 6 16:43:32.219: %LINK-3-UPDOWN: Interface FastEthernet2/0/42, changed state to up filter f_rewrite_cisco_program_2 { match('^[\*\.]?(?:Jan|Feb|Mar|Apr|May|Jun|Jul|Aug|Sep|Oct|Nov|Dec)\s+\d{1,2}\s\d{1,2}:\d{1,2}:\d{1,2}(?:\.\d+)?(?: [A-Z]{3})?: (%[^:]+): ([^\n]+)' value("MSGONLY") type("pcre") flags("store-matches" "nobackref")); }; # Others where MSGONLY starts with PROGRAM filter f_rewrite_cisco_program { match('^(%[A-Z]+\-\d\-[0-9A-Z]+): ([^\n]+)' value("MSGONLY") type("pcre") flags("store-matches" "nobackref")); }; rewrite r_cisco_program { set("$1", value("PROGRAM") condition(filter(f_rewrite_cisco_program) or filter(f_rewrite_cisco_program_2) or filter(f_rewrite_cisco_program_3))); set("$2", value("MESSAGE") condition(filter(f_rewrite_cisco_program) or filter(f_rewrite_cisco_program_2) or filter(f_rewrite_cisco_program_3))); }; This works, but is there a better way?

3 8

How to use parser
by 大町靖 07 Feb '12

07 Feb '12

Hello Experts, Please let me know how to use parser setting. This is the syslog message. Msg1 #AA# Msg2 I want to divide this message. I expect the below. Msg1 Msg2 But It doesn't work well , though I wrote the configuration below. parser p1{ csv-parser(columns("Msg1","Msg2") delimiters("#AA#") template ("$Msg1\n$Msg2"); }; log { source(s_sys); parser(p1); destination(d_mesg); }; I check the output. The result wasn't unexpected. Msg1 #AA# Msg2 If my configuration is wrong, please let me know. Thanks, Yasu

2 1