- syslog-ng - lists.balabit.hu

Help with pattern db correlation
by Martin Holste 18 Dec '10

18 Dec '10

I'm trying to build a correlation message for Ironport similar to the Postfix example that was referred to in the discussions. Here's what I have so far, but it's not quite working: <ruleset> <rules> <rule class="10" id="10" context-id="ironport-mid" context-timeout="10" context-scope="program"> <patterns> <pattern>Info: Start MID @NUMBER:mid:@ ICID @NUMBER:icid:@</pattern> </patterns> <values> <value name="icid">$icid</value> </values> <examples> <example> <test_message program="ironport_mail_logs">Info: Start MID 144753300 ICID 696117306</test_message> </example> </examples> </rule> <rule class="10" id="10" context-id="ironport-mid" context-timeout="10" context-scope="program"> <patterns> <pattern>Info: Message finished MID @NUMBER:mid:@ done</pattern> </patterns> <actions> <action> <message> <values> <value name="MESSAGE">IronPort message complete: icid: $icid, mid: $mid</value> </values> </message> </action> </actions> <examples> <example> <test_message program="ironport_mail_logs">Info: Message finished MID 144753300 done</test_message> </example> </examples> </rule> </rules> </ruleset> I'm getting the triggered action, but the icid is null while the mid is filled in. What am I missing?

3 4

load failures in afsocket and afsql
by Matthew Hall 17 Dec '10

17 Dec '10

I am getting the following on load: Error opening plugin module; module='afsocket', error='/home/y/lib64/syslog-ng/libafsocket.so: undefined symbol: libnet_build_ipv4' Error opening plugin module; module='afsql', error='/home/y/lib64/syslog-ng/libafsql.so: undefined symbol: dbi_result_free' Error opening plugin module; module='afsocket', error='/home/y/lib64/syslog-ng/libafsocket.so: undefined symbol: libnet_build_ipv4' The rpath looks OK: megahall@logproxy2:~$ readelf -a /home/y/lib64/syslog-ng/libafsocket.so | fgrep -i rpath 0x000000000000000f (RPATH) Library rpath: [/home/y/lib64] megahall@logproxy2:~$ readelf -a /home/y/lib64/syslog-ng/libafsql.so | fgrep -i rpath 0x000000000000000f (RPATH) Library rpath: [/home/y/lib64] megahall@logproxy2:~$ megahall@logproxy2:~$ ldd /home/y/lib64/syslog-ng/libafs* | fgrep -i '(dbi|net)' megahall@logproxy2:~$ The libraries are in a reasonable location: /home/y/lib64/libdbi.so.1.0.0 /home/y/lib64/libnet.so.1.5.0 /home/y/lib64/dbd/libdbdsqlite3.so /home/y/lib64/dbd/libdbdmysql.so /home/y/lib64/libnet.so.1 /home/y/lib64/libdbi.so.1 /home/y/lib64/libdbi.so Reading through the glib docs for glib modules, it seems like the .la files are maybe not containing the right library dependencies, or something like this. However adding the library directories using LD_LIBRARY_PATH as a temporary test does not help. Because this step fails, it's not possible to use tcp, udp, or any of the other important drivers you need to collect logs. I could really use some advice on this one! Matthew.

5 25

Help with db_parser()
by Giovanni Mancuso 17 Dec '10

17 Dec '10

Hi, I configure my Java application to log with log4j in syslog server, and i configure my syslog-ng to store data in mysql database, but this doen't work. My syslog-ng configuration is: @version: 3.0 options { chain_hostnames(no); stats_freq(43200); }; source src { unix-stream("/dev/log" max-connections(256)); internal(); udp(ip("127.0.0.1") port(514)); file("/proc/kmsg"); }; destination mcs { file("/var/log/mw-collaboration/mw-collaboration-loginfile.log"); }; destination mcs_sql { sql( type(mysql) host("localhost") username("syslogng") password("syslogng") database("mcslogin") table("mcslogin") columns("date varchar(32)","loginuser varchar(32) ","ipsource varchar(32)") values("${S_YEAR}-${S_MONTH}-${S_DAY} ${S_HOUR}:${S_MIN}:${S_SEC}","${LOGIN_USER}","${IP_SOURCE}") ); }; filter f_mcs { facility(local2); }; parser p_mcs { db_parser(file("/etc/syslog-ng/patterndb.d/mcs.xml")); }; log { source(src); filter(f_mcs); parser(p_mcs); destination(mcs_sql); destination(mcs); }; I created also a db_parser file that is: <patterndb version='1' pub_date='2010-12-14'> <program name='mcs'> <pattern>mcs</pattern> <rule id='mcs' class='system'> <pattern>###############Accesso dell'utente @STRING:LOGIN_USER@ da ip @IPv4:IP_SOURCE@</pattern> </rule> </program> </patterndb> With this configuration, i have that in "mcs" destination it writes all information: Dec 16 11:55:44 localhost mcs[123] ###############Accesso dell'utente xxxx(a)xxx.mailware.it da ip 111.222.333.444 Dec 16 12:53:23 localhost mcs[123] ###############Accesso dell'utente xxxx.1(a)xxx.mailware.it da ip 111.222.333.444 Dec 16 14:07:40 localhost mcs[123] ###############Accesso dell'utente xxxx.1(a)xxx.mailware.it da ip 111.222.333.444 but in "mcs_sql" destination, it writes only date: mysql> desc mcslogin; +-----------+-------------+------+-----+---------+-------+ | Field | Type | Null | Key | Default | Extra | +-----------+-------------+------+-----+---------+-------+ | date | varchar(32) | YES | MUL | NULL | | | loginuser | varchar(32) | YES | | NULL | | | ipsource | varchar(32) | YES | | NULL | | +-----------+-------------+------+-----+---------+-------+ 3 rows in set (0.00 sec) mysql> select * from mcslogin limit 10 -> ; +---------------------+-----------+----------+ | date | loginuser | ipsource | +---------------------+-----------+----------+ | 2010-12-15 11:02:16 | | | | 2010-12-15 11:11:09 | | | | 2010-12-15 17:53:01 | | | | 2010-12-15 18:11:55 | | | | 2010-12-15 18:12:54 | | | | 2010-12-15 18:35:07 | | | | 2010-12-16 11:55:36 | | | | 2010-12-16 11:55:44 | | | | 2010-12-16 11:55:44 | | | | 2010-12-16 12:53:23 | | | +---------------------+-----------+----------+ 10 rows in set (0.00 sec) Can you help me? Thanks

3 6

memory exhausted in /opt/syslog-ng/etc/syslog-ng.conf at line 5347
by Mukarram Syed 17 Dec '10

17 Dec '10

Hi , I am getting the following error when starting syslog-ng.conf file which has more than 5000 lines in it and getting the following error: # service syslog-ng start memory exhausted in /opt/syslog-ng/etc/syslog-ng.conf at line 5347. syslog-ng documentation: http://www.balabit.com/support/documentation/?product=syslog-ng mailing list: https://lists.balabit.hu/mailman/listinfo/syslog-ng [FAILED] When I truncate the file to < 5000 lines it starts up fine. This is running version syslog-ng-3.0.8-1.rhel5.x86_64 Mailing list does not have any clues to it. Anything in the syslog-ng.conf file that needs to be done to accept more lines? Any ideas would be appreciated. Thanks # mukarram Mukarram Syed

6 23

Patch for launch path bug in syslog-ng.h
by Matthew Hall 17 Dec '10

17 Dec '10

Hello, When compiling syslog-ng 3.2.1 into my environment I discovered a bug in how this configure option is applied: --libexecdir=/home/y/libexec64 When this option is entered, this is generated in lib/syslog-ng.h: #if ENABLE_ENV_WRAPPER #define PATH_SYSLOGNG PATH_PREFIX "/libexec/syslog-ng" #endif As you can see this is a hard coded libexec path which does not match what my environment expects. Unfortunately it appears that the libexec variable never makes it into a C macro so I'm not totally clear on how to make a permanent patch for this one. I will work around it with something like this temporary patch for now. --- /home/megahall/syslog-ng.h 2010-12-13 15:54:56.000000000 -0800 +++ syslog-ng-3.2.1/lib/syslog-ng.h 2010-12-13 16:06:22.000000000 -0800 @@ -44,7 +44,7 @@ #define PATH_PIDFILE PATH_PIDFILEDIR "/syslog-ng.pid" #define PATH_CONTROL_SOCKET PATH_PIDFILEDIR "/syslog-ng.ctl" #if ENABLE_ENV_WRAPPER -#define PATH_SYSLOGNG PATH_PREFIX "/libexec/syslog-ng" +#define PATH_SYSLOGNG PATH_PREFIX "/libexec64/syslog-ng" #endif #define PATH_PERSIST_CONFIG PATH_LOCALSTATEDIR "/syslog-ng.persist" Another issue. Because of this bug syslog-ng fails to launch with return code 127. But it prints nothing to the console explaining why it failed: execve("/home/y/libexec/syslog-ng", ["/home/y/sbin64/syslog-ng"], [/* 21 vars */]) = -1 ENOENT (No such file or directory) exit_group(127) = ? I think some message should be added in syslog-ng/wrapper.c. This is an uncompiled example but should be very close. --- /home/megahall/wrapper.c 2010-12-13 16:15:29.000000000 -0800 +++ syslog-ng-3.2.1/syslog-ng/wrapper.c 2010-12-13 16:14:21.000000000 -0800 @@ -45,6 +45,7 @@ putenv(ldlibpath); } #endif + fprintf(stderr, "syslog-ng failed to launch because of XYZ\n"); execv(PATH_SYSLOGNG, argv); return 127; } Thanks for maintaining syslog-ng. I hope this helps keep the product great! Regards, Matthew Hall.

2 2

Convert syslog to traps
by Jay 14 Dec '10

14 Dec '10

Have a requirement to convert all incoming syslogs to SNMP traps and send it to another host. One option I could think of is to use program () destination. When I tried this option, I find that syslog-ng is continuously sending newline characters to the specified program. i.e. even when no syslog is received, syslog-ng seems to be pumping newline chars to the specified program. Also I read the warning message in admin guide that, it will open up the door to DOS attack. Could someone let me know the best way to achieve this, please ? Thanks in advance for the help. John

5 8

Microsoft SQL connect issue
by Shawn Cannon 14 Dec '10

14 Dec '10

When a Microsoft SQL connection is attempted, I get the following errors. I am trying to connect to a names instance of MS SQL over TCP 1785. Any ideas? Thanks.... Dec 13 10:33:23 syslog01 syslog-ng[27963]: Error running SQL query; type='freetds', host='sql.hidden.local\instance', port='1785', user='user', database='syslog', error='', query='CREATE TABLE syslogng (datetime varchar(16), host varchar(32), program varchar(32), pid varchar(8), message varchar(4096))' Dec 13 10:33:23 syslog01 syslog-ng[27963]: Error creating table, giving up; table='syslogng' Dec 13 10:33:23 syslog01 syslog-ng[27963]: Error checking table, disconnecting from database, trying again shortly; time_reopen='60' Shawn Cannon IT Professional http://www.shawncannon.com Contact Me <http://www.linkedin.com/pub/shawn-cannon/5/84/81a> LinkedIn<http://www.linkedin.com/pub/shawn-cannon/5/84/81a> <http://www.facebook.com/rolltidega72> Facebook<http://www.facebook.com/rolltidega72> <http://twitter.com/rolltidega> Twitter <http://twitter.com/rolltidega> My blog: My thoughts on VMworld… a week later…<http://www.shawncannon.com/?p=19>

3 13

Help configuring a basic setup
by Shawn Cannon 10 Dec '10

10 Dec '10

OK I want to setup syslog-ng to listen for syslog traffic on the standard UDP port and have the messages received logged to a Microsoft SQL database. I am having trouble finding documentation I can understand. I am a very GUI driven admin so all the command line stuff is messing with me! If anyone can assist I would appreciate it. Thanks.... Shawn Cannon IT Professional http://www.shawncannon.com Contact Me <http://www.linkedin.com/pub/shawn-cannon/5/84/81a> LinkedIn<http://www.linkedin.com/pub/shawn-cannon/5/84/81a> <http://www.facebook.com/rolltidega72> Facebook<http://www.facebook.com/rolltidega72> <http://twitter.com/rolltidega> Twitter <http://twitter.com/rolltidega> My blog: My thoughts on VMworld… a week later…<http://www.shawncannon.com/?p=19>

1 0

Re: [syslog-ng] Creating a "log" block (OSE 3.2.1)
by Eric Berggren 10 Dec '10

10 Dec '10

> thanks for noticing this error. I've corrected it in the source repo of the > adminguide, and will update the docs on the website when someone tells me if > using {} within blocks is permitted (but now buggy) or forbidden. > changing the type of the block to root should workaround the problem. I see, it's really about using { }'s within the block definition. Making it a "root" block (as a completely separately log {} stanza) is probably cleaner in this particular case, but the same restriction applies at the moment -- the example from the OSE 3.2 documentation [5.3.2]: block root mylogs() { source s_file { file("/var/log/mylogs.log" follow_freq(1)); }; destination d_local { file("/var/log/messages"); }; log { source(s_file); destination(d_local); }; }; doesn't parse either (same error on the final "};") The documentation (html/PDF) could use some minor updates in this area for clarity: - block definitions also require ()'s after the name (including non-parameterized); e.g. block root foo() { ... } - perhaps add an additional example for using parameterized blocks; e.g. block destination foo ( logname("some.log") ) { file( "/var/log/`logname`" ... ); }; destination bar { foo( logname("other.log") ) }; destination baz { foo() }; regards, -eric ps: I have yet to actually *test* any of this new functionality myself yet (later today :)

3 4

Re: [syslog-ng] Creating a "log" block (OSE 3.2.1)
by Eric Berggren 10 Dec '10

10 Dec '10

Admittedly, I ended up just using a separate log {} statement (obviating the need for the block statement in this particular instance, but using blocks successfully elsewhere -- very nice). I tested the earlier lexer patch and found it works (as I imagined you did), and sure I'll find more creative uses . Allowing the extra ";" when referencing blocks indeed makes it look more consistent . thanks again -eric > I've just committed these 3 patches that implement block support for > block statements. I've also resolved the '{' issue and also made it > possible to use several ';' characters in place of a single one.

1 0

keep_timestamp() and use_time_recvd() equivalent?
by Craig Bell 09 Dec '10

09 Dec '10

Q: Is option keep_timestamp(no) reasonably equivalent to use_time_recvd(yes)? Background: We recently upgraded from OSE v2.1 to v3.1.2. This went well, but our global options include the deprecated use_time_recvd(). We see this advisory on startup: > Your configuration file uses an obsoleted keyword, please update your configuration; keyword='use_time_recvd', change='Use R_ or S_ prefixed macros in templates' We don't modify the default templates, so adding these for the first time wouldn't be my first choice. Instead, can I simply use global option keep_timestamp(no) as a replacement? I looked around for advice, but didn't see anything obvious. Thanks in advance! -cheers, CSB

2 2

advice/assistance with parsing attempt requested
by syslog-ng2010＠hushmail.com 08 Dec '10

08 Dec '10

i've spent the better part of the past week reading and trying to understand both the documentation and list posts trying to sort this out, if anyone can offer some advice as to whether this is possible or not and if so, what i'm doing wrong; i would really appreciate it! … i have a simple enough task, or so i thought! i've got a syslog stream being received by syslog-ng with too much data. what i'd like to do is parse out pieces of the stream and write only those to a file. the tricky part is that the order of the stream is very variable so that sometimes the desired named parser preceding strings and associated values are present and sometimes not. furthermore, the extra data is also quite variable. can this challenge even be addressed with syslog-ng ose? if so, can it be done with patterned without creating a pattern for EVERY variation of possible streams? for clarification, we've tried to leverage an external perl script which does this using regexs but, it seems that it can't keep up with the stream, we only receive 10% of the original events in the output. if this (external parsing script) only way this can be done, we will continue our efforts to enhance the external script but, if this is possible to be done natively within syslog-ng, i'd rather do that. with the following configuration, i am able to generate output log entries which correctly contain the global macros of $DATE $FULLHOST $PROGRAM as well as the strings preceding the named parser variables but, not the named parser macros. my output looks like this: Dec 2 11:11:11 127.0.0.1 ABC: 0 namedparser1= namedparser2= namedparser3= namedparser4= namedparser5= *****examples of entries in source stream***** Dec 2 11:11:11 127.0.0.1 ABC: 0 namedparser1=namedparser1value extra1=extravalue1 namedparser2=namedparser2value namedparser3=namedparser3value extra2=extravalue2 namedparser4=namedparser4value namedparser5=namedparser5value extra3=extravalue3 Dec 2 11:11:11 127.0.0.1 ABC: 0 extra1=extravalue1 namedparser3=namedparser3value extra2=extravalue2 namedparser4=namedparser4value namedparser5=namedparser5value extra3=extravalue3 extra4=extravalue4 Dec 2 11:11:11 127.0.0.1 ABC: 0 namedparser1=namedparser1value extra1=extravalue1 namedparser2=namedparser2value namedparser3=namedparser3value extra2=extravalue2 namedparser4=namedparser4value extra3=extravalue3 *****examples of desired output***** Dec 2 11:11:11 127.0.0.1 ABC: 0 namedparser1=namedparser1value namedparser2=namedparser2value namedparser3=namedparser3value namedparser4=namedparser4value namedparser5=namedparser5value Dec 2 11:11:11 127.0.0.1 ABC: 0 namedparser3=namedparser3value namedparser4=namedparser4value namedparser5=namedparser5value Dec 2 11:11:11 127.0.0.1 ABC: 0 namedparser1=namedparser1value namedparser2=namedparser2value namedparser3=namedparser3value namedparser4=namedparser4value *****included in conf file***** parser pattern_db { db_parser(file("/opt/syslog-ng/config/patterndb.xml") }; template reduced { template("$DATE $FULLHOST $PROGRAM: 0 namedparser1=$NAMEDPARSER1 namedparser2=$NAMEDPARSER2 namedparser3=$NAMEDPARSER3 namedparser4=$NAMEDPARSER4 namedparser5=$NAMEDPARSER5 \n"); template_escape(no); } destination d_logfile { file("/opt/syslog-ng/logs/logfile" template(reduced)); } log { source(source); parser(pattern_db); destination(d_logfile); }; *****patterndb.xml contents***** <patterndb version='3' pub_date=''> <ruleset name='globe' id='1234567890'> <pattern>ABC</pattern> <rules> <rule provider='someone' id='123' class='system'> <patterns> <pattern>ABC namedparser1=@ESTRING:NAMEDPARSER1:\ @ namedparser2=@ESTRING:NAMEDPARSER2:\ @ namedparser3=@ESTRING:NAMEDPARSER3:\ @ namedparser4=@ESTRING:NAMEDPARSER4:\ @ namedparser5=@ESTRING:NAMEDPARSER5:\ @</pattern> <patterns> </rule> <rules> </ruleset> </patterndb> MANY thanks in advance!

5 8

[Bug 99] New: syslog-ng unable to write to files owned by another user and not world writable
by bugzilla＠bugzilla.balabit.com 08 Dec '10

08 Dec '10

https://bugzilla.balabit.com/show_bug.cgi?id=99 Summary: syslog-ng unable to write to files owned by another user and not world writable Product: syslog-ng Version: 3.1.x Platform: PC OS/Version: Linux Status: NEW Severity: minor Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: mimosinnet(a)ningunlugar.org Type of the Report: bug Estimated Hours: 0.0 I used to have a new destination that logged messages into a file owned by a certain user. This worked in syslog-ng-3.0.4: destination problem { file("/var/log/problem.log" owner("o3o") group("users")); }; # ls -lisah problem.log 6268890 920K -rw------- 1 o3o users 915K 7 des 12:52 problem.log After upgrading to syslog-ng-3.1.1, I get the following message: Dec 7 12:52:37 o3o syslog-ng[12594]: Error opening file for writing; filename='/var/log/problem.log', error='Permission denied (13)' Dec 7 12:52:37 o3o syslog-ng[12594]: syslog-ng internal() messages are looping back, preventing loop by suppressing further messages; recurse_count='2' I can write to the file as the o3o user (i.e. echo "test" >> /var/log/problem.log). I have tried to put the file in the home directory with the same result: destination problem { file("/home/o3o/problem.log" owner("o3o") group("users")); }; I had to give world permissions to the file for syslog-ng to be able to write to it: destination problem { file("/var/log/problem.log" owner("o3o") group("users") perm(0622)); }; The file should only be writable for the specific user. This seems similar to this other bug: https://bugzilla.balabit.com/show_bug.cgi?id=52 Thanks for the attention! -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

3 5

Creating a "log" block (OSE 3.2.1)
by Eric Berggren 08 Dec '10

08 Dec '10

Howdy Looking to use the new fancy "block" construct in syslog-ng 3.2.1 (OSE) -- (the example in the documentation [5.3.2] doesn't parse, but figured out needs to include ()'s as well in the block definition). Meanwhile, trying to define a "log" block, that contains other log {} stanza's, but fails to parse: filter f_zort { level(err); }; destination d_weblog_unknown { file("/tmp/foo"); }; block log bl_weblog() { filter(f_zort); log { destination(d_weblog_unknown); }; }; Error parsing main, syntax error, unexpected '}', expecting $end in /usr/local/etc/syslog-ng.d/10-weblog at line 7, column 1: included from /etc/syslog-ng.conf line 15, column 37 >> }; << The idea being I would include this within another "log { }" stanza in the main file. If I remove "log { destination(...); };" from the "bl_weblog()" block, it parses okay. I have another ("destination") block within the config that also parses okay (even parameterized). Strangely, commenting out the entry with a "#" at the start of that line still results in a syntax error. The error is independent of whether bl_weblog() is even referenced anywhere. The main config may include only 'include "10-weblog"' and nothing else to demonstrate. Seems otherwise a valid scenario, no ? thanks, -eric

3 3

[Bug 101] New: Typo in modules/afsocket/Makefile.am
by bugzilla＠bugzilla.balabit.com 08 Dec '10

08 Dec '10

https://bugzilla.balabit.com/show_bug.cgi?id=101 Summary: Typo in modules/afsocket/Makefile.am Product: syslog-ng Version: 3.2.x Platform: PC OS/Version: All Status: NEW Severity: normal Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: vinschen(a)redhat.com Type of the Report: --- Estimated Hours: 0.0 Created an attachment (id=27) --> (https://bugzilla.balabit.com/attachment.cgi?id=27) Patch to fix issue "Typo in modules/afsocket/Makefile.am" Usually $DESTDIR is used without a trailing slash, for instance: $(DESTDIR)$(prefix) In modules/afsocket/Makefile.am, it's erroneously used with a trailing slash in the rules creating the libafsocket.so symlink: $(DESTDIR)/$(moduledir) Assuming $(DESTDIR) is empty, this may result in an erroneous double slash at the beginning of the resulting path: moduledir=/usr/lib/syslog-ng ==> $(DESTDIR)/$(moduledir) == //usr/lib/syslog-ng Per POSIX, a double slash at the start of the path is allowed to have a system-dependent meaning, so this shouldn't happen. I've attached a patch to fix this issue. Corinna -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 1

[Bug 102] New: Add system type "CYGWIN" to system() plugin
by bugzilla＠bugzilla.balabit.com 08 Dec '10

08 Dec '10

https://bugzilla.balabit.com/show_bug.cgi?id=102 Summary: Add system type "CYGWIN" to system() plugin Product: syslog-ng Version: 3.2.x Platform: PC OS/Version: Other Status: NEW Severity: normal Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: vinschen(a)redhat.com Type of the Report: enhancement Estimated Hours: 0.0 The scl/system/generate-system-source.sh script driving the system() plugin does not recognize Cygwin. The attached patch adds Cygwin to the script so the system() plugin does the right thing. Corinna -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 2

Re: [syslog-ng] advice/assistance with parsing attempt requested
by syslog-ng2010＠hushmail.com 08 Dec '10

08 Dec '10

thanks for all the help ... in parser.py, i was using sys.stdin.readlines() which, i wound up changing to sys.stdin.xreadlines() to get the desired results. many thanks again! On Wed, 08 Dec 2010 16:50:37 +0100 Lars Kellogg-Stedman <lars(a)oddbit.com> wrote: >> >> the issue has me really stumped. how/why would the output from >perl >> make it to /opt/syslog-ng/logs/logfile in "real-time" yet, the >> output from python only make it to /opt/syslog-ng/logs/logfile >as a >> batch upon stopping the service!?!? >> > >This sort of thing typically happens because the program is using >buffered >IO, and output is only flushed when the buffer is full. If you >were to >leave things running for a while and generate a lot of log >traffic, you >would probably see the output show up periodically in large >chunks. > > This thread on Stackoverflow discusses a variety of solutions to >the >problem: > > http://stackoverflow.com/questions/107705/python-output- >buffering > >The simplest is probably to run Python with the "-u" flag (or >output to >stderr instead of stdout).

1 0

Re: [syslog-ng] advice/assistance with parsing attempt requested
by syslog-ng2010＠hushmail.com 08 Dec '10

08 Dec '10

MANY thanks for all of the great answers guys! as the pattern permutations would be less than desirable to maintain, i've gone ahead and given the program destination another attempt. this time leveraging python as opposed to perl (even though i too believe perl ought be able to handle the volume (roughly 300k per 5 minutes). i didn't write the original non- functional script and happen to favor python. by setting two destinations (one writing the raw stream to file and the other sending to a program which does parsing) i can see that this external parsing does indeed keep a one to one ratio! now, i'm now faced with a new challenge which for the life of me can't begin to get my head around ... original script had: destination d_parser { program("/opt/bin/perl /opt/syslog-ng/config/parser.pl >> /opt/syslog-ng/logs/logfile"); }; and, for what it was able to output (albeit anemic), worked fine. however ... current script has: destination d_parser { program("/opt/bin/python /opt/syslog-ng/config/parser.py >> /opt/syslog-ng/logs/logfile"); }; when i run "/opt/bin/python /opt/syslog-ng/config/parser.py" from the cmd line, and pipe input into it (i'm reading stdin); it runs as expected and the desired output is sent to screen/stdout when i start the syslog-ng service (which uses the conf file which in turn calls the python program in destination d_parser), unlike the perl script which does the same thing (print to screen/stdout) and does in fact add entries to /opt/syslog-ng/logs/logfile; no output is ever sent to the /opt/syslog-ng/logs/logfile ... UNTIL, i stop the syslog service and then, all the expected output comes flushing into /opt/syslog-ng/logs/logfile the issue has me really stumped. how/why would the output from perl make it to /opt/syslog-ng/logs/logfile in "real-time" yet, the output from python only make it to /opt/syslog-ng/logs/logfile as a batch upon stopping the service!?!? to clarify, i'm only sending to file currently to test, eventually this will be handed off to another syslog server via: destination d_logdevice { tcp("10.0.0.1" port(11235)); }; i hope this is something simple because as soon as i can get the output from the python like the perl, this project is almost done! lastly, this all being done with syslog-ng ose 3.1.1 thanks in advance! On Mon, 06 Dec 2010 21:01:37 +0100 Bill Anderson <Bill.Anderson(a)bodybuilding.com> wrote: >On Dec 6, 2010, at 12:37 PM, Martin Holste wrote: > >>> Agreed, Perl is plenty quick, hence my wondering about the >actual volume. If it is too much for Perl I'd go w/C++. >> >> From what I can tell, PCRE in Perl (or Python or whatever) is >really >> close to C/C++ speeds because they're essentially using the same >> library and therefore mostly the same syscalls. I'd be really >> interested if anyone has benchmarks. I'd expect something like >10% >> better performance in C, but not much more, assuming that the >vast >> majority of CPU time is spent on PCRE. > >Yeah I was thinking the overhead might be in what is done, as >opposed to just the RE portion. Of course, the OP script might be >implemented rather differently. ;) > > >> >>> Personally, I'd make the last step routing back into syslog-ng >with a source on a custom port and letting syslog handle the >writing to disk. That way you can still use macros such as >timestamps, etc.. Then again, that may be because I do that all >the time. ;) A log statement that takes everything from the custom >source and logs to a file should work beautifully; no need for >filters though you could still do additional processing if needed. >That said I'd also consider running a daemon that accepted all the >input, formatted it, and then sent it to syslog-ng, pointing the >clients at the custom daemon if that was possible. >>> >>> One advantage to the daemon route is that it wouldn't *have* to >reside on the same system. >> >> Yep, you could definitely let Syslog-NG handle the last mile as >well. >> I was trying to keep the scope as narrow as possible in my >example. >> >> I wonder if you could build an NFA state machine by >conditionally >> looping output from a pattern-db parsed message into a source in >> Syslog-NG with a different pattern-db, depending on the previous >> output. Something like a token parser pdb that does an ESTRING >up >> until " " and another one that only expects the key/val pair to >be >> sent to it as the message. So it comes in as k1=v1 k2=v2 and >the >> first kv gets gobbled up and then sent to another pdb source >with a >> pdb which only matches if the message starts with certain terms. > Then >> the rest of the original message is looped back to itself using >> @ANYSTRING@ to capture the remainder, that is, minus the kv >which was >> sent to the kv pdb. It would keep recursively looping like that >until >> there's no message left. If that all worked, your pattern db >would be >> extremely simple as it would just be a pattern per key you were >> looking for, and order would no longer be an issue. > >Maybe I'm nuts, but that sounds awesome to me. :D > >> Of course there's >> still the problem of demuxing the whole thing back into a >coherent >> message, but I think that could be done a number of ways by >passing >> the MSGID token with each part and using the new conditionals >present >> in OSE 3.2. > >Well, there is message correlation in 3.2.1 right? muahahaha > >> If OSE 3.3 can really do close to 1 million msgs/sec, >> then the overhead of resubmitting the same log many times may be >> bearable, especially with the threading. > >True the rate might be the downside to that mechanism. However, >the terseness of the messages might make up for some of it. > > >___________________________________________________________________ >___________ >Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng >Documentation: >http://www.balabit.com/support/documentation/?product=syslog-ng >FAQ: http://www.campin.net/syslog-ng/faq.html

3 2

[Bug 100] New: Hostnames that start with 4+ digits mess up the time system
by bugzilla＠bugzilla.balabit.com 08 Dec '10

08 Dec '10

https://bugzilla.balabit.com/show_bug.cgi?id=100 Summary: Hostnames that start with 4+ digits mess up the time system Product: syslog-ng Version: 2.0.x Platform: PC OS/Version: Linux Status: NEW Severity: normal Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: D.Ojalvo(a)f5.com Type of the Report: --- Estimated Hours: 0.0 Created an attachment (id=26) --> (https://bugzilla.balabit.com/attachment.cgi?id=26) My patch proposal If syslog is handling a message with an rfc 3164-type timestamp then a hostname that begins with 4 digits, the time parser will mess up because it thinks that it is dealing with a linksys-style timestamp that includes the year, when the rfc3164 timestamp does not. I wrote a patch that adds a strict rfc3164 parsing option to the syslog startup. -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

2 2

Re: [syslog-ng] [Bug 100] New: Hostnames that start with 4+ digits mess up the time system
by Patrick H. 08 Dec '10

08 Dec '10

Ahh, didnt know that was overridden. Thanks :-) -Patrick Sent: Tue Dec 07 2010 17:17:42 GMT-0700 (Mountain Standard Time) From: Daniel Ojalvo <D.Ojalvo(a)F5.com> To: Patrick H. <syslogng(a)feystorm.net>, Syslog-ng users' and developers' mailing list <syslog-ng(a)lists.balabit.hu> Subject: Re: [syslog-ng] [Bug 100] New: Hostnames that start with 4+ digits mess up the time system > > Yes, but that is superseded by rfc 1123. > http://tools.ietf.org/html/rfc1123#page-13 > > > > It's definitely an edge case though. I googled around for a while and > I found no other mentions of a bug like this. > > > > Daniel > > > > *From:* feystorm(a)q.com [mailto:feystorm@q.com] *On Behalf Of *Patrick H. > *Sent:* Tuesday, December 07, 2010 4:07 PM > *To:* Syslog-ng users' and developers' mailing list; Daniel Ojalvo > *Subject:* Re: [syslog-ng] [Bug 100] New: Hostnames that start with 4+ > digits mess up the time system > > > > Well to be fair, hostnames that start with a number are a violation of > RFC 952. > http://www.ietf.org/rfc/rfc952.txt > "The first character must be an alpha character" > > -Patrick > > > Sent: Tue Dec 07 2010 16:49:03 GMT-0700 (Mountain Standard Time) > From: bugzilla(a)bugzilla.balabit.com <mailto:bugzilla@bugzilla.balabit.com> > To: syslog-ng(a)lists.balabit.hu <mailto:syslog-ng@lists.balabit.hu> > Subject: [syslog-ng] [Bug 100] New: Hostnames that start with 4+ > digits mess up the time system > > https://bugzilla.balabit.com/show_bug.cgi?id=100 > > Summary: Hostnames that start with 4+ digits mess up the time > system > Product: syslog-ng > Version: 2.0.x > Platform: PC > OS/Version: Linux > Status: NEW > Severity: normal > Priority: unspecified > Component: syslog-ng > AssignedTo: bazsi(a)balabit.hu <mailto:bazsi@balabit.hu> > ReportedBy: D.Ojalvo(a)f5.com <mailto:D.Ojalvo@f5.com> > Type of the Report: --- > Estimated Hours: 0.0 > > > Created an attachment (id=26) > --> (https://bugzilla.balabit.com/attachment.cgi?id=26) > My patch proposal > > If syslog is handling a message with an rfc 3164-type timestamp then a hostname that begins with 4 digits, > the time parser will mess up because it thinks that it is dealing with a linksys-style timestamp that includes the year, > when the rfc3164 timestamp does not. I wrote a patch that adds a strict rfc3164 parsing option to the syslog startup. > > >

1 0

Support of welf format
by Yann I. 07 Dec '10

07 Dec '10

Hello, I would like to know whether syslog-ng can receive and manage logs which have the welf format ? Regards, Yann I.

3 2

Software for Analysing and Reporting Syslog Logs
by Heinze, René 07 Dec '10

07 Dec '10

Hello, I'm looking for a software to analyse and creating reports from syslog logs. It there anything you can recommend? Thanks, Best, René Heinze

3 2

Syslog-ng and Mysql Not working
by Girish-Agarwal (Contractor) 06 Dec '10

06 Dec '10

Hi, We are trying to setup syslog-ng to directly log messages to mysql but I get the following Error in the client messages file:- Dec 1 14:45:27 testbox syslog-ng[15637]: Error running SQL query; type='mysql', host='10.10.10.10', port='4406', user='syslog', database='testdb', error='1142: create command denied to user: \'syslog(a)10.20.30.40\' for table \'logs\'', query='CREATE TABLE logs (host text, facility text, priority text, tag text, datetime text, program text, server text, msg text)' Dec 1 14:45:27 testbox syslog-ng[15637]: Error checking table, disconnecting from database, trying again shortly; time_reopen='60' I am really confused as to why it is trying to create Table into the database. I already have the table created and I have verified successful connection from this box to the database server. It needs to insert the entries into the database and should not try to CREATE TABLE. I am using syslog-ng 3.0.6 version . Regards, Girish

3 10

Host name handling : FULLHOST_FROM explanation
by Christophe Brocas 06 Dec '10

06 Dec '10

Hello I am working on the right and consistent way to store log messages in destination files. I try to obtain a single file per host, based on HOST value in log messages.. Currently, I have inconsistent values in HOST part of log messages, so destination files are also inconsistents. To answer to this problem, I am going to use at first the SOURCEIP macro in my destination file paths. But I want to improve this situation. So I would like to be sure of the understanding I have of the FULLHOST_FROM macro. If use_dns() is set to yes, does the expansion of FULLHOST_FROM macro follow this stream ? 1. Syslog-NG takes the IP address of the host sending the message 2. Syslog-NG tries to get the reverse value (as dig -x) from the IP address 3. Syslog-NG expands the macro with the FQDN obtained in point 2 (What if DNS reverse fails ? The macro returns only the IP address ?) Bye Christophe -- Christophe Brocas keyid : 3072R/0x0661CBBA ***************************************************** "Le contenu de ce courriel et ses eventuelles pièces jointes sont confidentiels. Ils s'adressent exclusivement à la personne destinataire. Si cet envoi ne vous est pas destiné, ou si vous l'avez reçu par erreur, et afin de ne pas violer le secret des correspondances, vous ne devez pas le transmettre à d'autres personnes ni le reproduire. Merci de le renvoyer à l'émetteur et de le détruire. Attention : L'Organisme de l'émetteur du message ne pourra être tenu responsable de l'altération du présent courriel. Il appartient au destinataire de vérifier que les messages et pièces jointes reçus ne contiennent pas de virus. Les opinions contenues dans ce courriel et ses éventuelles pièces jointes sont celles de l'émetteur. Elles ne reflètent pas la position de l'Organisme sauf s'il en est disposé autrement dans le présent courriel." ******************************************************

2 3

[Bug 97] New: syslog-ng crashes after SIGHUP
by bugzilla＠bugzilla.balabit.com 04 Dec '10

04 Dec '10

https://bugzilla.balabit.com/show_bug.cgi?id=97 Summary: syslog-ng crashes after SIGHUP Product: syslog-ng Version: 3.0.x Platform: PC OS/Version: Linux Status: NEW Severity: normal Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: zhengxiang.pan(a)alcatel-lucent.com Type of the Report: bug Estimated Hours: 0.0 Syslog-ng 3.0.6 crashes with following trace: Program terminated with signal 11, Segmentation fault. [New process 6311] #0 0x08063558 in log_source_group_queue (s=0x8ca6e00, msg=0x8caecc0, path_options=0xbfffaf18) at sgroup.c:98 98 (*self->processed_messages)++; (gdb) bt #0 0x08063558 in log_source_group_queue (s=0x8ca6e00, msg=0x8caecc0, path_options=0xbfffaf18) at sgroup.c:98 #1 0x08070580 in log_source_queue (s=0x8cadc78, msg=0x8caecc0, path_options=0xbfffaf74) at logpipe.h:121 #2 0x08063e80 in log_reader_fd_dispatch (source=0x8cadd10, callback=0, user_data=0x0) at logpipe.h:121 #3 0x08080081 in g_main_context_dispatch () #4 0x08083103 in g_main_context_iterate () #5 0x0808327f in g_main_context_iteration () #6 0x0804b52f in main_loop_run (cfg=0xbfffb0e8) at main.c:163 #7 0x0804bbcd in main (argc=147613904, argv=0xbfffb110) at main.c:443 The problem happens from version 3.0.6 ( I did not test other version lower than that) until the lastest 3.1.3. I guess what happens is that when SIGHUP is recevied, some connections are persisted, while the persistent connections points to the already-freed pipe (via pipe_next), if there are messages coming before the freed-pipes are replaced, the messages are forwarded to the freed-pipe handler. For version 3.1.3, in logpipe.c, log_pipe_forward_msg() may add check the pipe_next flags, however this may be not enough. log_pipe_forward_msg(LogPipe *self, LogMessage *msg, const LogPathOptions *path_options) { if (self->pipe_next && self->flags&PIF_INITIALIZED) { log_pipe_queue(self->pipe_next, msg, path_options); -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 5