June 2010 - syslog-ng - lists.balabit.hu

[Bug 42] New: capabilities, chown, chmod
by bugzilla＠bugzilla.balabit.com 20 Dec '12

20 Dec '12

https://bugzilla.balabit.com/show_bug.cgi?id=42 Summary: capabilities, chown, chmod Product: syslog-ng Version: 3.0.x Platform: PC OS/Version: Linux Status: NEW Severity: normal Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: zbyniu(a)pld-linux.org Type of the Report: bug Estimated Hours: 0.0 Let's take a look at syslog-ng-3.0.1/src/affile.c lines 60-83 1. CAP_SYS_ADMIN is needed only for /proc/kmsg, it is added w/o check 2. CAP_DAC_READ_SEARCH should be added only if open fail with errno 13 2a. CAP_DAC_OVERRIDE should be added only if open fail with errno 13 and with CAP_DAC_READ_SEARCH set 3. fchown needs CAP_CHOWN unconditionaly 4. fchmod needs CAP_FOWNER if file owner != euid (root here) 5. all caps should be restored summary: - CAP_SYS_ADMIN and CAP_DAC_OVERRIDE are set always even if unnecessary, and permanently - owner, group and perm doesn't work -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

2 10

[Bug 14] New: Maintain symlinks to current and previous log output files...
by bugzilla＠bugzilla.balabit.com 28 Sep '11

28 Sep '11

https://bugzilla.balabit.com/show_bug.cgi?id=14 Summary: Maintain symlinks to current and previous log output files... Product: syslog-ng Version: 2.0.x Platform: PC URL: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=218688 OS/Version: Linux Status: NEW Severity: minor Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: sasa(a)debian.org Type of the Report: enhancement Estimated Hours: 0.0 >From bugs.debian.org: I'm hoping that upstream might consider adding functionality to syslog-ng to allow it to track current and previous log files and maintain symlinks to them, as a user option. This replicates one of the options that I (and others) find useful in cronolog. This has been previously discussed on the syslog-ng mailing list, though apparently nothing fully came of it. The archive of that discussion is at: https://lists.balabit.hu/pipermail/syslog-ng/2002-June/003452.html Obviously, it's probably somewhat of a low priority, but it would be nice. -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 1

filter/parse based on certificate attributes?
by Christian Tramnitz 22 Feb '11

22 Feb '11

With syslog 3.0 and ssl support using mutual certificate authentication is it possible to parse/filter/tag based on attributes in the certificate? I'd like to have a common listener and then relay or append messages based on client-cert attribute values... (Sorry if this message is arriving more than once, I seem to have problems with posting through Gmane.) Regards, Christian

4 4

[Bug 87] New: csv-parser: specifying drop-invalid and greedy flags always fails.
by bugzilla＠bugzilla.balabit.com 14 Aug '10

14 Aug '10

https://bugzilla.balabit.com/show_bug.cgi?id=87 Summary: csv-parser: specifying drop-invalid and greedy flags always fails. Product: syslog-ng Version: 3.0.x Platform: PC OS/Version: Linux Status: NEW Severity: normal Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: alex-pub.balabit(a)reflexion.net Type of the Report: bug Estimated Hours: 0.0 In this example, I am trying to parse the postfix log statement. Postfix specifies $PROGRAM as a pair separated by slash, e.g "postfix/master" or "postfix/postfix-script". The "postfix" part is configurable, and I would like to parse it out from the $PROGRAM macro. I tried it first with this parser parser p_rfx_postfix_program_parser { csv-parser( columns( "POSTFIX.SERVICE_NAME", "POSTFIX.PROGRAM" ) delimiters( "/" ) flags( drop-invalid, escape-none, greedy ) template( "${PROGRAM}" ) ); }; The intention here is that ${PROGRAM} should always contain at least 1 forward slash, and I only want to parse the part before the first slash. However this parser always fails. Removing greedy flag makes this parser succeed, however it does change the meaning of the action that I am trying to perform. Looking at the logparser.c: log_csv_parser_process, I've noticed that when LOG_CSV_PARSER_GREEDY flag is on, then `cur_column` variable is reset to NULL. Then right before the function ends it checks if `cur_column` is null and LOG_CSV_PARSER_DROP_INVALID flag is on, then fails the function. I think the check at the end of the function should take into account whether GREEDY operation has been performed or not. -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 5

[Bug 84] New: Program destination blocks if program stops reading input
by bugzilla＠bugzilla.balabit.com 02 Aug '10

02 Aug '10

https://bugzilla.balabit.com/show_bug.cgi?id=84 Summary: Program destination blocks if program stops reading input Product: syslog-ng Version: 3.0.x Platform: PC OS/Version: Linux Status: NEW Severity: major Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: erempel(a)uvic.ca Type of the Report: --- Estimated Hours: 0.0 Syslog-ng OSE 3.0.5 and 3.0.6 - binary download from balabit and self compiled all behave the same. If the program in a destination stops reading its input, syslog-ng buffers a small number of messages and then stops reading the source (in my case a pipe). I would expect syslog-ng to read from its input until the log_fifo_size limit was reached and then drop messages. an strace is attached. The configuration file is -------------- options { log_fifo_size(1000000); use_fqdn(yes); keep_hostname(yes); chain_hostnames(yes); time_reap(60); time_reopen(5); flush_lines(1000); flush_timeout(1000); }; template t_application { template("$MESSAGE\n"); template_escape(no); }; template t_localsyslog { template("<$PRI>$S_ISODATE $HOST erempel: $MSGONLY\n"); template_escape(no); }; source local { pipe("/var/log/syslog.pipes/erempel" flags(no-parse) ); }; source int { internal(); }; destination localsyslog { tcp("localhost" port(1514) localip(localhost) template(t_localsyslog) log_fifo_size(5000) ); }; destination erempel { program("/home1l/erempel/read-noread.pl" template(t_application) log_fifo_size(8000000) ); }; log { source(local); destination(erempel); }; log { source(int); destination(localsyslog); }; ---------------------- -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 8

monitoring SQL-Server-ERRORLOG
by Schöppel Marcus 14 Jul '10

14 Jul '10

Hi all@syslog-ng-mailing-list! I'm trying to monitor the MSSQLServer ERROR-logfile but I get a logentry for every single letter that is added to the logfile: -> May 18 13:57:40 test mssql: L May 18 13:57:40 test mssql: o May 18 13:57:40 test mssql: g May 18 13:57:40 test mssql: i May 18 13:57:40 test mssql: n May 18 13:57:40 test mssql: May 18 13:57:40 test mssql: f May 18 13:57:40 test mssql: a May 18 13:57:40 test mssql: i May 18 13:57:40 test mssql: l May 18 13:57:40 test mssql: e May 18 13:57:40 test mssql: d <- This seems to stem from the encoding of the file (UTF-16LE) because with other logfiles this problem doesn't arise. I suppose it would help if the "encoding"-Parameter of the file-source would work but syslog-ng/cygwin (Version 3.0.1) won't start when I add it no matter which encoding I use. I tried this workaround-cronjob: - copy file somewhere - convert to UTF-8 using iconv - give the same timestamp to the converted logfile so that syslog-ng doesn't transfer the whole file again - monitoring the converted file with syslog-ng This doesn't work completely correct as despite assigning the same timestamp the whole file is sometimes transferred again (and not just the new entries). Is this problem (following this UTF-16LE-Windowsfile (easily)) not resolvable by using the cygwin-version of syslog-ng?? Or am I maybe using a wrong encoding spelling? Does it work using the commercial windows-version of the agent?

2 1

[announce] patterndb project
by Balazs Scheidler 13 Jul '10

13 Jul '10

Hi, By now probably most of you know about patterndb, a powerful framework in syslog-ng that lets you extract structured information from log messages and perform classification at a high speed: http://www.balabit.com/dl/html/syslog-ng-ose-v3.1-guide-admin-en.html/conce… Until now, syslog-ng offered the feature, but no release-quality patterns were produced by the syslog-ng developers. Some samples based on the logcheck database were created, but otherwise every syslog-ng user had to create her samples manually, possibly repeating work performed by others. Since this calls out to be a community project, I'm hereby starting one. Goals ===== Create release-quality pattern databases that can simply be deployed to an existing syslog-ng installation. The goal of the patterns is to extract structured information from the free-form syslog messages, e.g. create name-value pairs based on the syslog message. Since the key factor when doing something like this is the naming of fields, we're going to create our generic naming guidelines that can be applied to any application in the industry. It is not our goal to implement correllation or any other advanced form of analysis, although we feel that with the results of this project, event correllation and analysis can be performed much easier than without it. Related projects ================ I know there are other efforts in the field, why not simply join them? CEF - is the log message format for a proprietary log analysis engine, primarily meant to be used to hold IP security device logs (firewalls, IPSs, virus gateways etc). The patterndb project aims to create patterns for a wider range of device logs and be more generic in the approach. On the other hand we feel that it might be useful to create a solution for converting db-parser output to the CEF format. CEE - Common Event Expression project by Mitre has a focus on creating a nv pair dictionary for all kinds of devices/log messages out there. Although I might be missing something, but I didn't find the concrete results so far, apart from a nicely looking white paper. If the CEE delivers something, then patterndb would probably adapt the naming/taxonomy structure. But I guess not all devices will start logging in the new shiny format, thus the existing devices would need their logs converted, so the patterndb work wouldn't be wasted. Infrastructure ============== Our original patterndb related plans were to create an easy to use web based interface for editing patterns, but since that project is progressing slowly, I'm calling for a minimalist approach: git based version control of simple plain text files. Of course once the nice web based interface is finished, we're going to be ready to use it. First steps =========== I have created a git repository at: http://git.balabit.hu/bazsi/syslog-ng-patterndb.git This contains the initial version of the naming policy document and a simple schema for SIEM-style and a user login-logout naming schema. If you are interested please read the file README.txt in the git archive, or if you prefer a web browser, use this link: http://git.balabit.hu/?p=bazsi/syslog-ng-patterndb.git;a=blob;f=README.txt;… Licensing ========= I do not have a decision yet, but for sure this is going to use one of the open source licenses or Creative Commons. Let me know if you have a preference in this area. Getting involved ================ Join the syslog-ng mailing list, a start discussing! If you have existing patterns, great. If you don't, it is not late to join. http://lists.balabit.hu/mailman/listinfo/syslog-ng -- Bazsi

3 20

[Bug 90] New: csv-parser's flags(drop-invalid) doesn' t work with escape-{backslash, double-char}
by bugzilla＠bugzilla.balabit.com 09 Jul '10

09 Jul '10

https://bugzilla.balabit.com/show_bug.cgi?id=90 Summary: csv-parser's flags(drop-invalid) doesn't work with escape-{backslash,double-char} Product: syslog-ng Version: 3.0.x Platform: PC OS/Version: Linux Status: NEW Severity: normal Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: andrewn(a)locus.net Type of the Report: bug Estimated Hours: 0.0 Created an attachment (id=19) --> (https://bugzilla.balabit.com/attachment.cgi?id=19) Possible fix The example parser p_apache in syslog-ng-v3.0-guide-admin-en.pdf on numbered page 63 does not work when drop-invalid is added. The 2 main loops of log_csv_parser_process seem to treat cur_column and src differently upon termination. The patch attached attempts to fix that. It also checks to see that the whole string was parsed. I don't know if this is the correct fix, but so far it seems to Work For Me (tm). -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 5

[Bug 86] New: flush_timeout has no effect
by bugzilla＠bugzilla.balabit.com 09 Jul '10

09 Jul '10

https://bugzilla.balabit.com/show_bug.cgi?id=86 Summary: flush_timeout has no effect Product: syslog-ng Version: 3.0.x Platform: PC OS/Version: Linux Status: NEW Severity: major Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: erempel(a)uvic.ca Type of the Report: --- Estimated Hours: 0.0 After upgrading to version 3.0.7 OSE the flush_timeout option has no effect for file destinations. The messages are buffered until flush_lines is reached and then they are flushed. TCP destinations are no affected by this issue. -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 7

[Bug 88] New: rewrite FACILITY does not work.
by bugzilla＠bugzilla.balabit.com 01 Jul '10

01 Jul '10

https://bugzilla.balabit.com/show_bug.cgi?id=88 Summary: rewrite FACILITY does not work. Product: syslog-ng Version: 3.0.x Platform: PC OS/Version: Windows Status: NEW Severity: normal Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: alex-pub.balabit(a)reflexion.net Type of the Report: --- Estimated Hours: 0.0 Here is the fragment of my syslog-ng.conf file that demonstrates the problem source remote_sys { tcp( ip( "0.0.0.0" ) port( 514 ) max-connections( 1152 ) ); udp( ip( "0.0.0.0" ) port( 514 ) ); }; destination d_remote_consolidated_log { file( "/var/log/remote/$FACILITY/$FACILITY.log" create_dirs( yes ) perm( 0644 ) dir_perm( 0755 ) template( "$R_ISODATE $HOST -$LEVEL- $PROGRAM[$PID]: $MSGONLY\n" ) frac_digits( 6 ) template_escape( no ) ); }; filter f_apache { facility( local1 ); }; rewrite r_rewrite_apache_facility { set("apache" value("FACILITY")); }; log { source( remote_sys ); filter( f_apache ); rewrite( r_rewrite_apache_facility ); destination( d_remote_consolidated_log ); flags( final ); }; If I issue the following command from the host connected to this syslog-ng server: $ echo this is apache log test | logger -t apache-test -p local1.info Then the output is sent to "/var/log/remote/local1/local1.log" instead of expected "/var/log/remote/apache/apache.log". Also, if I include $FACILITY into the destination template it prints "local1" If I run syslog-ng with -d flag I see this in output: >>Rewrite expression evaluation result; value='FACILITY', new_value='apache' Below is a full log for this test >>Incoming log entry; line='<142>Jun 22 17:36:11 host-1 apache-test: this is a test 2' >>Filter rule evaluation begins; filter_rule='f_filter1' >>Filter node evaluation result; filter_result='not-match', filter_type='facility' >>Filter rule evaluation result; filter_result='not-match', filter_rule='f_filter1' >>Filter rule evaluation begins; filter_rule='f_filter2' >>Filter node evaluation result; filter_result='not-match', filter_type='facility' >>Filter rule evaluation result; filter_result='not-match', filter_rule='f_filter2' >>Filter rule evaluation begins; filter_rule='f_apache' >>Filter node evaluation result; filter_result='match', filter_type='facility' >>Filter rule evaluation result; filter_result='match', filter_rule='f_apache' >>Rewrite expression evaluation result; value='FACILITY', new_value='apache' So this makes me believe that rewrite does happen, but the value is not picked up later. BTW, the documentation for `set` command says that there should be a comma between "apache" and value, but other samples do not put it there. I've tried it both ways to no avail. -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 9