February 2014 - syslog-ng - lists.balabit.hu

patterndb
by Fabien Wernli 03 Mar '14

03 Mar '14

Hi, I'm currently organizing my patterndb files into something easily maintainable, and I ran into the following problem, best explained using an example: sshd and pam_unix. Some of sshd specific patterns reside in a specific ruleset, e.g. <ruleset name='sshd' id='A'> <patterns> <pattern>sshd</pattern </patterns> <rules> <rule> ... </rule> </rules> </ruleset> Some of sshd's messages come from pam_unix, e.g. modules/pam_unix/support.c, and are shared among other daemons: <ruleset name='pam_unix' id='B'> <patterns> <pattern>crond</pattern> <pattern>imap</pattern> <pattern>login</pattern> <pattern>pam</pattern> <pattern>pure-ftpd</pattern> <pattern>proftpd</pattern> <pattern>sshd</pattern> <pattern>su</pattern> <pattern>sudo</pattern> ... <patterns> <rules> <rule> ... </rule> </rules> </ruleset> The problem is, when a sshd message is parsed, even if it doesn't match any rules in ruleset 'A', ruleset 'B' isn't even considered for matching. So how do I properly organize my pdb tree, without duplicating rules in multiple source files? Thanks in advance

3 4

Error parsing source
by Jesus M Diaz 28 Feb '14

28 Feb '14

Hello, I have just came to 'syslog-ng' and I am having some troubles to setting it up. I am using the "syslog-ng.conf" provided by "slackbuilds.org", and I have add just few lines. Well, those lines make the daemon not to run. the error: ++++++++++++ Starting syslog-ng daemon: /usr/sbin/syslog-ng Error parsing source, source plugin s_router not found in /etc/syslog-ng/syslog-ng.conf at line 59, column 14: log { source{s_router}; destination{d_router}; }; ^^^^^^^^ syslog-ng documentation: http://www.balabit.com/support/documentation/?product=syslog-ng mailing list: https://lists.balabit.hu/mailman/listinfo/syslog-ng ******************* the release: ++++++++++++ root@liet:/etc/syslog-ng# syslog-ng -V syslog-ng 3.5.2 Installer-Version: 3.5.2 Revision: ssh+git://algernon@git.balabit/var/scm/git/syslog-ng/syslog-ng-ose--mainline--3.5#master#a31bdb7a7f57382a90305bd033753b2762469854 Compile-Date: Feb 28 2014 19:53:40 Available-Modules: afsocket-notls,afamqp,linux-kmsg-format,basicfuncs,affile,dbparser,afsocket-tls,csvparser,afsocket,afstomp,system-source,afmongodb,cryptofuncs,afprog,syslogformat,afuser,confgen Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-IPv6: on Enable-Spoof-Source: off Enable-TCP-Wrapper: off Enable-Linux-Caps: on Enable-Pcre: on ******************* And the config file (my lines remarked with <<<<<<): +++++++++++++++++ @version: 3.5 # Drop-in replacement for a stock Slackware syslog.conf # For info about the format of this file, see "man syslog-ng.conf" # Written by Mario Preksavec <mario(a)slackware.hr> options { flush_lines(0); time_reopen(60); log_fifo_size(10240); log_msg_size(8192); chain_hostnames(no); use_dns(no); use_fqdn(no); create_dirs(yes); keep_hostname(yes); owner("root"); group("root"); perm(0640); dir_perm(0755); stats_freq(0); check_hostname(yes); dns_cache(no); }; source s_router { udp(ip(192.168.1.1) port(514)); }; <<<<<<<<<<<<< source s_system { internal(); unix-dgram("/dev/log"); file("/proc/kmsg" program_override("kernel")); }; filter f_messages { level(info,notice) and not facility(authpriv,cron,mail,news); }; filter f_syslog { level(warn..emerg) and not facility(authpriv,cron,mail,news); }; filter f_debug { level(debug); }; filter f_authpriv { facility(authpriv); }; filter f_cron { facility(cron); }; filter f_mail { facility(mail); }; filter f_emerg { level(emerg); }; filter f_uucp { facility(uucp); }; destination d_messages { file("/var/log/messages"); }; destination d_syslog { file("/var/log/syslog"); }; destination d_debug { file("/var/log/debug"); }; destination d_secure { file("/var/log/secure"); }; destination d_cron { file("/var/log/cron"); }; destination d_maillog { file("/var/log/maillog"); }; destination d_usertty { usertty("*"); }; destination d_spooler { file("/var/log/spooler"); }; destination d_router { file("/var/log/router"); }; <<<<<<<<<<<<< ############ ## Router ########################### log { source{s_router}; destination{d_router}; }; <<<<<<<<<<<<< # Log anything 'info' or higher, but lower than 'warn'. # Exclude authpriv, cron, mail, and news. These are logged elsewhere. log { source(s_system); filter(f_messages); destination(d_messages); }; # Log anything 'warn' or higher. # Exclude authpriv, cron, mail, and news. These are logged elsewhere. log { source(s_system); filter(f_syslog); destination(d_syslog); }; # Debugging information is logged here. log { source(s_system); filter(f_debug); destination(d_debug); }; # Private authentication message logging: log { source(s_system); filter(f_authpriv); destination(d_secure); }; # Cron related logs: log { source(s_system); filter(f_cron); destination(d_cron); }; # Mail related logs: log { source(s_system); filter(f_mail); destination(d_maillog); }; # Emergency level messages go to all users: log { source(s_system); filter(f_emerg); destination(d_usertty); }; # This log is for news and uucp errors: log { source(s_system); filter(f_uucp); destination(d_spooler); }; # Uncomment this to see kernel messages on the console. #filter f_kern { facility(kern); }; #destination d_console { file("/dev/console"); }; #log { source(s_system); filter(f_kern); destination(d_console); }; # Uncomment these if you'd like INN to keep logs on everything. # You won't need this if you don't run INN (the InterNetNews daemon). #filter f_news_crit { facility(news) and level(crit); }; #filter f_news_err { facility(news) and level(err); }; #filter f_news_notice { facility(news) and level(notice); }; #destination d_news_crit { file("/var/log/news/news.crit"); }; #destination d_news_err { file("/var/log/news/news.err"); }; #destination d_news_notice { file("/var/log/news/news.notice"); }; #log { source(s_system); filter(f_news_crit); destination(d_news_crit); }; #log { source(s_system); filter(f_news_err); destination(d_news_err); }; #log { source(s_system); filter(f_news_notice); destination(f_news_notice); }; *************************** If I comment my "log" line, it works fine, but if I use it, it complains about the "source" entry. Surely it is a silly mistake, but I can't see it, so if any of you could help me I really would be very gratefull thanks a lot! JM Diaz

2 2

GSoC 2014 - Configuration visualisation project
by Sandeep K Chaudhary 28 Feb '14

28 Feb '14

Hi guys, I am interested in working on the project 'Configuration visualisation'. The mentor for this project is Viktor Tusa but I don't know how to reach Viktor directly so I am emailing to this list. Would reaching out to the IRC channel be better? Please let me know. Thanks, Sandeep.

2 2

Escaping open bracket and close bracket.
by C. L. Martinez 27 Feb '14

27 Feb '14

2 2

Porposal (Rough Draft): Type Hinting from the start
by Rajul Srivastava 26 Feb '14

26 Feb '14

Hi all, I have come up with a very rough draft of a proposal for the project idea "Type Hinting from the start" which I have upload here: https://drive.google.com/file/d/0B2mFOuJDrIXmT1AzbngtUXp3eVk/edit?usp=shari… This is the preliminary draft, where I have tried to just list the background and a brief write-up about what I can understand about the project idea. I request you all to kindly take a look at it and I shall be very grateful to anyone who can give me suggestions about how may I go about the design and implementation details about this project. I have not included the timeline for the project, which I plan to do after I have finalized the design and implementation details in the proposal. I shall really be grateful for any suggestions and feedback. Thanking You, Kind regards, Rajul

2 1

[Bug 271] New: using json parser twice in two unnested log statements causes segfault
by bugzilla＠bugzilla.balabit.com 25 Feb '14

25 Feb '14

https://bugzilla.balabit.com/show_bug.cgi?id=271 Summary: using json parser twice in two unnested log statements causes segfault Product: syslog-ng Version: 3.5.x Platform: PC OS/Version: Linux Status: NEW Severity: normal Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: bugzilla.balabit(a)faxm0dem.org Type of the Report: --- Estimated Hours: 0.0 Created an attachment (id=91) --> (https://bugzilla.balabit.com/attachment.cgi?id=91) minimal configuration to trigger segfault The attached configuration causes a segfault on startup (before any messages make it to source). Here's a stack trace: -- (gdb) bt full #0 0x00007fd54fe6e86f in __strlen_sse42 () from /lib64/libc.so.6 No symbol table info available. #1 0x00007fd54d2c9766 in log_json_parser_set_marker () from /lib64/syslog-ng/libjson-plugin.so No symbol table info available. #2 0x00007fd54d2c9e1b in log_json_parser_clone () from /lib64/syslog-ng/libjson-plugin.so No symbol table info available. #3 0x00007fd551884a1e in log_pipe_clone () from /lib64/libsyslog-ng-3.5.3.so No symbol table info available. #4 0x00007fd55188560c in cfg_tree_compile_single () from /lib64/libsyslog-ng-3.5.3.so No symbol table info available. #5 0x00007fd551886224 in cfg_tree_compile_node () from /lib64/libsyslog-ng-3.5.3.so No symbol table info available. #6 0x00007fd551885c02 in cfg_tree_compile_sequence () from /lib64/libsyslog-ng-3.5.3.so No symbol table info available. #7 0x00007fd551886267 in cfg_tree_compile_node () from /lib64/libsyslog-ng-3.5.3.so No symbol table info available. #8 0x00007fd551885a98 in cfg_tree_compile_reference () from /lib64/libsyslog-ng-3.5.3.so No symbol table info available. #9 0x00007fd551886247 in cfg_tree_compile_node () from /lib64/libsyslog-ng-3.5.3.so No symbol table info available. #10 0x00007fd551885c02 in cfg_tree_compile_sequence () from /lib64/libsyslog-ng-3.5.3.so No symbol table info available. #11 0x00007fd551886267 in cfg_tree_compile_node () from /lib64/libsyslog-ng-3.5.3.so No symbol table info available. #12 0x00007fd551886306 in cfg_tree_compile_rule () from /lib64/libsyslog-ng-3.5.3.so No symbol table info available. #13 0x00007fd551886639 in cfg_tree_compile () from /lib64/libsyslog-ng-3.5.3.so No symbol table info available. #14 0x00007fd55188667d in cfg_tree_start () from /lib64/libsyslog-ng-3.5.3.so No symbol table info available. #15 0x00007fd55188045d in cfg_init () from /lib64/libsyslog-ng-3.5.3.so No symbol table info available. #16 0x00007fd55189cc8a in main_loop_initialize_state () from /lib64/libsyslog-ng-3.5.3.so No symbol table info available. #17 0x00007fd55189d253 in main_loop_init () from /lib64/libsyslog-ng-3.5.3.so No symbol table info available. #18 0x0000000000401805 in main () No symbol table info available. -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 3

Introduction
by Rajul Srivastava 25 Feb '14

25 Feb '14

Hi all, My name in Rajul, and I am a final year undergraduate student at the Indian Institute of Technology Kharagpur. I wish to participate in Google Summer of Code 2014, and while going through the list of organisations, I came across The syslog-ng project". I am proficient with programming languages C/C++, Python, Java, Groovy, Ruby. I am very interested in the fields of Algorithms, Machine Learning, Complex Networks, Information Retrieval. I have always been interested in programming and in the past I have participated in Google Summer of Code 2012, with the organisation Network Time Foundation,working on the project "improving the Logging/Debugging System of Network Time Protocol Software". I have also interned in the Global Technology division of Barclays, during the summers of 2013, working with the Market Risk IT team. Besides I have worked on a few Research projects in the fields of Computational Finance, Complex Networks, and Computational Chemistry. I am currently working on my Thesis project in the field of Computational Sciences on a project titled "Network Analysis of Chemical Reactions". I have had courses in the fields of Programming and Data Structures, Complex Networks, Distributed Systems, Algorithms, Operations Research in the past. I have gone through the list of project ideas and I found most of the project ideas very interesting. Although I find all the projects listed worth a while, I am particularly interested in the "Configuration Visualization" and "Type Hinting from the start" projects. I suppose that my programming background is suitable for these projects. I shall be grateful if anyone can help me and give me reference to the literature that I may use and also shed some light on how I can go about making a successful proposal. Thanks!! Best Regards, Rajul -- Rajul Final year Undergraduate Student, Indian Institute of Technology Kharagpur

1 0

Per-destination log_msg_size?
by David Schweikert 21 Feb '14

21 Feb '14

Hi, We have some setups where we want to have a large log_msg_size, but need to send some logs to an external party which only can handle standard BSD syslog messages with a maximum size of 1024 bytes. Question: would it be possible to add a per-destination log-msg-size to truncate the messages on a per-destination basis? I think that it would be very useful. Something like: destination d_remote { udp("192.168.65.10" port(514) log_msg_size(1024)); }; Do you find it a good idea and is there any chance that this might be implemented? Thanks David

5 7

Re: [syslog-ng] syslog-ng Digest, Vol 106, Issue 12
by Terry Slattery 18 Feb '14

18 Feb '14

Why not stand up a test server and add it to the logging configs of the non-working and a couple of working switches? It could be a VM, since it won't have much load. You could also capture the log messages and play them back at the test server without impacting your production server or data archives. -tcs On 2/18/14 3:12 PM, syslog-ng-request(a)lists.balabit.hu wrote: > Date: Tue, 18 Feb 2014 15:12:50 -0500 > From: Chris Moody<chris(a)node-nine.com> > Subject: [syslog-ng] need help debugging some network received logs > that aren't writing to files > To: Syslog-ng users' and developers' mailing list > <syslog-ng(a)lists.balabit.hu> > Message-ID:<5303BEC2.9010405(a)node-nine.com> > Content-Type: text/plain; charset=ISO-8859-1; format=flowed > > Hello. > > First off, thanks a __TON__ for syslog-ng. I've sworn by this awesome > code for years now. I've built all sorts of logging infrastructure with > it. > > I seem to have hit on something though that's got me scratching my head > and lacking for explanation. Perhaps I've just been staring at it and > debugging it too long and am missing something obvious. -- Terry Slattery CCIE# 1026

1 0

log path with two destinations - flow control problem
by Tomáš Novosad 18 Feb '14

18 Feb '14

Hello to all i'm trying to set syslog to store sendmaill messages into PostgreSQL tables. One simple table (tableA) which stores all messages, and another table (tableB) which stores only some relevant iformation. All messages are parsed in my parser, then logged into tableA, then checked against the filter, an those messages which pass are logged into tableB. it all looks like this: log { source(pipe_load); filter(f_sendmail); #-- basic filter for sendmail program parser(maillog); #-- parse message details destination(maillog_base); #-- store message in tableA - basic info only filter(f_sendmail_from_or_to); #-- filter only messages which have "from" or "to" address defined destination(maillog_detail); #-- strore the deatiled info into tableB flags(flow-control); } filterinhg itself works fine, but i have problem with flow-control Messages are send from file (old archived maillog) into syslog pipe source. Problemm is, that alhough flow-control is enabled, messages get droped on the maillog_base destination. Im doing "cat old_maillog > syslog-pile" where old_maillog has approx 1 800 000 records, but only about 400 000 records gets logged into tableA. Flow control does not seem to work in this scenario. In case when i comment-out lines with filter(f_sendmail_from_or_to) and destination(maillog_detail) effectively disabling logging into the 2nd table, all messages from input file are processed and stored into tableA. The flow control works fine in this case. Is there something wrong with such filtering and storing only some messages in the 2nd destination? I also thought about having two different log path for each table, but in that case, the parsing of the message should be done twice, which i want to avoid. My question is, if is it OK to use filtering like in my example, - filter - than destination - then filter again - then another destination + flow control Many thanks in advance -- Tomáš Novosad LinuxBox.cz, s.r.o. 28. října 168, 709 00 Ostrava tel.: +420 591 166 221 mobil: +420 737 238 655 email: tomas.novosad(a)linuxbox.cz jabber: novosad(a)linuxbox.cz www.linuxbox.cz mobil servis: +420 737 238 656 email servis: servis(a)linuxbox.cz

3 3