- syslog-ng - lists.balabit.hu

need help debugging some network received logs that aren't writing to files
by Chris Moody 06 Mar '14

06 Mar '14

Hello. First off, thanks a __TON__ for syslog-ng. I've sworn by this awesome code for years now. I've built all sorts of logging infrastructure with it. I seem to have hit on something though that's got me scratching my head and lacking for explanation. Perhaps I've just been staring at it and debugging it too long and am missing something obvious. I've got an installation with a couple thousand network devices logging successfully to output spools on our log aggretor. This is rockin' and works beautifully. I've got things configured whereby each network source logs to it's own individual spool file with the source-ip as the spool name. I'm running into a case though where I have a Cisco switch sending logs to my log aggregator but the log-server isn't writing the output to the device's spool file. It is working however for many many more devices just like this switch. I've confirmed via tcpdump that this log traffic does actually hit the box, but it never gets recorded into the log spool for that network device. Since the host is -super- busy receiving logs from other gear enterprise-wide, I have to treat it very gingerly, so can't enable too much debugging...but I'm really confused why the logs wouldn't show up in the log spool.. Here's some bits of the config that are relevant: ===== options { keep_hostname(yes); use_dns(no); use_fqdn(no); stats_freq(600); stats_level(2); # Allow large messages log_msg_size(65536); }; # ===================== # UDP Packet Source source s_udp { udp(); }; # ===================== # TCP Packet Source source s_tcp { tcp(ip(aaa.bbb.ccc.ddd) port(514) max-connections(50000)); }; # ===================== destination net_perhost { file("/data/log/per-host/$HOST" owner(root) group(nwadmin) perm(0775) ); }; # ===================== log { source(s_tcp); source(s_udp); destination(net_perhost); }; ===== I've checked around for perhaps a different spool name, thinking perhaps the data was getting recognized as something other than it's source-ip, but haven't seen anything. Any thoughts? Cheers, -Chris

8 15

[Bug 260] New: exit with code 134
by bugzilla＠bugzilla.balabit.com 06 Mar '14

06 Mar '14

https://bugzilla.balabit.com/show_bug.cgi?id=260 Summary: exit with code 134 Product: syslog-ng Version: 3.4.x Platform: PC OS/Version: Linux Status: NEW Severity: normal Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: erempel(a)uvic.ca Type of the Report: --- Estimated Hours: 0.0 I just did a reload on version 3.4.4 (with latest patches) and triggered 2013-11-05T21:24:33-08:00 host daemon.crit supervise/syslog-ng[25895]: Daemon exited due to a deadlock/signal/failure, restarting; exitcode='134' Is it possible that the return code of syslog-ng is somehow linked to the return code of a child program destination? core file sent to Balazs in private e-mail -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 11

syslog-ng incubator packaged for openSUSE and Fedora
by Peter Czanik 06 Mar '14

06 Mar '14

Hi, syslog-ng incubator had a new release this week. Highlights are: ** Feature: LuaJIT support ** Feature: monitor source ** Feature: Templating support for the Lua destination ** Feature: Lua destinations have easy access to the timestamps ** Feature: The Lua modules now have a deinit hook ** Feature: The Lua modules has access to logging functions ** Feature: The Lua destination can have global variables set ** Feature: graphite output template function ** Feature: An ElasticSearch SCL ** Bugfix: RSS destination does not require newer glib For details check the NEWS file at https://github.com/balabit/syslog-ng-incubator/blob/master/NEWS I did some syslog-ng incubator packaging for openSUSE and Fedora. Functionality is not tested, as I don't have ElasticSearch, graphite, Riemann or lua scripts at hand (yet). As it's untested, there is no guarantee, that it does not eat your server :-) On the other hand testing and feedback is very welcome! openSUSE: https://build.opensuse.org/project/show/home:czanik:syslog-ng35 Fedora: http://copr.fedoraproject.org/coprs/czanik/syslog-ng35/ You should add the repository to your package manager, so all necessary dependencies are automagically resolved. Have a nice weekend! Bye, -- Peter Czanik (CzP) <czanik(a)balabit.hu> BalaBit IT Security / syslog-ng upstream http://czanik.blogs.balabit.com/ https://twitter.com/PCzanik

3 3

patterndb
by Fabien Wernli 03 Mar '14

03 Mar '14

Hi, I'm currently organizing my patterndb files into something easily maintainable, and I ran into the following problem, best explained using an example: sshd and pam_unix. Some of sshd specific patterns reside in a specific ruleset, e.g. <ruleset name='sshd' id='A'> <patterns> <pattern>sshd</pattern </patterns> <rules> <rule> ... </rule> </rules> </ruleset> Some of sshd's messages come from pam_unix, e.g. modules/pam_unix/support.c, and are shared among other daemons: <ruleset name='pam_unix' id='B'> <patterns> <pattern>crond</pattern> <pattern>imap</pattern> <pattern>login</pattern> <pattern>pam</pattern> <pattern>pure-ftpd</pattern> <pattern>proftpd</pattern> <pattern>sshd</pattern> <pattern>su</pattern> <pattern>sudo</pattern> ... <patterns> <rules> <rule> ... </rule> </rules> </ruleset> The problem is, when a sshd message is parsed, even if it doesn't match any rules in ruleset 'A', ruleset 'B' isn't even considered for matching. So how do I properly organize my pdb tree, without duplicating rules in multiple source files? Thanks in advance

3 4

Error parsing source
by Jesus M Diaz 28 Feb '14

28 Feb '14

Hello, I have just came to 'syslog-ng' and I am having some troubles to setting it up. I am using the "syslog-ng.conf" provided by "slackbuilds.org", and I have add just few lines. Well, those lines make the daemon not to run. the error: ++++++++++++ Starting syslog-ng daemon: /usr/sbin/syslog-ng Error parsing source, source plugin s_router not found in /etc/syslog-ng/syslog-ng.conf at line 59, column 14: log { source{s_router}; destination{d_router}; }; ^^^^^^^^ syslog-ng documentation: http://www.balabit.com/support/documentation/?product=syslog-ng mailing list: https://lists.balabit.hu/mailman/listinfo/syslog-ng ******************* the release: ++++++++++++ root@liet:/etc/syslog-ng# syslog-ng -V syslog-ng 3.5.2 Installer-Version: 3.5.2 Revision: ssh+git://algernon@git.balabit/var/scm/git/syslog-ng/syslog-ng-ose--mainline--3.5#master#a31bdb7a7f57382a90305bd033753b2762469854 Compile-Date: Feb 28 2014 19:53:40 Available-Modules: afsocket-notls,afamqp,linux-kmsg-format,basicfuncs,affile,dbparser,afsocket-tls,csvparser,afsocket,afstomp,system-source,afmongodb,cryptofuncs,afprog,syslogformat,afuser,confgen Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-IPv6: on Enable-Spoof-Source: off Enable-TCP-Wrapper: off Enable-Linux-Caps: on Enable-Pcre: on ******************* And the config file (my lines remarked with <<<<<<): +++++++++++++++++ @version: 3.5 # Drop-in replacement for a stock Slackware syslog.conf # For info about the format of this file, see "man syslog-ng.conf" # Written by Mario Preksavec <mario(a)slackware.hr> options { flush_lines(0); time_reopen(60); log_fifo_size(10240); log_msg_size(8192); chain_hostnames(no); use_dns(no); use_fqdn(no); create_dirs(yes); keep_hostname(yes); owner("root"); group("root"); perm(0640); dir_perm(0755); stats_freq(0); check_hostname(yes); dns_cache(no); }; source s_router { udp(ip(192.168.1.1) port(514)); }; <<<<<<<<<<<<< source s_system { internal(); unix-dgram("/dev/log"); file("/proc/kmsg" program_override("kernel")); }; filter f_messages { level(info,notice) and not facility(authpriv,cron,mail,news); }; filter f_syslog { level(warn..emerg) and not facility(authpriv,cron,mail,news); }; filter f_debug { level(debug); }; filter f_authpriv { facility(authpriv); }; filter f_cron { facility(cron); }; filter f_mail { facility(mail); }; filter f_emerg { level(emerg); }; filter f_uucp { facility(uucp); }; destination d_messages { file("/var/log/messages"); }; destination d_syslog { file("/var/log/syslog"); }; destination d_debug { file("/var/log/debug"); }; destination d_secure { file("/var/log/secure"); }; destination d_cron { file("/var/log/cron"); }; destination d_maillog { file("/var/log/maillog"); }; destination d_usertty { usertty("*"); }; destination d_spooler { file("/var/log/spooler"); }; destination d_router { file("/var/log/router"); }; <<<<<<<<<<<<< ############ ## Router ########################### log { source{s_router}; destination{d_router}; }; <<<<<<<<<<<<< # Log anything 'info' or higher, but lower than 'warn'. # Exclude authpriv, cron, mail, and news. These are logged elsewhere. log { source(s_system); filter(f_messages); destination(d_messages); }; # Log anything 'warn' or higher. # Exclude authpriv, cron, mail, and news. These are logged elsewhere. log { source(s_system); filter(f_syslog); destination(d_syslog); }; # Debugging information is logged here. log { source(s_system); filter(f_debug); destination(d_debug); }; # Private authentication message logging: log { source(s_system); filter(f_authpriv); destination(d_secure); }; # Cron related logs: log { source(s_system); filter(f_cron); destination(d_cron); }; # Mail related logs: log { source(s_system); filter(f_mail); destination(d_maillog); }; # Emergency level messages go to all users: log { source(s_system); filter(f_emerg); destination(d_usertty); }; # This log is for news and uucp errors: log { source(s_system); filter(f_uucp); destination(d_spooler); }; # Uncomment this to see kernel messages on the console. #filter f_kern { facility(kern); }; #destination d_console { file("/dev/console"); }; #log { source(s_system); filter(f_kern); destination(d_console); }; # Uncomment these if you'd like INN to keep logs on everything. # You won't need this if you don't run INN (the InterNetNews daemon). #filter f_news_crit { facility(news) and level(crit); }; #filter f_news_err { facility(news) and level(err); }; #filter f_news_notice { facility(news) and level(notice); }; #destination d_news_crit { file("/var/log/news/news.crit"); }; #destination d_news_err { file("/var/log/news/news.err"); }; #destination d_news_notice { file("/var/log/news/news.notice"); }; #log { source(s_system); filter(f_news_crit); destination(d_news_crit); }; #log { source(s_system); filter(f_news_err); destination(d_news_err); }; #log { source(s_system); filter(f_news_notice); destination(f_news_notice); }; *************************** If I comment my "log" line, it works fine, but if I use it, it complains about the "source" entry. Surely it is a silly mistake, but I can't see it, so if any of you could help me I really would be very gratefull thanks a lot! JM Diaz

2 2

GSoC 2014 - Configuration visualisation project
by Sandeep K Chaudhary 28 Feb '14

28 Feb '14

Hi guys, I am interested in working on the project 'Configuration visualisation'. The mentor for this project is Viktor Tusa but I don't know how to reach Viktor directly so I am emailing to this list. Would reaching out to the IRC channel be better? Please let me know. Thanks, Sandeep.

2 2

Escaping open bracket and close bracket.
by C. L. Martinez 27 Feb '14

27 Feb '14

2 2

Porposal (Rough Draft): Type Hinting from the start
by Rajul Srivastava 26 Feb '14

26 Feb '14

Hi all, I have come up with a very rough draft of a proposal for the project idea "Type Hinting from the start" which I have upload here: https://drive.google.com/file/d/0B2mFOuJDrIXmT1AzbngtUXp3eVk/edit?usp=shari… This is the preliminary draft, where I have tried to just list the background and a brief write-up about what I can understand about the project idea. I request you all to kindly take a look at it and I shall be very grateful to anyone who can give me suggestions about how may I go about the design and implementation details about this project. I have not included the timeline for the project, which I plan to do after I have finalized the design and implementation details in the proposal. I shall really be grateful for any suggestions and feedback. Thanking You, Kind regards, Rajul

2 1

[Bug 271] New: using json parser twice in two unnested log statements causes segfault
by bugzilla＠bugzilla.balabit.com 25 Feb '14

25 Feb '14

https://bugzilla.balabit.com/show_bug.cgi?id=271 Summary: using json parser twice in two unnested log statements causes segfault Product: syslog-ng Version: 3.5.x Platform: PC OS/Version: Linux Status: NEW Severity: normal Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: bugzilla.balabit(a)faxm0dem.org Type of the Report: --- Estimated Hours: 0.0 Created an attachment (id=91) --> (https://bugzilla.balabit.com/attachment.cgi?id=91) minimal configuration to trigger segfault The attached configuration causes a segfault on startup (before any messages make it to source). Here's a stack trace: -- (gdb) bt full #0 0x00007fd54fe6e86f in __strlen_sse42 () from /lib64/libc.so.6 No symbol table info available. #1 0x00007fd54d2c9766 in log_json_parser_set_marker () from /lib64/syslog-ng/libjson-plugin.so No symbol table info available. #2 0x00007fd54d2c9e1b in log_json_parser_clone () from /lib64/syslog-ng/libjson-plugin.so No symbol table info available. #3 0x00007fd551884a1e in log_pipe_clone () from /lib64/libsyslog-ng-3.5.3.so No symbol table info available. #4 0x00007fd55188560c in cfg_tree_compile_single () from /lib64/libsyslog-ng-3.5.3.so No symbol table info available. #5 0x00007fd551886224 in cfg_tree_compile_node () from /lib64/libsyslog-ng-3.5.3.so No symbol table info available. #6 0x00007fd551885c02 in cfg_tree_compile_sequence () from /lib64/libsyslog-ng-3.5.3.so No symbol table info available. #7 0x00007fd551886267 in cfg_tree_compile_node () from /lib64/libsyslog-ng-3.5.3.so No symbol table info available. #8 0x00007fd551885a98 in cfg_tree_compile_reference () from /lib64/libsyslog-ng-3.5.3.so No symbol table info available. #9 0x00007fd551886247 in cfg_tree_compile_node () from /lib64/libsyslog-ng-3.5.3.so No symbol table info available. #10 0x00007fd551885c02 in cfg_tree_compile_sequence () from /lib64/libsyslog-ng-3.5.3.so No symbol table info available. #11 0x00007fd551886267 in cfg_tree_compile_node () from /lib64/libsyslog-ng-3.5.3.so No symbol table info available. #12 0x00007fd551886306 in cfg_tree_compile_rule () from /lib64/libsyslog-ng-3.5.3.so No symbol table info available. #13 0x00007fd551886639 in cfg_tree_compile () from /lib64/libsyslog-ng-3.5.3.so No symbol table info available. #14 0x00007fd55188667d in cfg_tree_start () from /lib64/libsyslog-ng-3.5.3.so No symbol table info available. #15 0x00007fd55188045d in cfg_init () from /lib64/libsyslog-ng-3.5.3.so No symbol table info available. #16 0x00007fd55189cc8a in main_loop_initialize_state () from /lib64/libsyslog-ng-3.5.3.so No symbol table info available. #17 0x00007fd55189d253 in main_loop_init () from /lib64/libsyslog-ng-3.5.3.so No symbol table info available. #18 0x0000000000401805 in main () No symbol table info available. -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 3

Introduction
by Rajul Srivastava 25 Feb '14

25 Feb '14

Hi all, My name in Rajul, and I am a final year undergraduate student at the Indian Institute of Technology Kharagpur. I wish to participate in Google Summer of Code 2014, and while going through the list of organisations, I came across The syslog-ng project". I am proficient with programming languages C/C++, Python, Java, Groovy, Ruby. I am very interested in the fields of Algorithms, Machine Learning, Complex Networks, Information Retrieval. I have always been interested in programming and in the past I have participated in Google Summer of Code 2012, with the organisation Network Time Foundation,working on the project "improving the Logging/Debugging System of Network Time Protocol Software". I have also interned in the Global Technology division of Barclays, during the summers of 2013, working with the Market Risk IT team. Besides I have worked on a few Research projects in the fields of Computational Finance, Complex Networks, and Computational Chemistry. I am currently working on my Thesis project in the field of Computational Sciences on a project titled "Network Analysis of Chemical Reactions". I have had courses in the fields of Programming and Data Structures, Complex Networks, Distributed Systems, Algorithms, Operations Research in the past. I have gone through the list of project ideas and I found most of the project ideas very interesting. Although I find all the projects listed worth a while, I am particularly interested in the "Configuration Visualization" and "Type Hinting from the start" projects. I suppose that my programming background is suitable for these projects. I shall be grateful if anyone can help me and give me reference to the literature that I may use and also shed some light on how I can go about making a successful proposal. Thanks!! Best Regards, Rajul -- Rajul Final year Undergraduate Student, Indian Institute of Technology Kharagpur

1 0