- syslog-ng - lists.balabit.hu

Re: [syslog-ng] Integrating with snmp traps
by Jim Hendrick 09 Jun '15

09 Jun '15

Couldn't that be done using patterndb? Sent from my Verizon Wireless 4G LTE smartphone -------- Original message -------- From: Fabien Wernli <wernli(a)in2p3.fr> Date: 06/09/2015 10:29 AM (GMT-05:00) To: Syslog-ng users' and developers' mailing list <syslog-ng(a)lists.balabit.hu> Subject: Re: [syslog-ng] Integrating with snmp traps Hi, On Tue, Jun 09, 2015 at 06:26:40AM -0700, Evan Rempel wrote: > When you say you "would actually like to parse all the key-values from > the original payload" what do you mean? > Do you want to process them with syslog-ng filters etc, or are you > feeding this to another type of structured worker thread? I'd like to basically have the structured SNMP message parsed in syslog-ng, in the same way json-parser() parses JSON payload, and makes the key-values available as macros in syslog-ng. ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq

2 1

syslog() issue
by westlake 09 Jun '15

09 Jun '15

I'm not sure why i'm getting this error, output: syslog-ng --syntax-only Error parsing afsocket, syntax error, unexpected KW_IP, expecting LL_IDENTIFIER or LL_STRING in /etc/syslog-ng/syslog-ng.conf at line 31, column 9: syslog( ip(1.1.1.1) port(601) transport("tcp") ); mind me i'm new to syslog but i'm following the example from the online manual, not sure why this isn't working thanks

2 2

Solaris & Linux (RH6) Console and SSH Use Logging
by wiskbroom＠hotmail.com 08 Jun '15

08 Jun '15

Greetings; This is question is slightly OT, but I can't imagine a better place to ask; so please, no flames. I would like to begin logging all attempts (succesful, failed, no password given, etc) to login to Sparc/Solaris machines, as well as RedHat 6 Linux boxes. My clients are using stock syslog, but my server is running syslog-NG; my second goal is to redirect all login type logs to just one file for ALL of my Solaris & Linux servers. Does anyone have a known good syslog config file for both Solaris, and Linux? Also, a good syslog-NG entry to force just the console and ssh data into a separate file? Thank you! .vp

2 2

mongodb not found in /etc/syslog-ng/syslog-ng.conf at line 30, column 23:
by Rick Silacci 08 Jun '15

08 Jun '15

Hi, Can you tell what I'm doing wrong here? I entered destination mongodb { mongodb(); }; log { source(src); destination(mongodb); }; in syslogng.conf as in the directions and this is what happens: user@syslog:~/syslogng-web$ sudo service syslog-ng restart * Stopping system logging syslog-ng [ OK ] * Starting system logging syslog-ng Error parsing destination, destination plugin mongodb not found in /etc/syslog-ng/syslog-ng.conf at line 30, column 23: destination mongodb { mongodb() ; }; ^^^^^^^

2 2

Syslog-ng OSE with Juniper SIEM
by Muhammad Asim 08 Jun '15

08 Jun '15

Dear All I would like to know if there has been any integration done among syslog-ng OSE with Juniper Secure Analytics. Regards, Muhammad Asim | Senior Systems Engineer - ME, Pakistan | Juniper Networks, Inc. | (M) +923018510555 (O) +92518435204, +92518445530

3 3

Syslog-ng OSE with IBM Qradar
by Muhammad Asim 08 Jun '15

08 Jun '15

Dear All I would like to know if there has been any integration done among syslog-ng OSE with IBM Qradar?

2 1

odd configuration file
by craig bowser 05 Jun '15

05 Jun '15

Hi, I'm just learning about configuring syslog-ng. I'm using the latest version (info at end of email) and I've run into an error I can't figure out. I made a syntax error in my syslog-ng.conf that caused syslog-ng to fail to start. I corrected the error, but now when I try and start syslog-ng, it still fails. trying to start it with 'service syslog-ng start' or 'service syslog-ng restart' gets me: Job for syslog-ng.service failed. See "systemctl status syslog-ng.service" and "journalctl -xe" for details. But the error is buried in the systemd logs. Using '/usr/sbin/syslog-ng -F' gets me the following error: Error resolving reference; content='source', name='s_name', location='/etc/syslog-ng/syslog-ng.conf:132:7' What is that file? Can I safely delete it? Why can't I make it use the new .conf file? When I try '/usr/sbin/syslog-ng -f /etc/syslog-ng/syslog-ng.conf -F' I get the same error. syslog-ng 3.5.6 Installer-Version: 3.5.6 Revision: 3.5.6-2 [@416d315] (Ubuntu/15.04) Compile-Date: Oct 25 2014 14:14:32 Available-Modules: afuser,afsocket-notls,linux-kmsg-format,afprog,tfgeoip,csvparser,confgen,afamqp,json-plugin,redis,basicfuncs,afsmtp,affile,afsql,afmongodb,cryptofuncs,afsocket-tls,afsocket,syslogformat,afstomp,system-source,dbparser Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-IPv6: on Enable-Spoof-Source: on Enable-TCP-Wrapper: on Enable-Linux-Caps: on Enable-Pcre: on Craig L Bowser ____________________________ This email is measured by size. Bits and bytes may have settled during transport.

3 3

Reminder: GSoC students IRC meeting
by Czanik, Péter 02 Jun '15

02 Jun '15

This is a reminder, that we will have the IRC meeting in two days Hi, We organize an IRC meeting, where GSoC students and the broader syslog-ng community can meet. It would be about an hour long event, where students could talk about their projects and ideas with long time syslog-ng users and developers. It will be on the 4th of June, 17:00 CET (Budapest) time. It's 8AM in San Francisco and 20:30 in Delhi, so still day time for most of us. The event will take place on the freenode IRC servers, on channel #syslog-ng. If you don't have an IRC client, you can also use http://webchat.freenode.net/ to participate. I hope, many of you can participate! Bye, Peter Czanik (CzP) <peter.czanik(a)balabit.com> BalaBit IT Security / syslog-ng upstream http://czanik.blogs.balabit.com/ https://twitter.com/PCzanik

1 0

GSoC students IRC meeting
by Czanik, Péter 26 May '15

26 May '15

Hi, We organize an IRC meeting, where GSoC students and the broader syslog-ng community can meet. It would be about an hour long event, where students could talk about their projects and ideas with long time syslog-ng users and developers. It will be on the 4th of June, 17:00 CET (Budapest) time. It's 8AM in San Francisco and 20:30 in Delhi, so still day time for most of us. The event will take place on the freenode IRC servers, on channel #syslog-ng. If you don't have an IRC client, you can also use http://webchat.freenode.net/ to participate. I hope, many of you can participate! Bye, Peter Czanik (CzP) <peter.czanik(a)balabit.com> BalaBit IT Security / syslog-ng upstream http://czanik.blogs.balabit.com/ https://twitter.com/PCzanik

1 0

Configuring multithreading in syslog-ng need HELP !!
by nagesh mallappa 22 May '15

22 May '15

Hi All, I have added this line in my syslog-ng.conf file, *options {threaded(yes) ; };* I have 4 core system , root@nageshm-HVM-domU:/etc/syslog-ng# lscpu Architecture: i686 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian *CPU(s): 4* On Enabling mutithreading also, I do not the main syslog-ng thread spawning multiple threads, This is ps output, root@nageshm-HVM-domU:~# ps -elf | grep syslog-ng 1 S root 24871 1 0 80 0 - 1091 wait 08:22 ? 00:00:00 supervising syslog-ng 5 S root 24872 24871 0 80 0 - 4156 ep_pol 08:22 ? 00:00:00 /usr/sbin/syslog-ng -p /var/run/syslog-ng.pid 0 S root 24882 24255 0 80 0 - 1170 pipe_w 08:22 pts/13 00:00:00 grep --color=auto syslog-ng As we can only 2 processes are running, but I expect four threads to run, since I have 4 core system. Please Let me know, any configuration I am missing. Hoping for urgent help !!. Thanks, Nagesh

3 2

threading question
by Jim Hendrick 21 May '15

21 May '15

Hi, Could someone help clarify something for me? I am running with threading enabled, and I have separate redis destinations (three currently) all pointing to the same server. My thought was that each destination would use a separate connection and thread. I then filter log sources and send (roughly) 1/3 to each destination. My question is this. Does each destination get a separate thread? (reading the documentation it left room for doubt). Also - more generally is this a reasonable way to get better throughput to a redis data store? (note: both syslog-ng and redis are running locally, so the connections use 'localhost') Thanks! Jim

4 3

NEWS: syslog-ng docker images
by Budai, László 21 May '15

21 May '15

Hi, syslog-ng has a new github repo[1] where we provide some Dockerfiles which can help for developers and users. This repo serves as a basis for autobuilt docker images available at our docker.io repo[2]. Any feedback are welcomed :-) Bugreports, requests are handled in the github repo. [1] https://github.com/balabit/syslog-ng-docker [2] https://registry.hub.docker.com/repos/balabit/ regards, syslog-ng-team

1 0

Flag "no-multiline" not working on Syslog-ng
by Alan Sam 20 May '15

20 May '15

Hello community, I have the following diagram between some PE and Syslog-ng: Cisco devices -> Syslog-ng (running on Solaris) Syslog-ng version: o NTPSYSLOG# syslog-ng -V o syslog-ng 3.0.4 o Revision: ssh+git://bazsi@git.balabit //var/scm/git/syslog-ng/syslog-ng-ose--mainline--3.0#master#1b5d618e301ad94aa20e692ffba16469dece8d10 o Compile-Date: Sep 2 2009 06:15:53 o Enable-Threads: off o Enable-Debug: off o Enable-GProf: off o Enable-Memtrace: off o Enable-Sun-STREAMS: on o Enable-Sun-Door: on o Enable-IPv6: on o Enable-Spoof-Source: on o Enable-TCP-Wrapper: off o Enable-SSL: on o Enable-SQL: off o Enable-Linux-Caps: off o Enable-Pcre: on One of the cisco devices sends a particular log line that is splited in two lines (there is a line-break in between): *Mar 13 10:33:13: %BGP-3-INVALID_MPLS: Invalid MPLS label (1)* * received in update for prefix XXXX:XXXX:XXX.XXX.XXX.X/XXX from X.X.X.X* When the log reaches the Syslog-ng on Solaris server, it is logged like this: *Mar 13 10:33:14 PE06PVAL01 1182434: Mar 13 10:33:13: %BGP-3-INVALID_MPLS: Invalid MPLS label (1)* *Mar 13 10:33:14 PE06PVAL01 1182435: received in update for prefix XXXX:XXXX:XXX.XXX.XXX.X/XXX from X.X.X.X* The log is transfered by UDP from the cisco device to the Solaris server (where the syslog-ng runs). This is the configuration used in syslog-ng: *ntpsyslog> more /etc/syslog-ng/syslog-ng.conf* *@version: 3.0* *#* *# syslog-ng configuration file.* *#* *# See syslog-ng(8) and syslog-ng.conf(5) for more information.* *#* *options {* * stats_freq (0);* * flush_lines (0);* * time_reopen (10);* * log_fifo_size (1000);* * long_hostnames(off);* * use_dns (yes);* * use_fqdn (no);* * create_dirs (no);* * keep_hostname (yes);* * perm(0640);* *};* *source s_sys { sun-streams ("/dev/log" door("/etc/.syslog_door")); internal(); udp(flags("no-multi-line")); };* *destination d_cons { file("/dev/console"); };* *destination d_mesg { file("/var/adm/messages"); };* *destination d_mail { file("/var/log/syslog"); };* *destination d_auth { file("/var/log/authlog"); };* *destination d_mlop { usertty("operator"); };* *destination d_mlrt { usertty("root"); };* *destination d_mlal { usertty("*"); };* *destination cisco { file("/respaldo/syslog/cisco/cisco.log"); };* *#----------------------------------------------------------------------* *# Forward to a nisip server* *#* *destination cnc-cisco { udp("X.X.X.X" port(X)); };* *#----------------------------------------------------------------------* *filter f_filter1 { level(err) or* * (level(notice) and facility (auth, kern)); };* *filter f_filter2 { level(err) or* * (facility(kern) and level(notice)) or* * (facility(daemon) and level(notice)) or* * (facility(mail) and level(crit)); };* *filter f_filter3 { level(alert) or* * (facility(kern) and level(err)) or* * (facility(daemon) and level(err)); };* *filter f_filter4 { level(alert); };* *filter f_filter5 { level(emerg); };* *filter f_filter6 { facility(kern) and level(notice); };* *filter f_filter7 { facility(mail) and level(debug); };* *#filter f_filter10 { level(alert); };* *filter f_filter9 { facility(user) and level(alert); };* *filter f_cisco { facility(local2); };* *# Alternativa* *log { source(s_sys_cisco); filter(f_cisco); destination(cisco); };* *# Alternativa* *source s_juniper { file("/respaldo/syslog/juniper/juniper.log"); };* *destination d_juniper_tcp { tcp("X.X.X.X" port(X)); };* *filter f_juniper_tcp {not match("TOPO|/kernel:|snmpd|trace_*|PING_*|BGP_*|bgp_*|repeated|task|task_connect|EVENT|received iff message|rshd|cron" value("MESSAGE* *")); };* *log { source(s_juniper); filter(f_juniper_tcp); destination(d_juniper_tcp); };* *source s_cisco { file("/respaldo/syslog/cisco/cisco.log"); };* *destination d_cisco_tcp { tcp("X.X.X.X" port(X)); };* *log { source(s_cisco); destination(d_cisco_tcp); };* *##################################* *# FWD from Syslog to CNC Cisco* *##################################* *source s_cisco { file("/respaldo/syslog/cisco/cisco.log"); };* *log {source(s_cisco); destination(cnc-cisco); };* *ntpsyslog>* I have tried different configurations in order to make the "no-multi-line" flag work. However, none of them have worked: destination cisco { file("/respaldo/syslog/cisco/cisco.log" flags(no-multi-line)); }; source s_sys { sun-streams ("/dev/log" door("/etc/.syslog_door")); internal(); udp(flags("no-multi-line")); }; If more information is required, please do not hesitate to ask for it. Thank you beforehand for your help. Alan Sam

5 10

plugin options for Riak destination
by Parth Oberoi 20 May '15

20 May '15

Hello everyone, In order to build a riak destination , the first task is to figure out the options syslog-ng plugin would need to understand, For this we have considered the following two use cases 1) Inserting each log message into its own unique key.A templatable bucket would store keys(time-stamps) and its respective values(log messages) 2) Inserting each log message into a Set contained under a key. In order to use a set we need to have a bucket type as "set".Rest is similar to the first case. To switch between the two cases an option mode() is used with parameters "set" for using a set (2nd case) and "store" for the first case. Here is a config snippet for the destination: destination d_riak { riak( server("localhost") port(8087) bucket-type("message") bucket("logs_${YEAR}${MONTH}${DAY}") key("${UNIXTIME}-$(uuid)") value("$(format-json --scope selected-macros)") mode("set") ); }; I would really like to know if the users would prefer to set the mode and bucket type as separate options or as sub-options under bucket() as below: destination d_riak { riak( server("localhost") port(8087) bucket("logs_${YEAR}${MONTH}${DAY}" type("message") mode("store")) key("${UNIXTIME}-$(uuid)") value("$(format-json --scope selected-macros)") ); }; Any other feedback on the topic is also welcome. Thanks, Parth

3 2

Webmail Bildirimi
by balazs.scheidler＠balabit.com 19 May '15

19 May '15

1 0

3.2.5 and Multiline(?) messages from Solaris
by Ray Van Dolson 12 May '15

12 May '15

Admittedly haven't done enough searching or testing on this, but am hoping someone might have a quick answer. Recently moved from the 2.x verions to 3.2.5 (as part of EPEL on RHEL6). Have noticed that we're no longer getting the full messages from some Solaris boxen using the tcp() and udp() source definitions. Messages like this: May 10 02:29:30 dev-zfs2 scsi: [ID 365881 kern.info] /pci@0,0/pci8086,3410@9/pci15d9,400@0 (mpt_sas0): May 10 02:29:30 dev-zfs2 Log info 0x31080000 received for target 24. May 10 02:29:30 dev-zfs2 scsi_status=0x0, ioc_status=0x804b, scsi_state=0x0 Come through looking like this: May 10 02:29:30 dev-zfs2 scsi: [ID 365881 kern.info] /pci@0,0/pci8086,3410@9/pci15d9,400@0 (mpt_sas0): (Only the initial line) However, messages like this one: May 9 04:12:57 dev-zfs2 scsi: [ID 243001 kern.warning] WARNING: /pci@0,0/pci8086,3410@9/pci15d9,400@0 (mpt_sas0): May 9 04:12:57 dev-zfs2 mptsas_handle_event_sync: IOCStatus=0x8000, IOCLogInfo=0x31110610 .. do seem to be coming through "whole" (I do note that the priority is different in both). Relevant config items are as follows: log { source(remote); filter(syslog); destination(hosts_syslog); }; source remote { udp(); tcp(); # udp(ip(0.0.0.0) port(514)); # tcp(ip(0.0.0.0) port(514)); }; destination hosts_syslog { file("/logs/hosts/$HOST/$YEAR/$MONTH/syslog.$HOST.$YEAR.$MONTH.log" create_dirs(yes)); pipe("/logs/hosts/everything.fifo"); }; filter syslog { (not facility(mail) and not filter(f_ucgw) and not filter(f_esx)); }; Will try and do some packet captures to confirm Solaris is, in fact, sending the entire message (I believe it is since this worked on syslog-ng 2.x). Thanks, Ray

2 4

Error resolving reference; content='source', name='src', location='/etc/syslog-ng/syslog-ng.conf:26:7
by Rick Silacci 12 May '15

12 May '15

I can't figure out why I'm getting this message. Keep in mind, I just started using syslog. Here's the cfg: @version: 3.5 @include "scl.conf" @include "`scl-root`/system/tty10.conf" # Syslog-ng configuration file, compatible with default Debian syslogd # installation. # First, set some global options. options { chain_hostnames(off); flush_lines(0); use_dns(no); use_fqdn(no); owner("root"); group("adm"); perm(0640); stats_freq(0); bad_hostname("^gconfd$"); }; ######################## # Sources ######################## # This is the default behavior of sysklogd package # Logs may come from unix stream, but not from another machine. # #source s_src { # system(); # internal(); #}; destination mongodb { mongodb(); }; log { source(); destination(mongodb); }; # If you wish to get logs from remote machine you should uncomment # this and comment the above source line. # source s_net { tcp(ip(127.0.0.1) port(1000) keep-alive(yes)); }; ######################## # Destinations ######################## # First some standard logfile # destination d_auth { file("/var/log/auth.log"); }; destination d_cron { file("/var/log/cron.log"); }; destination d_daemon { file("/var/log/daemon.log"); }; destination d_kern { file("/var/log/kern.log"); }; destination d_lpr { file("/var/log/lpr.log"); }; destination d_mail { file("/var/log/mail.log"); }; destination d_syslog { file("/var/log/syslog"); }; destination d_user { file("/var/log/user.log"); }; destination d_uucp { file("/var/log/uucp.log"); }; #destination mongodb { file("/var/log/mongodb.log"); }; # This files are the log come from the mail subsystem. # destination d_mailinfo { file("/var/log/mail.info"); }; destination d_mailwarn { file("/var/log/mail.warn"); }; destination d_mailerr { file("/var/log/mail.err"); }; # Logging for INN news system # destination d_newscrit { file("/var/log/news/news.crit"); }; destination d_newserr { file("/var/log/news/news.err"); }; destination d_newsnotice { file("/var/log/news/news.notice"); }; # Some `catch-all' logfiles. # destination d_debug { file("/var/log/debug"); }; destination d_error { file("/var/log/error"); }; destination d_messages { file("/var/log/messages"); }; # The root's console. # destination d_console { usertty("root"); }; # Virtual console. # destination d_console_all { file(`tty10`); }; # The named pipe /dev/xconsole is for the nsole' utility. To use it, # you must invoke nsole' with the -file' option: # # $ xconsole -file /dev/xconsole [...] # destination d_xconsole { pipe("/dev/xconsole"); }; # Send the messages to an other host # #destination d_net { tcp("127.0.0.1" port(1000) log_fifo_size(1000)); }; # Debian only destination d_ppp { file("/var/log/ppp.log"); }; ######################## # Filters ######################## # Here's come the filter options. With this rules, we can set which # message go where. filter f_dbg { level(debug); }; filter f_info { level(info); }; filter f_notice { level(notice); }; filter f_warn { level(warn); }; filter f_err { level(err); }; filter f_crit { level(crit .. emerg); }; filter f_debug { level(debug) and not facility(auth, authpriv, news, mail); }; filter f_error { level(err .. emerg) ; }; filter f_messages { level(info,notice,warn) and not facility(auth,authpriv,cron,daemon,mail,news); }; filter f_auth { facility(auth, authpriv) and not filter(f_debug); }; filter f_cron { facility(cron) and not filter(f_debug); }; filter f_daemon { facility(daemon) and not filter(f_debug); }; filter f_kern { facility(kern) and not filter(f_debug); }; filter f_lpr { facility(lpr) and not filter(f_debug); }; filter f_local { facility(local0, local1, local3, local4, local5, local6, local7) and not filter(f_debug); }; filter f_mail { facility(mail) and not filter(f_debug); }; filter f_news { facility(news) and not filter(f_debug); }; filter f_syslog3 { not facility(auth, authpriv, mail) and not filter(f_debug); }; filter f_user { facility(user) and not filter(f_debug); }; filter f_uucp { facility(uucp) and not filter(f_debug); }; filter f_cnews { level(notice, err, crit) and facility(news); }; filter f_cother { level(debug, info, notice, warn) or facility(daemon, mail); }; filter f_ppp { facility(local2) and not filter(f_debug); }; filter f_console { level(warn .. emerg); }; ######################## # Log paths ######################## log { source(s_src); filter(f_auth); destination(d_auth); }; log { source(s_src); filter(f_cron); destination(d_cron); }; log { source(s_src); filter(f_daemon); destination(d_daemon); }; log { source(s_src); filter(f_kern); destination(d_kern); }; log { source(s_src); filter(f_lpr); destination(d_lpr); }; log { source(s_src); filter(f_syslog3); destination(d_syslog); }; log { source(s_src); filter(f_user); destination(d_user); }; log { source(s_src); filter(f_uucp); destination(d_uucp); }; log { source(s_src); filter(f_mail); destination(d_mail); }; #log { source(s_src); filter(f_mail); filter(f_info); destination(d_mailinfo); }; #log { source(s_src); filter(f_mail); filter(f_warn); destination(d_mailwarn); }; #log { source(s_src); filter(f_mail); filter(f_err); destination(d_mailerr); }; log { source(s_src); filter(f_news); filter(f_crit); destination(d_newscrit); }; log { source(s_src); filter(f_news); filter(f_err); destination(d_newserr); }; log { source(s_src); filter(f_news); filter(f_notice); destination(d_newsnotice); }; #log { source(s_src); filter(f_cnews); destination(d_console_all); }; #log { source(s_src); filter(f_cother); destination(d_console_all); }; #log { source(s_src); filter(f_ppp); destination(d_ppp); }; log { source(s_src); filter(f_debug); destination(d_debug); }; log { source(s_src); filter(f_error); destination(d_error); }; log { source(s_src); filter(f_messages); destination(d_messages); }; log { source(s_src); filter(f_console); destination(d_console_all); destination(d_xconsole); }; log { source(s_src); filter(f_crit); destination(d_console); }; # All messages send to a remote site # #log { source(s_src); destination(d_net); }; ### # Include all config files in /etc/syslog-ng/conf.d/ ### @include "/etc/syslog-ng/conf.d/*.conf"

2 6

Re: [syslog-ng] 3.2.5 and Multiline(?) messages from Solaris
by Jim Hendrick 12 May '15

12 May '15

You hit it I think. The first thing is to capture packets on at least one end to isolate where this is happening. Jim Sent from my Verizon Wireless 4G LTE smartphone -------- Original message -------- From: Ray Van Dolson <rvandolson(a)esri.com> Date: 05/11/2015 11:42 PM (GMT-05:00) To: syslog-ng(a)lists.balabit.hu Subject: [syslog-ng] 3.2.5 and Multiline(?) messages from Solaris Admittedly haven't done enough searching or testing on this, but am hoping someone might have a quick answer. Recently moved from the 2.x verions to 3.2.5 (as part of EPEL on RHEL6). Have noticed that we're no longer getting the full messages from some Solaris boxen using the tcp() and udp() source definitions. Messages like this: May 10 02:29:30 dev-zfs2 scsi: [ID 365881 kern.info] /pci@0,0/pci8086,3410@9/pci15d9,400@0 (mpt_sas0): May 10 02:29:30 dev-zfs2 Log info 0x31080000 received for target 24. May 10 02:29:30 dev-zfs2 scsi_status=0x0, ioc_status=0x804b, scsi_state=0x0 Come through looking like this: May 10 02:29:30 dev-zfs2 scsi: [ID 365881 kern.info] /pci@0,0/pci8086,3410@9/pci15d9,400@0 (mpt_sas0): (Only the initial line) However, messages like this one: May 9 04:12:57 dev-zfs2 scsi: [ID 243001 kern.warning] WARNING: /pci@0,0/pci8086,3410@9/pci15d9,400@0 (mpt_sas0): May 9 04:12:57 dev-zfs2 mptsas_handle_event_sync: IOCStatus=0x8000, IOCLogInfo=0x31110610 .. do seem to be coming through "whole" (I do note that the priority is different in both). Relevant config items are as follows: log { source(remote); filter(syslog); destination(hosts_syslog); }; source remote { udp(); tcp(); # udp(ip(0.0.0.0) port(514)); # tcp(ip(0.0.0.0) port(514)); }; destination hosts_syslog { file("/logs/hosts/$HOST/$YEAR/$MONTH/syslog.$HOST.$YEAR.$MONTH.log" create_dirs(yes)); pipe("/logs/hosts/everything.fifo"); }; filter syslog { (not facility(mail) and not filter(f_ucgw) and not filter(f_esx)); }; Will try and do some packet captures to confirm Solaris is, in fact, sending the entire message (I believe it is since this worked on syslog-ng 2.x). Thanks, Ray ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq

1 0

unix-dgram for Linux syslog stream
by ZeroUno 09 May '15

09 May '15

Hi, I've read some old discussions about unix-dgram vs unix-stream for getting the /dev/log syslog stream on Linux (e.g. https://bugs.archlinux.org/task/22153) but some years have passed by and I'd like to be sure my configuration is safe. I'm using syslog-ng 3.2.5 on a system composed of different Linux (RHEL6) machines. I cannot install a different version. There is a central log facility, but each machine collects and sends its logs using the following source config: source src { file ("/proc/kmsg" program_override("kernel: ")); unix-dgram ("/dev/log" flags(no-multi-line)); internal(); }; It was originally using unix-stream(), but I need to change it to unix-dgram() because some custom applications are sending multiline messages which need to be converted into single line, and unix-stream() does not support this flag. Can I be sure that no messages risk to be lost due to this change? This is a rather critical application. Thank you very much. -- 01

2 2

Re: [syslog-ng] Feedback for GSoC project - RIak Destination for Syslog-ng
by Gergely Nagy 06 May '15

06 May '15

Hi! I'm the mentor for the Riak destination for syslog-ng project, please allow me to answer the questions below: >>>>> "Fred" == Fred Dushin <fdushin(a)basho.com> writes: Fred> As far as I understand, you're talking about a mapping from keys to Fred> sets, but I'm unclear on a few things. The idea is to map a set of log messages to a Riak Set. Where both the key used for the set, and its contents are configurable by the user. There are no plans for a default at this time. There are many ways to configure a syslog-ng=>Riak setup with a destination like the one planned. One is to turn each log message (after parsing) to a Riak Map, and push those maps into a Riak Set. Another way is to format the parsed log messages (with all the extracted fiels, if any) into JSON, and push those into a set. So, for example, given the following syslog line: May 6 14:42:18 eowyn avahi-daemon[27812]: Invalid response packet from host fe80::5d0f:d53a:7b6:3680. We'd end up with a JSON like this: {"timestamp": "2015-05-06T14:42:18+02:00", "host": "eowyn", "program": "avahi-daemon", "pid": 27812, "message": "Invalid response packet from host fe80::5d0f:d53a:7b6:3680.", "avahi-daemon": { "type": "warning", "message": "Invalid response packet", "host": "fe80::5d0f:d53a:7b6:3680" } } We could either add that to a Riak set as-is, or turn it into a Riak map first. Fred> What are the keys you are thinking about? Time stamps? If Fred> timestamps, these are presumably the timestamps of the syslog Fred> event? Whatever the user configures. They may be time stamps (rounded, for predictable keys), or a combination of program name + current date (day granuality). Fred> Just a word of warning, if so. You might find a lot of Fred> variation in timestamp formats and granularity. Perhaps you Fred> can get something reliable out of syslog-ng, We get something sensible out of syslog-ng. But in the end, it is up to the user to configure the template used for keys. There may - and probably will - be examples, but no default. Fred> but that won't help you in the case where syslog-ng is Fred> functioning as a syslog relay, and you want to preserve the Fred> timestamp of the originator, which you should, if you want to Fred> preserve integrity of the logs (e.g, for compliance). In case of syslog-ng, we actually have access to a few kinds of timestamps: the timestamp from the log message (if any), the timestamp of receipt, and the current time. The granularity of timestamps is configurable to some extent. Fred> Or are you talking about a key being a (course grained) Fred> timestamp, say, an integral value in UTC seconds, for example? Fred> And the value(s) being all logs in that interval? Is that your Fred> motivation for sets? That's one way, yes. One could also use something like $PROGRAM/$YEAR-$MONTH-$DAY as key, if the program doesn't produce more than a megabyte of logs a day. So with the example above, our key in case of that log would be avahi-daemon/2015-05-06, and the message would be an element of the set underneath the key. Fred> How much of the syslog payload are you planning to parse? The destination itself is not going to do any parsing. Other parts of syslog-ng do that, and it is up to the user to set up a pipeline that feeds the destination. The source may be syslog, HTTP logs, the Journal, or any of the other sources syslog-ng supports. How much parsing is done, and what gets extracted, is no concern to the destination plugin. Fred> Another interesting problem is that the STRUCTURED-DATA element of Fred> 5424 uses OIDs to discriminate different data types that are encoded Fred> in the header. And while there is a kind of loosely coupled authority Fred> for OIDs, there is no infrastructure for determining a parsing Fred> strategy for these fields. They could really be anything, in the worst Fred> case. As far as I remember, syslog-ng treats all STRUCTURED-DATA elements as strings. But there are tools within syslog-ng to allow converting to other data types, but that must be done explicitly. Fred> But regardless of the deeply structured data, you could get some very Fred> interesting traction by just taking standard headers and indexing them Fred> through Yokozuna. Certainly, indexing the body of a syslog message is Fred> a great idea, as these messages are generally unstructured and fodder Fred> for lucene. This is something that Logstash/ElasticSearch can do Fred> pretty effectively today, and it would be cool to see the same in Riak Fred> + some syslog provider. Yep! When I proposed the idea, using Yokozuna is something I had in mind. Combine the parsing abilities of syslog-ng, Riak for archival purposes, and Yokozuna for searching. That sounds like a match made in heaven. Fred> Finally, it would be really nice if you could structure your plugin in Fred> such a way that they could eventually be ported to rsyslog [2]. The Fred> rsyslogd daemon is deployed by default on certain Linux favors and Fred> enjoys fairly widespread distribution. You might be able to get it Fred> supported in that community, as well. Part of the project is writing a small library to send data to Riak, From C. Just enough for syslog-ng's needs. That library could be used by rsyslog, too (like the MongoDB library originally written for syslog-ng's purposes is used by rsyslog too). But sharing more code than that is not practical, the two daemons work in widely different ways. -- |8]

1 0

Re: [syslog-ng] Feedback for GSoC project - RIak Destination for Syslog-ng
by Gergely Nagy 05 May '15

05 May '15

>>>>> "Christopher" == Christopher Meiklejohn <cmeiklejohn(a)basho.com> writes: >> The intention is to store log messages in each element of the set: >> either as a string (syslog or json, or whatever else the user sees fit), >> or as a map of key-value pairs (where values themselves can be maps >> too). >> >> On average, the log messages are a few kilobytes in size. There may be >> exceptions, but >1mb ones are fairly rare. How much data the set would >> hold... now that's a question that can't really be answered. It is >> really up to the syslog-ng user to configure that. Christopher> I’m referring to the size of the entire set, not the objects that will be members of Christopher> the set. Therefore, the performance penalty seen when using large objects would Christopher> be observed as soon as the size of the entire set (or map) has reached ~1 MB. Christopher> Given that restriction, I’d imagine you would only be able to store a few messages Christopher> in each set. That granularity seems like you are no longer getting the benefits Christopher> of the set. The granularity is configurable by the user: if they have small (say, a few hundred byte long) messages, then we can store a reasonable amount of them in a single Set. For example, assuming an average length of 384 bytes / message (the longest line in today's logs on my laptop), a Set would be able to store about 2k messages. That's not too bad. Christopher> Additionally, the primary benefit of the data types in Riak is that they converge Christopher> deterministically when dealing with concurrent Christopher> operations. Not only that: using sets makes the keys predictable. If I want to retrieve the logs, with sets, I can retrieve the 2015-05-01T15:12:10-T15:12:15 key for example, and have all the logs from those 5 seconds. If I used one message per Riak object, it would be much harder to read the data back. There may be multiple threads adding to the same set, so the deterministic convergence is useful still. Christopher> I’m curious if the set is the right choice here; could Christopher> you just use a custom set format inside of a normal Christopher> Riak object That's an option worth considering, yes. Thanks! Christopher> (or store one message per Riak object, given the write will be an Christopher> immutable log entry?) That's the first goal of the project, because that's the easiest and most straightforward to implement. The downside of one message per Riak object is that it's hard to retrieve the data, because making the keys predictable is not going to be easy. Boxing a few hundred (or few thousand) together into a Set has the advantage of making the keys predictable at the cost of transferring more data to the client when looking for a subset of the logs within the set. -- |8]

1 0

Re: [syslog-ng] Feedback for GSoC project - RIak Destination for Syslog-ng
by Gergely Nagy 05 May '15

05 May '15

>>>>> "Christopher" == Christopher Meiklejohn <cmeiklejohn(a)basho.com> writes: Christopher> I’m a bit concerned with your use of the set embedded in the Christopher> map. The original idea was to use a Set directly. The Set-in-Map thing was just a thought experiment (Map-in-Set would make more sense). Christopher> Large objects have traditionally been a big problem in Riak due Christopher> to the use of distributed Erlang and head of line blocking. I’m Christopher> curious if you could elaborate on what type of data you will be Christopher> storing in the set: how big you expect each item to be, how big you Christopher> expect the map to be, and the overall layout of data inside of the Christopher> data structure. The intention is to store log messages in each element of the set: either as a string (syslog or json, or whatever else the user sees fit), or as a map of key-value pairs (where values themselves can be maps too). On average, the log messages are a few kilobytes in size. There may be exceptions, but >1mb ones are fairly rare. How much data the set would hold... now that's a question that can't really be answered. It is really up to the syslog-ng user to configure that. Hope that made it a little more clear. -- |8], mentor for the riak destination for syslog-ng GSoC project

1 0

Feedback for GSoC project - RIak Destination for Syslog-ng
by Parth Oberoi 02 May '15

02 May '15

Hello Everyone, My proposal[0] for adding Riak as a destination to Syslog-ng was accepted for GSoC2015, so before my coding period begins i.e. 25th May, I wanted to do some ground work and take some feedback from the potential users. The proposal[0] contains all the initial implementation details which might require improvement . Therefore if some of you could take a look and give me some feedback on it, would help me greatly. Below are a few links to important snippets of the proposal. [0] - https://github.com/balabit/syslog-ng/wiki/GSoC2015-Proposal:-Riak-destinati… *Goals for the project :-* https://github.com/balabit/syslog-ng/wiki/GSoC2015-Proposal:-Riak-destinati… *Development Tools :-* https://github.com/balabit/syslog-ng/wiki/GSoC2015-Proposal:-Riak-destinati… *Deliverables :-* https://github.com/balabit/syslog-ng/wiki/GSoC2015-Proposal:-Riak-destinati… *My scheduled Plan:-* https://github.com/balabit/syslog-ng/wiki/GSoC2015-Proposal:-Riak-destinati… Thanks in advance. Regards, Parth Oberoi (@htrap)

1 0

insider 2015-04: 3.7 beta1; RedHat Summit; Hadoop; GSoC; RFC5424;
by Czanik, Péter 30 Apr '15

30 Apr '15

Dear syslog-ng users, This is the 41th issue of the syslog-ng Insider, a monthly newsletter that brings you syslog-ng-related news. NEWS syslog-ng 3.7 beta1 released ---------------------------- In the first beta of syslog-ng OSE 3.7, the Java and Python destinations were migrated from the syslog-ng incubator to the syslog-ng core. Also, syslog-ng now includes an interactive syslog-ng.conf debugger. Read about known problems and how to get started at https://czanik.blogs.balabit.com/2015/04/first-beta-of-syslog-ng-3-7-releas… syslog-ng at RedHat Summit -------------------------- BalaBit is a general sponsor of the Red Hat Summit. The event will be in Boston, June 23-26. http://www.redhat.com/summit/ Visit the BalaBit booth to talk about syslog-ng and log management! syslog-ng Hadoop support ------------------------ With the release of syslog-ng PE 5F3, support for Hadoop arrived. It will also be part of the upcoming syslog-ng OSE 3.7 release. Read more about how syslog-ng can send logs to a HDFS destination: https://jluby.blogs.balabit.com/2015/04/08/sending-logs-to-hadoop-using-sys… With this solution BalaBit became a certified MapR technology partner: https://www.mapr.com/partners/partner/balabit-it-security syslog-ng in Google Summer of Code 2015 --------------------------------------- This year we participate in GSoC again. The syslog-ng team received many good proposals, it will be a tough choice to decide on the finalists. We plan to have an IRC meeting with the participation of the winning GSoC students and members of the syslog-ng community. A separate e-mail will announce the exact time. The project and idea list is available at https://github.com/balabit/syslog-ng/wiki/GSoC2015-idea-&-project-list Forwarding name value pairs using RFC5424 ----------------------------------------- Recently many users asked how to forward the name of a file source over the network. It can easily be done using the RFC5424 syslog protocol. The same method can also be used to create and forward name value pairs between servers in general. Read more about it at https://czanik.blogs.balabit.com/2015/03/using-rfc5424-syslog-to-forward-fi… syslog-ng @ LOADays 2015 ------------------------ This year syslog-ng was presented again at LOADays, a small but very exciting Linux admin conference in Antwerp. The basics of syslog-ng, the importance of name value pairs and the latest developments in syslog-ng were presented. For details, check: https://czanik.blogs.balabit.com/2015/04/loadays-2015-logging-elasticsearch… NEW RELEASES syslog-ng 3.7.0beta1: https://github.com/balabit/syslog-ng/releases/tag/syslog-ng-3.7.0beta1 Your feedback and news tips about the next issue is welcome at documentation(a)balabit.com To read this newsletter on-line, visit: http://insider.blogs.balabit.com/ Peter Czanik (CzP) <peter.czanik(a)balabit.com> BalaBit IT Security / syslog-ng upstream http://czanik.blogs.balabit.com/ https://twitter.com/PCzanik

1 0

syslog-ng memory leak ? valgrind attached
by Jim Hendrick 29 Apr '15

29 Apr '15

From 3.6.2 OSE built with json & redis Seeing steady memory growth (> 10 GB after ~ a day at ~7000 EPS load) This was from a few minutes of runtime. (~5) Is this a known issue or did I trip over something? Thanks! Jim

5 4