- syslog-ng - lists.balabit.hu

syslog-ng as "shipper" into ELK stack
by Jim Hendrick 20 Oct '14

20 Oct '14

Hi, I am working on configuring Elasticsearch, Logstash & Kibana (ELK) to test it as a backend search tool for large volumes of logs. I decided to put Redis in front of Logstash as a "broker" for the incoming logs, and syslog-ng as the "shipper" so it looks like this: syslog-ng ==> redis ==> logstash ==> elasticsearch ==> apache ==> kibana It works very well using the redis destination in syslog-ng, although I am having performance problems with logstash & elasticsearch default configurations keeping up. (I topped out today sending ~7000 events per second, and saw an insane amount of swapping going on) Not so much a specific question (I'll be working on heap & thread settings and am pretty confident I can get it to handle at least this moderate load) but I was wondering if anyone else is working in this area. Also, in this configuration logstash is simply "parsing" the data it pulls from redis and sending it into elasticsearch. Seems like something syslog-ng might be able to do directly. Is anyone aware of any plans to implement an elasticsearch destination? Feel free to contact me on or off list if you want to discuss this. Thanks!! Jim

3 3

inserts into a sqlite3 database are not delayed
by Davide Alberani 15 Oct '14

15 Oct '14

Hi, I'm using a SQLite 3 database as a destination, but it seems that the options to delay the insert of new rows are ignored. The destination is something like: destination sqlite_db { sql(type(sqlite3) flags('explicit-commits') flush_timeout(60000) log_fifo_size(1000) flush_lines(1000) retry_sql_inserts(2) database("/tmp/logs.db") [...table, columns, values, indexes definitions...] }; Every time a log arrives, it's immediately written into the database, while I expected to have it delayed accordingly to flush_timeout and flush_lines. Currently I'm using syslog-ng OSE 3.4.2, but looking at the code of the latest version doesn't seem to have changed much. SQLite version is 3.8.4. libdbi 0.8.4 libdbi-drivers 0.8.3 libdbi-dbd-sqlite 0.8.3 libol 0.3.16 libevtlog 0.2.13 Any idea about how to fix it? Thanks in advance, -- Davide Alberani <davide.alberani(a)gmail.com> [PGP KeyID: 0x465BFD47] http://www.mimante.net/

2 1

Reg:Can two versions of Syslog-ng used on the same server
by Justin Kala 07 Oct '14

07 Oct '14

Can two versions of syslog-ng r reside on the same physical server. If so ,how do I handle the incoming messages.How do I configure syslog-ng.conf Please suggest -- Kaladhar

2 2

Re: [syslog-ng] syslog-ng as "shipper" into ELK stack
by Jim Hendrick 06 Oct '14

06 Oct '14

Thanks. Why rabbitmq instead of redis? Is it faster, or does it offer some additional functions? Jim Sent from my Verizon Wireless 4G LTE smartphone -------- Original message -------- From: Alexandre Biancalana <biancalana(a)gmail.com> Date:10/03/2014 7:01 PM (GMT-05:00) To: Syslog-ng users' and developers' mailing list <syslog-ng(a)lists.balabit.hu> Subject: Re: [syslog-ng] syslog-ng as "shipper" into ELK stack On Thu, Oct 2, 2014 at 9:33 PM, Jim Hendrick <jrhendri(a)roadrunner.com> wrote: Hi, I am working on configuring Elasticsearch, Logstash & Kibana (ELK) to test it as a backend search tool for large volumes of logs. I decided to put Redis in front of Logstash as a "broker" for the incoming logs, and syslog-ng as the "shipper" so it looks like this: syslog-ng ==> redis ==> logstash ==> elasticsearch ==> apache ==> kibana I've been using the following: syslog-ng => rabbitmq => elasticsearch syslog-ng + patterndb to parse logs and write then in json format on rabbitmq, after that is just use elasticsearch amqp river to consume the queue.

3 2

Re: [syslog-ng] syslog-ng as "shipper" into ELK stack
by Jim Hendrick 03 Oct '14

03 Oct '14

Thanks! I will. Jim Sent from my Verizon Wireless 4G LTE smartphone -------- Original message -------- From: Fabien Wernli <wernli(a)in2p3.fr> Date:10/03/2014 4:12 AM (GMT-05:00) To: Syslog-ng users' and developers' mailing list <syslog-ng(a)lists.balabit.hu> Subject: Re: [syslog-ng] syslog-ng as "shipper" into ELK stack Hi Jim, On Fri, Oct 03, 2014 at 12:33:41AM +0000, Jim Hendrick wrote: > syslog-ng ==> redis ==> logstash ==> elasticsearch ==> apache ==> kibana We've been using the following stack for over a year: syslog-ng ==> logstash ==> elasticsearch For various reasons, one being performance, we recently switched to: syslog-ng ==> elasticsearch This was done thanks to the syslog-ng-incubator perl module. I've set up a small github repository where you can see our configuration [1]. > (I topped out today sending ~7000 events per second, and saw an insane > amount of swapping going on) I've had tremendous issues with LS when the workload was darting up. Since we switched to perl, we still have issues, but they're certainly not performance related: with a single perl destination we could easily keep up 10k events per second on a mediumish virtual machine. > Is anyone aware of any plans to implement an elasticsearch destination? The upcoming 3.6 version will ship with a "native" elasticsearch destination, which currently however is only a wrapper script. I'd highly appreciate if you could test a similar config to ours, in order to share some experience. Cheers [1] https://github.com/faxm0dem/syslog_ng-elasticsearch ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq

1 0

Question on parsing
by Richards, James L - DOA 03 Oct '14

03 Oct '14

So I have a scenario I am having difficulties with. I have an IDS sensor (suricata), and it is generating a log-file at /log_file_dir/fast.log And I would like to parse this log and send it off to a remote syslog server. I have put the following in my syslog-ng.conf: source s_log_server { file("/log_file_dir/fast.log " program_override("snort")); }; added a destination for the remote server: destination d_log_server { udp ("fqdn.of.primary.elsa.box" port(514)); }; Then in the log{ section I have put this: Destination (d_log_server); Logs are making it to the remote box, but in an unparsed format... How do I get this to trigger a parser in syslog-ng? Thanks much, Jim

1 0

File Source limits in OSE and PE?
by Doug McClure 03 Oct '14

03 Oct '14

Are there limits to the number of unique files that can be monitored with an installation of OSE or PE? For example, could I monitor 500 unique log files on a given server? Tks! Doug

2 2

exclude pattern from logging
by fRANz 01 Oct '14

01 Oct '14

Hello guys, I'm trying to config syslog-ng to ignore and don't log some strings. I'm using syslog-ng-3.2.5-4.el6.x86_64 on a CentOS 6.5: the config is pretty simple: # cat /etc/syslog-ng/syslog-ng.conf @version:3.2 options { check_hostname(yes); keep_hostname(yes); stats_freq(0); chain_hostnames(no); }; source inputs { internal(); unix-stream("/dev/log"); udp(); tcp(max_connections(100)); }; destination logpile { file("/logs/$HOST/$YEAR/$MONTH/$DAY/$FACILITY" owner(root) group(root) perm(0600) create_dirs(yes) dir_perm(0700)); }; #filter vmware_filter { match("Section for VMware ESX" value ("MESSAGE")); }; #filter vmware_filter { message("Section for VMware ESX"); }; filter vmware_filter { not message("Section for VMware ESX"); }; #log { source(inputs); filter(vmware_filter); flags(final); }; #log { source(inputs); destination(logpile); }; log { source(inputs); filter(vmware_filter); destination(logpile); }; but I cannot find the right key. Wha'ts wrong in this config? You can see old configuration applied. Thanks for any tip. -f

2 4

Syslog-NG.conf to Fork to Two Log Aggregators
by wiskbroom＠hotmail.com 01 Oct '14

01 Oct '14

Hello; I have syslog clients that I would like to configure to send log-data to a middle-man/intermediary syslog-NG server. Once received on the intermediary, I want to immediately fork that data onto a different log-server, not syslog-NG; satisfying a requirement to feed two systems. The reason for the fork is because the non-syslog-NG-server is running a proprietary logging system, and it must, at least for now, be capable of seeing *most* of my logs. It, the non-syslog-NG-server, is incapable of retransmitting to my syslog-NG server, nor would I trust it to do so. My questions to the list are, 1. Has anyone successfully done something similar? 2. Any recommendations/gotchas I should be aware of? 3. Can I also configure syslog-NG to also resend Splunk data? Or do I have to run a Splunk Univ Forwarder configured similarly to my intermediary syslog-NG server to achieve that? (Yes, I know, OT question, sorry...) Thank you in advance, .vp

2 1

Continued struggle with getting shell env vars into log statements
by jetjnkr＠gmail.com 29 Sep '14

29 Sep '14

Greetings, I'm trying to get an environment variable (As defined in the shell - take for example "HOSTNAME"). echo $HOSTAME provides the expected string.... I then would like to append this to a set of messages based upon a specific destination. I'm using syslog-ng 3.2.5. In this case the 'hostname' is just an example, I know that the sending hostname is automatically prepended - used here because it's commonly defined on systems. I've included the following config lines from my syslog-ng.conf file. destination d_web_call { file("/var/log/web_call" template("<$PRI> $DATE $HOST $MSGHDR$MSG host=`HOSTNAME` \n") template_escape(no)); }; filter f_web_call { facility(local1) and ( match("^.*apache-call.*$" value("MSGHDR")) );}; log { source(s_sys); filter(f_web_call); destination (d_web_call); flags(final,flow-control); }; Using the following test string: /usr/bin/logger -t apache-call-tst -p local1.notice "This is a test message 25" I get the following: <141> Sep 26 17:21:18 lab.xxxx.com apache-call-tst: This is a test message25 host= I've received a few pointers from folks here (including going to 3.5 which can't be completed in the short term....). If tried things with 'define' and using the '$' but not back-tick for the shell variable. It is not clear when this value is set (if at all) within the config file. Thanks, Jim.

1 0

type hinting / mongodb / TAGS as array
by Lucas, Sascha 29 Sep '14

29 Sep '14

Hi, I'm again looking for a way to store patterndb classified / tagged events in mongodb. My greatest wish is to store the event tags as an array (list) in mongodb. This would greatly enhance querying for different combinations of tags. ATM TAGS is a string. According to the documentation, there seems no way to "type hint" that tags is an array. Or do I miss something? Thanks, Sascha. Aufsichtsratsvorsitzender: Herbert Vogel Geschäftsführung: Michael Krüger Sitz der Gesellschaft: Halle/Saale Registergericht: Amtsgericht Stendal | Handelsregister-Nr. HRB 208414 UST-ID-Nr. DE 158253683 Diese E-Mail enthält vertrauliche und/oder rechtlich geschützte Informationen. Wenn Sie nicht der richtige Empfänger sind oder diese E-Mail irrtümlich erhalten haben, informieren Sie bitte sofort den Absender und vernichten Sie diese Mail. Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser Mail oder des Inhalts dieser Mail sind nicht gestattet. Diese Kommunikation per E-Mail ist nicht gegen den Zugriff durch Dritte geschützt. Die GISA GmbH haftet ausdrücklich nicht für den Inhalt und die Vollständigkeit von E-Mails und den gegebenenfalls daraus entstehenden Schaden. Sollte trotz der bestehenden Viren-Schutzprogramme durch diese E-Mail ein Virus in Ihr System gelangen, so haftet die GISA GmbH - soweit gesetzlich zulässig - nicht für die hieraus entstehenden Schäden.

2 4

Re: [syslog-ng] SOLVED: syslog-ng and python program destination constantly being restarted again and again
by Andrew Bell 26 Sep '14

26 Sep '14

Hi,. Thanks for the feedback. I actually figured out the reason as to why this was happening, and it had to do with syslog-ng referencing a the default python path on the system instead of the custom python path I've installed in my home directory ("python" alias pointed to "/usr/bin/python" instead of my "/home/bin/python" path). The custom python path had all the third-party libraries that my script depended on for execution, and so that was why syslog-ng kept sending all these /var/log/messages because it couldn't import the libraries and was using the wrong python instance (Used an strace command against syslog-ng in order to confirm this). So I specified the fully qualified path to my custom home python instance and now my script is working and staying up indefinitely as expected. I also fully qualified the one error exception path as well. ~Andrew -----Original Message----- From: syslog-ng-bounces(a)lists.balabit.hu [mailto:syslog-ng-bounces@lists.balabit.hu] On Behalf Of Sandor Geller Sent: Friday, September 26, 2014 3:49 AM To: syslog-ng(a)lists.balabit.hu Subject: Re: [syslog-ng] syslog-ng and python program destination constantly being restarted again and again Hi, The logs you quoted imply that your custom script keeps dying therefore syslog-ng keeps respawning it. Without seeing the whole script I'd assume that the original problem occurs in the code path where the script tries to forward logs and an expection gets raised (BTW not all exceptions are derived from the Exception class so the exception handler is a bit incomplete but let's put this aside for now). In the exception handler you're trying to open a file containing '~' without using os.path.expanduser() so you end up with an invalid filename which can't get opened. As this exception isn't handled the interpreter aborts the script. So you should eliminate the root cause why the script can't forward logs and also improve the exception handling. hth, Sandor On 09/26/2014 01:12 AM, Andrew Bell wrote: > Hello all, > > Wondering if someone could help me with an issue I've run into > recently regarding the program destination in syslog-ng and a python > script I wrote to automate parsing log events sent over from our Web > Application firewall. For some months now, the script and syslog-ng > configuration were working swimmingly but then things recently started > to stop working and now the script refuses to remain up and forwarding as expected. > > First, here is what and destination looks like in my syslog-ng conf. > The general flow here being ASM events are generated on the firewall > (for whatever purpose), they get sent over to syslog-ng and my python > script then parses these logs and sends them over to another server > running ElasticSearch using the python requests library. This takes > place within the context of an infinite loop in my script - wait for a > log event to come in, execute, go back to waiting, and so forth. > > /destination asm_post {/ > > / program("python -u /home/data/asm_logs/asmlogPost.py" > flush_lines(1) flags(no_multi_line));/ > > /};/ > > I've specified the following flags based off other help forums that > suggested this would help to address the stdin buffer flushing issue > sometimes encountered with syslog-ng and python scripts. There are no > filters being applied here. > > // > > /source(sn_asm);/ > > /destination(asm_post);/ > > Now, my asmLogPost.py script is setup to always read in standard input > through an infinite loop as various best practice syslog-ng sources > have recommended. Here's an excerpt of where the reading in happens - > encapsulated within the context of a try/exception clause: > > /try:/ > > / while 1:/ > > / line = sys.stdin.readline()/ > > / #...read in line log event, log event parsing and POSTing to > other box executes in context of infinite loop...after each POST, it > should go back and wait to read in the next line/ > > /Exception, e:/ > > / # ...exception handling block to catch any errors, write to a file.../ > > / f = open('~/ error.txt','ab')/ > > / f.write('Error happened, here are the details - %s\n'% str(e))/ > > / f.close()/ > > As was said earlier, this has all worked out fine and dandy for quite > some time (and I haven't made any alterations to my script for about a > month or so). But just recently, syslog-ng now appears to be caught in > a loop where it constantly stops, starts, and restarts my script > continuously such that it can't log anything anymore sufficiently. > Also the PID is constantly changing for my script whenever I do a ps > -aef command, which would indicate that a new instance is constantly > being spun up by syslog-ng. > > Not only that, but there apparently appear to be the following > messages sprouting up in /var/log/messages constantly as well which I > think is due to syslog-ng always restarting my script > > /Sep 25 16:28:57 sysa03 abrt: can't communicate with ABRT daemon, is > it running? [Errno 2] No such file or directory/ > > /Sep 25 16:28:57 sysa03 abrt: detected unhandled Python exception in > '/home/data/asm_logs/asmlogPost.py'/ > > /Sep 25 16:28:57 sysa03 abrt: can't communicate with ABRT daemon, is > it running? [Errno 2] No such file or directory/ > > /Sep 25 16:28:57 sysa03 abrt: detected unhandled Python exception in > '/home/data/asm_logs/asmlogPost.py'/ > > /Sep 25 16:28:57 sysa03 abrt: can't communicate with ABRT daemon, is > it running? [Errno 2] No such file or directory/ > > /Sep 25 16:28:57 sysa03 abrt: detected unhandled Python exception in > '/home/data/asm_logs/asmlogPost.py'/ > > /Sep 25 16:28:57 sysa03 abrt: can't communicate with ABRT daemon, is > it running? [Errno 2] No such file or directory/ > > /Sep 25 16:28:57 sysa03 abrt: detected unhandled Python exception in > '/home/data/asm_logs/asmlogPost.py'/ > > /Sep 25 16:28:57 sysa03 abrt: can't communicate with ABRT daemon, is > it running? [Errno 2] No such file or directory/ > > /Sep 25 16:28:57 sysa03 abrt: detected unhandled Python exception in > '/home/data/asm_logs/asmlogPost.py'/ > > /Sep 25 16:28:57 sysa03 abrt: can't communicate with ABRT daemon, is > it running? [Errno 2] No such file or directory/ > > /Sep 25 16:28:58 sysa03 abrt: detected unhandled Python exception in > '/home/data/asm_logs/asmlogPost.py'/ > > /Sep 25 16:28:58 sysa03 abrt: can't communicate with ABRT daemon, is > it running? [Errno 2] No such file or directory/ > > /Sep 25 16:28:58 sysa03 abrt: detected unhandled Python exception in > '/home/data/asm_logs/asmlogPost.py'/ > > /Sep 25 16:28:58 sysa03 abrt: can't communicate with ABRT daemon, is > it running? [Errno 2] No such file or directory/ > > But the thing is, there are no exceptions or error files being > generated by my script even though these messages seem to say this > happening...if there were, my script should catch this and write it > down. Besides, it did used to do this initially as I was developing my > script but it would never halt overall execution or cause syslog-ng to > tear it down, it would just write out to an error.txt file within the same directory. > > // > > Can anyone offer any insight here? I'll admit I'm not familiar with > how the ABRT daemon functions and if it and syslog-ng are somehow > related here or not. And I'm pretty confident that there's nothing > wrong with my script as I am able to execute its current version just > fine from a shell prompt - it can send over the log event and > everything parsed as expected with no errors. I'm kind of at a dead > end, going through my script and these log messages, trying to restart > syslog-ng to refresh, trying to place debug log statements before the > "while 1:" loop, trying to move the while 1: loop to be outside the > try/exception clause, but not much of this seems to be helping so far. > > Many Thanks, > > Andrew > > > > ______________________________________________________________________ > ________ Member info: > https://lists.balabit.hu/mailman/listinfo/syslog-ng > Documentation: > http://www.balabit.com/support/documentation/?product=syslog-ng > FAQ: http://www.balabit.com/wiki/syslog-ng-faq > ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq

1 0

syslog-ng 3.6.0rc1 has been released
by devel＠balabit.hu 26 Sep '14

26 Sep '14

------------------------------------------------------------------------------ PACKAGE : syslog-ng VERSION : 3.6.0rc1 SUMMARY : new release candidate DATE : Sep 26, 2014 ------------------------------------------------------------------------------ DESCRIPTION: A new release candidate of syslog-ng Open Source Edition (3.6.0rc1) has been released. CHANGES: 3.6.0rc1 ======== Fri, 26 Sep 2014 11:17:31 +0200 This is the first Release Candidate of the syslog-ng OSE 3.6 branch. Based on our test results this release is almost production ready. Features -------- * Added groupset rewrite object. Groupset allows the user to modify multiple log message properties at once. It also allows referencing the old value of the property as the $_ macro. Bugfixes -------- * Fixed a memory leak during configuration parsing when using `rewrite()`. * Change control socket message from notice to debug * Fixes for retries() functionality. Retry counter incremented by every message write error (including network connection errors) which can lead to message lost. Credits ------- syslog-ng is developed as a community project, and as such it relies on volunteers, to do the work necessary to produce syslog-ng. Reporting bugs, testing changes, writing code or simply providing feedback are all important contributions, so please if you are a user of syslog-ng, contribute. We would like to thank the following people for their contribution: Balazs Scheidler, Brian De Wolf, Gergely Nagy, Laszlo Budai, Peter Czanik, Tibor Benke, Viktor Juhasz, Viktor Tusa. DOWNLOAD: You can download the source or binary packages from: http://www.balabit.com/downloads/files/syslog-ng/open-source-edition/3.6.0r… The documentation of the syslog-ng Open Source Edition is available in The syslog-ng Open Source Edition Administrator's Guide at http://www.balabit.com/support/documentation/

1 0

Re: [syslog-ng] Injecting environment variables?
by jetjnkr＠gmail.com 26 Sep '14

26 Sep '14

Ok sorry, but still struggling a bit with this: Running with the following version syslog-ng 3.2.5 Installer-Version: 3.2.5 Revision: ssh+git://bazsi@git.balabit//var/scm/git/syslog-ng/syslog-ng-ose--mainline--3.2#master#9d4bea28198bd731df1a61e980a2af5b88d81116 Compile-Date: Jan 15 2012 19:47:30 Enable-Threads: on Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-Sun-STREAMS: off Enable-IPv6: on Enable-Spoof-Source: on Enable-TCP-Wrapper: on Enable-SSL: off Enable-SQL: on Enable-Linux-Caps: off Enable-Pcre: on Enable-Pacct: off I've attempted a number of variations - all have made me sad..... template t_cust { template("CUST $CUST"); }; destination d_web_call { file("/var/log/web_call" template("<$PRI> $DATE $HOST $MSG template($t_cust) \n") template_escape(no)); }; or 'backtick'.... destination d_web_call { file("/var/log/web_call" template("<$PRI> $DATE $HOST $MSG `$CUST` \n") template_escape(no)); }; or... destination d_web_call { file("/var/log/web_call" template("<$PRI> $DATE $HOST $MSG Template("template $CUST")` \n") template_escape(no)); }; $CUST is definitely defined in the environment but the logs don't have any value appended.... Any further guidance appreciated.... On Sep 23, 2014, at 6:00 AM, syslog-ng-request(a)lists.balabit.hu wrote: > ---------------------- > > Message: 1 > Date: Mon, 22 Sep 2014 20:10:17 +0200 > From: Balazs Scheidler <bazsi77(a)gmail.com> > Subject: Re: [syslog-ng] Injecting environment variables? > To: "Syslog-ng users' and developers' mailing list" > <syslog-ng(a)lists.balabit.hu> > Message-ID: > <CAKcfE+a8ohm9rbAjyiBbHSs37iyE_mdP=SRU_Cgz_BSAccFFBg(a)mail.gmail.com> > Content-Type: text/plain; charset="utf-8" > > It came in with blocks and scl support. So if your version has support for > the system driver it will support this as well. My memories are vague with > 3.2. > > Environment variables are resolved at parsing time and should work like > this: > > Template("template ?envvar?") > > It works as if the environment variable was right there in the config file. > On Sep 21, 2014 3:14 PM, <jetjnkr(a)gmail.com> wrote: > >> Balazs, >> >> Thanks, do you have any examples in using the backtick it in a message >> template? Do you have to assign it to a config variable in a "define" >> first? >> >> How much 'earlier' versions of syslog-ng will this work (e.g. before 3.2 >> ?) >> >> Jim. >> >> >>> >>> >>> From: Balazs Scheidler <bazsi77(a)gmail.com> >>> Subject: Re: [syslog-ng] Injecting environment variables? >>> To: "Syslog-ng users' and developers' mailing list" >>> <syslog-ng(a)lists.balabit.hu>, wernli(a)in2p3.fr >>> Message-ID: >>> < >> CAKcfE+ZKqqPnPOOSJek81eq_EPa7y3jUrPyKYH94jc3mxvPsmg(a)mail.gmail.com> >>> Content-Type: text/plain; charset="utf-8" >>> >>> Earlier you could use the backtick env variable format. >>> >>> This one is substituted at config parsing. >>> On Sep 18, 2014 5:26 PM, "Fabien Wernli" <wernli(a)in2p3.fr> wrote: >>> >>>> Hi Jim, >>>> >>>> As of version 3.5 you can use the `env` template function: >>>> >>>> template("<$PRI> $DATE $HOST $MSG $(env <environment-variable>) \n") >>>> template_escape(no)); >>>> >>>> Cheers >>>> >>>> >>>> >> ______________________________________________________________________________ >>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng >>>> Documentation: >>>> http://www.balabit.com/support/documentation/?product=syslog-ng >>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq >>>> >>>> >>>

2 1

syslog-ng and python program destination constantly being restarted again and again
by Andrew Bell 26 Sep '14

26 Sep '14

Hello all, Wondering if someone could help me with an issue I've run into recently regarding the program destination in syslog-ng and a python script I wrote to automate parsing log events sent over from our Web Application firewall. For some months now, the script and syslog-ng configuration were working swimmingly but then things recently started to stop working and now the script refuses to remain up and forwarding as expected. First, here is what and destination looks like in my syslog-ng conf. The general flow here being ASM events are generated on the firewall (for whatever purpose), they get sent over to syslog-ng and my python script then parses these logs and sends them over to another server running ElasticSearch using the python requests library. This takes place within the context of an infinite loop in my script - wait for a log event to come in, execute, go back to waiting, and so forth. destination asm_post { program("python -u /home/data/asm_logs/asmlogPost.py" flush_lines(1) flags(no_multi_line)); }; I've specified the following flags based off other help forums that suggested this would help to address the stdin buffer flushing issue sometimes encountered with syslog-ng and python scripts. There are no filters being applied here. source(sn_asm); destination(asm_post); Now, my asmLogPost.py script is setup to always read in standard input through an infinite loop as various best practice syslog-ng sources have recommended. Here's an excerpt of where the reading in happens - encapsulated within the context of a try/exception clause: try: while 1: line = sys.stdin.readline() #...read in line log event, log event parsing and POSTing to other box executes in context of infinite loop...after each POST, it should go back and wait to read in the next line Exception, e: # ...exception handling block to catch any errors, write to a file... f = open('~/ error.txt','ab') f.write('Error happened, here are the details - %s\n'% str(e)) f.close() As was said earlier, this has all worked out fine and dandy for quite some time (and I haven't made any alterations to my script for about a month or so). But just recently, syslog-ng now appears to be caught in a loop where it constantly stops, starts, and restarts my script continuously such that it can't log anything anymore sufficiently. Also the PID is constantly changing for my script whenever I do a ps -aef command, which would indicate that a new instance is constantly being spun up by syslog-ng. Not only that, but there apparently appear to be the following messages sprouting up in /var/log/messages constantly as well which I think is due to syslog-ng always restarting my script Sep 25 16:28:57 sysa03 abrt: can't communicate with ABRT daemon, is it running? [Errno 2] No such file or directory Sep 25 16:28:57 sysa03 abrt: detected unhandled Python exception in '/home/data/asm_logs/asmlogPost.py' Sep 25 16:28:57 sysa03 abrt: can't communicate with ABRT daemon, is it running? [Errno 2] No such file or directory Sep 25 16:28:57 sysa03 abrt: detected unhandled Python exception in '/home/data/asm_logs/asmlogPost.py' Sep 25 16:28:57 sysa03 abrt: can't communicate with ABRT daemon, is it running? [Errno 2] No such file or directory Sep 25 16:28:57 sysa03 abrt: detected unhandled Python exception in '/home/data/asm_logs/asmlogPost.py' Sep 25 16:28:57 sysa03 abrt: can't communicate with ABRT daemon, is it running? [Errno 2] No such file or directory Sep 25 16:28:57 sysa03 abrt: detected unhandled Python exception in '/home/data/asm_logs/asmlogPost.py' Sep 25 16:28:57 sysa03 abrt: can't communicate with ABRT daemon, is it running? [Errno 2] No such file or directory Sep 25 16:28:57 sysa03 abrt: detected unhandled Python exception in '/home/data/asm_logs/asmlogPost.py' Sep 25 16:28:57 sysa03 abrt: can't communicate with ABRT daemon, is it running? [Errno 2] No such file or directory Sep 25 16:28:58 sysa03 abrt: detected unhandled Python exception in '/home/data/asm_logs/asmlogPost.py' Sep 25 16:28:58 sysa03 abrt: can't communicate with ABRT daemon, is it running? [Errno 2] No such file or directory Sep 25 16:28:58 sysa03 abrt: detected unhandled Python exception in '/home/data/asm_logs/asmlogPost.py' Sep 25 16:28:58 sysa03 abrt: can't communicate with ABRT daemon, is it running? [Errno 2] No such file or directory But the thing is, there are no exceptions or error files being generated by my script even though these messages seem to say this happening...if there were, my script should catch this and write it down. Besides, it did used to do this initially as I was developing my script but it would never halt overall execution or cause syslog-ng to tear it down, it would just write out to an error.txt file within the same directory. Can anyone offer any insight here? I'll admit I'm not familiar with how the ABRT daemon functions and if it and syslog-ng are somehow related here or not. And I'm pretty confident that there's nothing wrong with my script as I am able to execute its current version just fine from a shell prompt - it can send over the log event and everything parsed as expected with no errors. I'm kind of at a dead end, going through my script and these log messages, trying to restart syslog-ng to refresh, trying to place debug log statements before the "while 1:" loop, trying to move the while 1: loop to be outside the try/exception clause, but not much of this seems to be helping so far. Many Thanks, Andrew

2 1

Re: [syslog-ng] Injecting environment variables?
by jetjnkr＠gmail.com 22 Sep '14

22 Sep '14

Balazs, Thanks, do you have any examples in using the backtick it in a message template? Do you have to assign it to a config variable in a "define" first? How much 'earlier' versions of syslog-ng will this work (e.g. before 3.2 ?) Jim. > > > From: Balazs Scheidler <bazsi77(a)gmail.com> > Subject: Re: [syslog-ng] Injecting environment variables? > To: "Syslog-ng users' and developers' mailing list" > <syslog-ng(a)lists.balabit.hu>, wernli(a)in2p3.fr > Message-ID: > <CAKcfE+ZKqqPnPOOSJek81eq_EPa7y3jUrPyKYH94jc3mxvPsmg(a)mail.gmail.com> > Content-Type: text/plain; charset="utf-8" > > Earlier you could use the backtick env variable format. > > This one is substituted at config parsing. > On Sep 18, 2014 5:26 PM, "Fabien Wernli" <wernli(a)in2p3.fr> wrote: > >> Hi Jim, >> >> As of version 3.5 you can use the `env` template function: >> >> template("<$PRI> $DATE $HOST $MSG $(env <environment-variable>) \n") >> template_escape(no)); >> >> Cheers >> >> >> ______________________________________________________________________________ >> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng >> Documentation: >> http://www.balabit.com/support/documentation/?product=syslog-ng >> FAQ: http://www.balabit.com/wiki/syslog-ng-faq >> >> >

2 1

Re: [syslog-ng] syslog-ng Digest, Vol 113, Issue 9
by jetjnkr＠gmail.com 20 Sep '14

20 Sep '14

Hi Famien, Thanks - just need to get the Ops team to push out 3.5 and I'll give it a shot! Jim. On Sep 19, 2014, at 2:46 AM, syslog-ng-request(a)lists.balabit.hu wrote: > > Date: Thu, 18 Sep 2014 17:26:33 +0200 > From: Fabien Wernli <wernli(a)in2p3.fr> > Subject: Re: [syslog-ng] Injecting environment variables? > To: "Syslog-ng users' and developers' mailing list" > <syslog-ng(a)lists.balabit.hu> > Message-ID: <20140918152633.GC6867(a)ccfawe.in2p3.fr> > Content-Type: text/plain; charset=us-ascii > > Hi Jim, > > As of version 3.5 you can use the `env` template function: > > template("<$PRI> $DATE $HOST $MSG $(env <environment-variable>) \n") template_escape(no)); > > Cheers > > >

1 0

Injecting environment variables?
by jetjnkr＠gmail.com 20 Sep '14

20 Sep '14

New to the list, but looking for some pointers: It is possible to get shell variables interpreted and defined in the context of the syslog-ng.conf file? I'm trying to get the value from a shell/environment variable to be added to a logged message. I've tried various forms of templates in the destination without much success. template("<$PRI> $DATE $HOST $MSG $someENVvar \n") template_escape(no)); Static text works find, but the var ends up being blank... For reasons I won't belabor, I can't use tags as added by the source. Apologies if this has been covered previous - I didn't see anything from scanning the archives. Thanks. Jim.

3 2

classify/assign events to tenants/customers
by Lucas, Sascha 19 Sep '14

19 Sep '14

Hi, I'm looking for a way to "classify"/assign events to tenants/customers. A tenant/customer can be all events belonging to host or a combination of host/program. I thought patterndb can help doing this. However it seems there is no way? * classification: pattern is per program and limited to the message * correlation: context is limited to one host not many Do I miss something, or are there even better ways? Thanks, Sascha. Aufsichtsratsvorsitzender: Herbert Vogel Geschäftsführung: Michael Krüger Sitz der Gesellschaft: Halle/Saale Registergericht: Amtsgericht Stendal | Handelsregister-Nr. HRB 208414 UST-ID-Nr. DE 158253683 Diese E-Mail enthält vertrauliche und/oder rechtlich geschützte Informationen. Wenn Sie nicht der richtige Empfänger sind oder diese E-Mail irrtümlich erhalten haben, informieren Sie bitte sofort den Absender und vernichten Sie diese Mail. Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser Mail oder des Inhalts dieser Mail sind nicht gestattet. Diese Kommunikation per E-Mail ist nicht gegen den Zugriff durch Dritte geschützt. Die GISA GmbH haftet ausdrücklich nicht für den Inhalt und die Vollständigkeit von E-Mails und den gegebenenfalls daraus entstehenden Schaden. Sollte trotz der bestehenden Viren-Schutzprogramme durch diese E-Mail ein Virus in Ihr System gelangen, so haftet die GISA GmbH - soweit gesetzlich zulässig - nicht für die hieraus entstehenden Schäden.

3 3

insider 2014-09: 3.6 beta; eCSI; DevOps; anonymization; GSoC;
by Czanik, Péter 19 Sep '14

19 Sep '14

Dear syslog-ng users, This is the 37th issue of the syslog-ng Insider, a monthly newsletter that brings you syslog-ng related news. FEATURED NEWS Beta testing of syslog-ng 3.6 started ------------------------------------- Beta testing period of syslog-ng 3.6 started two weeks ago with a beta1 release. It was quickly followed by a beta2 release, fixing mostly portability. Some of the new features are coming from the syslog-ng incubator, like Riemann support or the graphite template function. The journal source and many enhancements to existing features are coming from the syslog-ng PE team. Also, the @cim prefix can be used as a local "rich" log transport on /dev/log, and messages will be parsed by the JSON parser if available. Beta2 is already available in the FreeBSD ports in sysutils/syslog-ng-devel. For RPM distributions, check https://czanik.blogs.balabit.com/2014/09/journal-source-in-syslog-ng-3-6-be… BalaBit syslog-ng team takes over the syslog-ng OSE development --------------------------------------------------------------- algernon, the full-time syslog-ng OSE developer at BalaBit decided to look for new challenges. First of all, thank you very much for all your hard work, bugfixes nifty new features, and everything else you did for syslog-ng OSE! We all wish you the best, and hope you'll have a great time outside BalaBit as well :) To keep the development, maintenance, and releases of syslog-ng OSE on track, the developer team of syslog-ng Premium Edition will take over the tasks related to syslog-ng OSE: they will manage bugfixes, patches, pull requests, and also the general development of syslog-ng OSE. Naturally, this does not affect the current or future openness of syslog-ng OSE in any way: Your contribution is as welcome as ever. For more details about this change, see algernon's blog post at http://asylum.madhouse-project.org/blog/2014/09/15/looking-ahead/ eCSI training ------------- BalaBit provides now a free training, which provides a fresh insight on logmanagement. The first level introduces the listener to compliance, planning an infrastructure and logs in forensics situations. Once your knowledge is tested, you can go to the next level and learn about IT security and eCSI: http://www.balabit.com/lp/ecsi-training-camp syslog-ng, riemann, collectd-notifications, elasticsearch --------------------------------------------------------- How to build an event-based infrastructure to push structured messages to different subsystems for alerting, reporting and storage. Using syslog-ng, each message is normalized into a structured event, optionally correlated with other messages, and conditionally routed to systems. Read more at: http://devops.com/features/guide-modern-monitoring-alerting/ Data Privacy, Anonymization and Log Data ---------------------------------------- Strong data privacy laws are arriving slowly but surely in Europe, which also affects logging. Raw data contains too much information, on the other hand anonymized data does not have enough information to handle a security incident. Read about a possible solution and how syslog-ng can help at https://jluby.blogs.balabit.com/2014/09/10/data-privacy-anonymization-and-l… Google Summer of Code: success ------------------------------ Google Summer of Code ended a few weeks ago. All of our students successfully completed their projects. We would like to thank for the hard work of students and their mentors, and Google for the opportunity! You can read more about the completed projects at https://www.google-melange.com/gsoc/org2/google/gsoc2014/syslog_ng NEW RELEASES syslog-ng OSE 3.6 beta2: https://lists.balabit.hu/pipermail/syslog-ng-announce/2014-September/000197… syslog-ng PE 5LTS (5.0.6a): https://lists.balabit.hu/pipermail/syslog-ng-announce/2014-September/000196… Your feedback and news tips about the next issue is welcome at documentation(a)balabit.com To read this newsletter on-line, visit: http://insider.blogs.balabit.com/ Peter Czanik (CzP) <peter.czanik(a)balabit.com> BalaBit IT Security / syslog-ng upstream http://czanik.blogs.balabit.com/ https://twitter.com/PCzanik

1 0

Re: [syslog-ng] classify/assign events to tenants/customers
by Jim Hendrick 18 Sep '14

18 Sep '14

I would try basic filters on host (or netmask) plus facility, priority and if necessary program or "match". Jim Sent from my Verizon Wireless 4G LTE smartphone -------- Original message -------- From: "Lucas, Sascha" <Sascha.Lucas(a)gisa.de> Date:09/18/2014 2:53 AM (GMT-05:00) To: Syslog-ng users' and developers' mailing list <syslog-ng(a)lists.balabit.hu> Cc: Subject: [syslog-ng] classify/assign events to tenants/customers Hi, I'm looking for a way to "classify"/assign events to tenants/customers. A tenant/customer can be all events belonging to host or a combination of host/program. I thought patterndb can help doing this. However it seems there is no way? * classification: pattern is per program and limited to the message * correlation: context is limited to one host not many Do I miss something, or are there even better ways? Thanks, Sascha. Aufsichtsratsvorsitzender: Herbert Vogel Geschäftsführung: Michael Krüger Sitz der Gesellschaft: Halle/Saale Registergericht: Amtsgericht Stendal | Handelsregister-Nr. HRB 208414 UST-ID-Nr. DE 158253683 Diese E-Mail enthält vertrauliche und/oder rechtlich geschützte Informationen. Wenn Sie nicht der richtige Empfänger sind oder diese E-Mail irrtümlich erhalten haben, informieren Sie bitte sofort den Absender und vernichten Sie diese Mail. Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser Mail oder des Inhalts dieser Mail sind nicht gestattet. Diese Kommunikation per E-Mail ist nicht gegen den Zugriff durch Dritte geschützt. Die GISA GmbH haftet ausdrücklich nicht für den Inhalt und die Vollständigkeit von E-Mails und den gegebenenfalls daraus entstehenden Schaden. Sollte trotz der bestehenden Viren-Schutzprogramme durch diese E-Mail ein Virus in Ihr System gelangen, so haftet die GISA GmbH - soweit gesetzlich zulässig - nicht für die hieraus entstehenden Schäden. ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq

1 0

3.6 beta2 rpm packages, FreeBSD ports
by Czanik, Péter 16 Sep '14

16 Sep '14

Hi, A bit later than usual (I was on sick leave), but my 3.6 beta2 packages are ready for openSUSE/SLES and Fedora/EPEL7. They are available at the usual locations: openSUSE/SLES: https://build.opensuse.org/project/show/home:czanik:syslog-ng36 Fedora/EPEL7: http://copr.fedoraproject.org/coprs/czanik/syslog-ng36/ Thanks to Cy, the sysutils/syslog-ng-devel port in FreeBSD was also updated to the latest beta, while I was unavailable. And this version also works on earlier releases, including FreeBSD 8.4. As usual, these packages are not yet recommended for production use, but any testing and feedback are very welcome! Bye, Peter Czanik (CzP) <peter.czanik(a)balabit.com> BalaBit IT Security / syslog-ng upstream http://czanik.blogs.balabit.com/ https://twitter.com/PCzanik

1 0

Help settings regarding Syslog NG
by Simon OBOUNOU 15 Sep '14

15 Sep '14

Hi I need helps regarding setting between syslog ng server and security solutions like checkpoint firewall, stormshield IDS and so one. -- Bien cordialement, Kind Regards HIFA, Chairman 32 rue de la République 92190 MEUDON - FRANCE Phone: +33 1 46 31 44 25 Mobile: +33 6 11 30 36 57 email: simon.obounou(a)hifa.biz

2 1

syslog-ng 3.6.0beta2 has been released
by devel＠balabit.hu 11 Sep '14

11 Sep '14

------------------------------------------------------------------------------ PACKAGE : syslog-ng VERSION : 3.6.0beta2 SUMMARY : new beta release DATE : Sep 11, 2014 ------------------------------------------------------------------------------ DESCRIPTION: A new beta version of syslog-ng Open Source Edition (3.6.0beta2) has been released. Being the - hopefully - last beta release, we would be very grateful for any feedback, to stabilize the release further. CHANGES: 3.6.0beta2 ========== Thu, 11 Sep 2014 13:09:57 +0200 This is the second beta release of the upcoming syslog-ng OSE 3.6 branch. Compared to the previous beta, this release contains a few minor features and bugfixes. We expect the next release to be a release candidate, focusing on stability and bugfixes. Testing is most appreciated! Features -------- * It is now possible to load not only a single certificate when using TLS, but a certificate chain. * The `system()` source will not include `/dev/kmsg` (or `/proc/kmsg`) when running inside a Linux container. Bugfixes -------- * The `in-list()` filter was fixed to look at all elements of the list, instead of only the last one. * Fixed an assertion when using the `match()` filter under certain circumstances. * The `system()` source will not add `/dev/kmsg` (or `/proc/kmsg` on older kernels) to the default sources if using the systemd journal, because kernel logs are included in the journal. Credits ------- syslog-ng is developed as a community project, and as such it relies on volunteers, to do the work necessary to produce syslog-ng. Reporting bugs, testing changes, writing code or simply providing feedback are all important contributions, so please if you are a user of syslog-ng, contribute. We would like to thank the following people for their contribution: Andras Mitzki, Balazs Scheidler, Gergely Nagy, Gyorgy Pasztor, Peter Czanik, Tibor Benke. DOWNLOAD: You can download the source or binary packages from: http://www.balabit.com/downloads/files/syslog-ng/open-source-edition/3.6.0b… The documentation of the syslog-ng Open Source Edition is available in The syslog-ng Open Source Edition Administrator's Guide at http://www.balabit.com/support/documentation/

1 0

Re: [syslog-ng] Individual Logs Files to each Forward to Different server/port?
by Jim Hendrick 08 Sep '14

08 Sep '14

Quick thought. Look at filters on the receiving end. Might be easier. Jim Sent from my Verizon Wireless 4G LTE smartphone -------- Original message -------- From: wiskbroom(a)hotmail.com Date:09/05/2014 5:42 PM (GMT-05:00) To: syslog-ng(a)lists.balabit.hu Cc: Subject: [syslog-ng] Individual Logs Files to each Forward to Different server/port? Greetings! Is is possible to configure a syslog-ng client to forward logs to a syslog-ng server based on file? I am thinking of the following as an example: destination named-LOGS { udp(ip(192.168.1.100) port(555)); }; source named { file("/var/log/named/bind.log" log_prefix("BIND-LOGS")); }; log { source(named); destination(named-LOGS); }; destination dhcpd-LOGS { udp(ip(192.168.1.100) port(556)); }; source dhcpd { file("/var/log/dhcpd/dhcp.log" log_prefix("DHCPD-LOGS")); }; log { source(dhcpd); destination(dhcpd-LOGS); }; In this example, I am sending each to the same destination IP address, although that is configurable, but each log file to a different port, and with a different log_prefix as well. Does this OK,or is their a simpler way? Many thanks, .vp Vadim Anatoly Pushkin -- The Ukranian Stallion --

2 2