- syslog-ng - lists.balabit.hu

Syslog-ng time stamp reporting issue.
by vijay amruth 24 Mar '17

24 Mar '17

Hello guys, Hope everyone is doing great. I have run into a issue where syslog-ng reports time in MST when the source is in UTC and we want in UTC, how do we control this ? Source (UTC) --------> Syslog-ng server (UTC) but syslog is writing logs in MST. Any help is appreciated. Thank you, Vijay Amrut.

2 1

Re: [syslog-ng] Is it possible to include a external python script?
by Jorge Pereira 22 Mar '17

22 Mar '17

Hi Balázs, Thanks for the answer. But, My question is: It is possible to include a external python script. e.g: $ cat foobar.conf: python { @include "caipirinha.py" } template-function foobar "$(python foobar)" $ cat caipirinha.py from foobar import foobar_template_function # my python class/code is separated ........... $ got it? maybe using the "confgen" module? -- Jorge Pereira On Mon, Mar 20, 2017 at 2:35 AM, Scheidler, Balázs < balazs.scheidler(a)balabit.com> wrote: > Hi, > > The example Python code should be unindented, or you see a different error? > > Also, you can always import code in the python block. > > With destinations you can specify a path to a python file, so the python > block is not needed. But with template functions I've figured the $(python) > call would be too crowded if we were passing explicit filename parameters. > > But you can encapsulate the python code as well as a wrapper to the python > code into a separate file. > > Foobar.conf: > > python { > > from foobar import foobar_template_function > > } > > template-function foobar "$(python foobar)" > > ---------- > > Then you can use: > > @include "Foobar.conf" > > ... > template("$(foobar)"); > ... > > Unfortunately then template function declarations cant take parameters yet > from the callsite, which may be a serious limitation in your use-case. But > with a conceot above you can completely hide implementation details your > actual config. > > On Mar 20, 2017 03:53, "Jorge Pereira" <jpereiran(a)gmail.com> wrote: > >> Well, based on this[1] document. We should use the *python {}* statement. >> But, I would like to keep my python script externally in a separated file >> and do something like. >> >> python filter_001 { >> /my/script/python001.py >> } >> >> log { >> .... >> filter(filter_001); >> .... >> }; >> >> p.s: Btw, this sample https://www.balabit.com >> /documents/syslog-ng-ose-3.7-guides/en/syslog-ng-ose-guide- >> admin/html-single/index.html#example-python-template-functions >> don't work in the syslog-ng 3.9.1 >> >> [1] https://www.balabit.com/sites/default/files/documents/syslog >> -ng-ose-3.7-guides/en/syslog-ng-ose-v3.7-guide-admin/html-si >> ngle/index.html#example-python-template-functions >> -- >> Jorge Pereira >> >> ____________________________________________________________ >> __________________ >> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng >> Documentation: http://www.balabit.com/support/documentation/?product= >> syslog-ng >> FAQ: http://www.balabit.com/wiki/syslog-ng-faq >> >> >> > ____________________________________________________________ > __________________ > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng > Documentation: http://www.balabit.com/support/documentation/? > product=syslog-ng > FAQ: http://www.balabit.com/wiki/syslog-ng-faq > > >

4 6

Is it possible to include a external python script?
by Jorge Pereira 20 Mar '17

20 Mar '17

Well, based on this[1] document. We should use the *python {}* statement. But, I would like to keep my python script externally in a separated file and do something like. python filter_001 { /my/script/python001.py } log { .... filter(filter_001); .... }; p.s: Btw, this sample https://www.balabit.com/documents/syslog-ng-ose-3.7-guides/en/syslog-ng-ose… don't work in the syslog-ng 3.9.1 [1] https://www.balabit.com/sites/default/files/documents/ syslog-ng-ose-3.7-guides/en/syslog-ng-ose-v3.7-guide- admin/html-single/index.html#example-python-template-functions -- Jorge Pereira

2 1

syslog-ng 3.9.x & mod-lua is deprecated?
by Jorge Pereira 20 Mar '17

20 Mar '17

Hi, syslog-ng 3.9.x & mod-lua is deprecated? btw, is it possible to enable the mod-lua in the 3.9.x? I am asking because using Lua is more easy to port the Lua scripts/decoder/encoders from the heka to syslog-ng. -- Jorge Pereira

2 2

Insider 2017-03: Docker; Central Log Management; Hadoop & Kafka performance; SCALE 2017;
by Czanik, Péter 16 Mar '17

16 Mar '17

Dear syslog-ng users, This is the 56th issue of syslog-ng Insider, a monthly newsletter that brings you syslog-ng-related news. NEWS Docker: 500K+ pulls ------------------- There are over 500K pulls for the syslog-ng image from the Docker hub at https://hub.docker.com/u/balabit/. You can use one of the -dev images if you want to compile syslog-ng yourself. If you use any of these images, share your use case and experiences with us. Gartner & RSA: Central Log Management ------------------------------------- With the increased reliance on analytics to drive security operations, IT security professionals are paying particular attention to one of the oldest tools in their toolbox, log management. A recently released report by Gartner focuses on the role of Central Log Management (CLM) in improving threat monitoring and detection: https://www.balabit.com/blog/renaissance-central-log-management/ The key takaway from the RSA conference was also CLM: https://www.balabit.com/blog/key-takeaways-about-central-log-management-fro… Upcoming webinar about CLM: https://pages2.balabit.com/central-log-management-webinar/ Hadoop and Kafka performance ---------------------------- The syslog-ng application has been supporting Hadoop and Kafka destinations for some time now, as it is an integral part of a versatile central log management solution. Seamless integration is great, but what about performance? Read our blogs to learn more: Hadoop: https://www.balabit.com/blog/testing-performace-log-streaming-hdfs-syslog-n… Kafka: https://www.balabit.com/blog/testing-the-performance-of-log-streaming-to-ka… SCALE 2017 ---------- It is the third year that syslog-ng has participated at Southern California Linux Expo or, as better known to many, SCALE ‒ the largest Linux event in the USA. Balabit had a booth in the expo area and on Sunday I gave a presentation on how to deal with security logs using syslog-ng. Read more at https://www.balabit.com/blog/syslog-ng-scale-2017/ Your feedback and news, or tips about the next issue are welcome at documentation(a)balabit.com. To read this newsletter online, visit: https://syslog-ng.org/ Peter Czanik (CzP) <peter.czanik(a)balabit.com> Balabit / syslog-ng upstream https://www.balabit.com/blog/author/peterczanik/ https://twitter.com/PCzanik

1 0

hashing and verify
by Domenico Vitali 15 Mar '17

15 Mar '17

Dear all, I am writing a syslog-ng configuration able to rewrite (or templating) the MESSAGE macro of a syslog entry in such way that it includes its own hash. Consider for example the line: Mar 15 08:23:15 void systemd[1]: Started System Logger Daemon. i would like to rewrite the message obtaining this output: Mar 15 08:23:15 void systemd[1]: Started System Logger Daemon. hash: 87602bdc780a764ae26f30ddc3f09176 where "8ba7a67ce6ac67b9facb6a14b3095960" is the hash of the line (md5, sha1 or other hash funcition). I evaluate both the use of the template function as well as the use of the rewrite module. Unfortunatelly, once i got the line with the hash i can't understand how to verify if the hash is correct. In contrast, when I evalute the hash using the command line, the hash values mismatch. $ echo "Mar 15 08:23:15 void systemd[1]: Started System Logger Daemon." | md5sum - 8ba7a67ce6ac67b9facb6a14b3095960 - In order to perform my test I'm using this configuration: --------------------------------------------------------------------------- source sourcetest { file("/var/log/md5/input.log" flags(no-parse)); }; rewrite msghashing { set("$(md5 $MESSAGE)", value("MYMESS")); set ("-$MESSAGE- hash: $MYMESS", value("MESSAGE"), on-error("fallback-to-string")); }; destination testdestination { file("/var/log/md5/output.log"); }; log { source( sourcetest ); rewrite( msghashing ); # rewrite( r_rewrite_host ); destination( testdestination ); }; --------------------------------------------------------------------------- i can't find any hint in the documentation files, does anybody knows how can i correctly verify the hash ? Is the syslog-ng software using a salt by default ? thanks a lot, -- Domenico

2 1

syslog-ng in docker
by Czanik, Péter 13 Mar '17

13 Mar '17

Hi, Recently I figured out, that there are 500K+ pulls for syslog-ng from the docker hub: https://hub.docker.com/u/balabit/ On the other hand I don't know a single actual user of it yet. If you are one of them, let me know your use case and experiences! Also, if you are on Twitter, help spreading the word and re-tweet: https://twitter.com/PCzanik/status/841192391512059906 Thanks, Peter Czanik (CzP) <peter.czanik(a)balabit.com> Balabit / syslog-ng upstream https://www.balabit.com/blog/author/peterczanik/ https://twitter.com/PCzanik

1 0

Increase file descriptors for syslog-ng.
by vijay amruth 11 Mar '17

11 Mar '17

Hello everyone, Hope you are all doing great. I am running into some issues with syslog-ng file descriptors, I have tried to set up ulimit -n value to 65535, *open files (-n) 65535* set the parameter to the follwoing in /etc/sysctl.conf, *fs.file-max = 512000* and tried raising the value with syslog-ng --fd-limit <number>, I get the following error ####syslog-ng --fd-limit 8192 WARNING: Configuration file format is too old, syslog-ng is running in compatibility mode Please update it to use the syslog-ng 3.5 format at your time of convinience, compatibility mode can operate less efficiently in some cases. To upgrade the configuration, please review the warnings about incompatible changes printed by syslog-ng, and once completed change the @version header at the top of the configuration file.; WARNING: global: the default value of log_fifo_size() has changed to 10000 in syslog-ng 3.3 to reflect log_iw_size() changes for tcp()/udp() window size changes; Using a global log-fetch-limit() option was removed, please use a per-source log-fetch-limit(); WARNING: window sizing for tcp sources were changed in syslog-ng 3.3, the configuration value was divided by the value of max-connections(). The result was too small, clamping to 100 entries. Ensure you have a proper log_fifo_size setting to avoid message loss.; orig_log_iw_size='0', new_log_iw_size='100', min_log_fifo_size='1300000' and I am unable to stop the syslog-ng service using systemctl, it immediately re-starts, I am using centOS 7.2, I have noticed few other people on the internet had the same problem, please advice, any help is appreciated. *OS: CentOS 7.2* *Hardware : VM* *Problems : 1) Unable to increase the file descriptors for syslog-ng. 2) Unable to stop the service using systemctl stop command, it automatically restarts right after with a new pid.* Thanks, Vijay Amrut.

4 4

Can't get basic syslog to work for my firewall logs?
by Tim Tyler 10 Mar '17

10 Mar '17

Syslog-ng experts. I am very new to syslog-ng. I installed syslog-ng on a fresh Redhat 7.3 server. It defaults working with internal logging. So I configured my firewall to send syslog with facility set to log_user. I turned on Wireshark on the syslog-ng server and observed the firewall sending traffic to the server on udp 514. But the syslog server never created the directory structure and logs. I disabled the redhat firewall just to eliminate it as a possibility. Still no logging. So I don’t know what I am doing wrong at this point. I don’t know if this is a permission problem or some other configuration issue. I found someone that had posted a very basic syslog-ng configuration for firewalls. So I copied It into a firewall.conf I put in conf.d. Can anyone see what might be wrong with it? #################### options { create_dirs(yes); owner(root); group(root); perm(0640); dir_owner(root); dir_group(root); dir_perm(0750); }; ################################################## source s_udp { udp(port(514)); }; #Template for a new firewall in the firewalls.conf file #Entries to be changed: NAMEOFTHEFIREWALL and IPOFTHEFIREWALL ################################################## filter f_NAMEOFTHEFIREWALL { host("192.168.30.1"); }; destination d_NAMEOFTHEFIREWALL { file("/var/log/firewalls/PA/$YEAR/$MONTH/$YEAR-$MONTH-$DAY.PA.log"); }; log { source(s_udp); filter(f_NAMEOFTHEFIREWALL); destination(d_NAMEOFTHEFIREWALL); }; Tim Tyler Network Engineer Beloit College

6 8

syslog-ng lock ups
by Evan Rempel 07 Mar '17

07 Mar '17

I have been having a problem with syslog-ng where is just stops processing all input. There are destinations that receive and count all of the messages that come out of syslog-ng and they stop getting any messages. This is occurring on two syslog servers that have similar configurations. One is a superset of the simpler one. On these hosts we run two syslog-ng instances. One for the regular OS log messages from /dev/lo, /proc/kmsg, localhost:1514 and the second one which only listens on the network port(s) which we call our syslog server. The syslog server has its internal log messages going to localhost:1514 so any internal events such as new connections etc should be logged to the "OS syslog" instance. We had this problem on 3.7.x and after upgrading to 3.9.1 we still encounter this problem. The OS is now the latest Redhat 6 (6.8), but the same problem occurred on any Redhat 6.x system. Our host monitoring shows that the syslog-ng process did NOT increase its memory footprint which indicates that it stopped reading its source. All of the hosts that send logs to syslog server showed increased message queueing which confirms that the source stopped being read. The CPU consumption by the syslog-ng process is zero during this time period. Our network monitoring infrastructure opens a connection to the syslog server every 30 seconds. This gets logged via the localhost:1514 connection. These accepted and closed log lines are missing during our problem window even though the monitoring tool WAS able to connect through the entire problem window. We have configured the syslog stats log line to be sent to an external program which invokes syslog-ng-ctl stats and it processes this data. This process did not get any such stats line, so the exact syslog-ng stats counters are unavailable during this problem window. The syslog-ng-ctl program was NOT hung on the socket. Attempting a graceful shutdown of the syslog-ng process either with syslog-ng-ctl or with a kill -TERM appears to have no affect. I assume this is because syslog-ng is unable to flush its buffers so it does not terminate. Killing syslog-ng and restarting it starts processing again correctly. Has anyone else had this symptom? Has anyone found a solution to this? I realize that this is all anecdotal but was hoping it would trigger someone's memory. Thanks in advance. Evan.

2 1