- syslog-ng - lists.balabit.hu

unknown symbol errors in jounalctl -xe
by vijay amruth 26 Jul '17

26 Jul '17

Hello All, hope everyone is doing great, I am unable to start syslog-ng, I see this messages... Any help is appreciated. Jul 25 11:36:14 server-1systemd[1]: Unit syslog.socket entered failed state. Jul 25 11:36:14 server-1systemd[1]: Unit syslog-ng.service entered failed state. Jul 25 11:36:14 server-1systemd[1]: syslog-ng.service failed. Jul 25 11:36:18 server-1kernel: dccp_diag: Unknown symbol dccp_hashinfo (err 0) Jul 25 11:36:18 server-1kernel: dccp_diag: Unknown symbol dccp_hashinfo (err 0) Jul 25 11:36:48 server-1systemd[1]: Listening on Syslog Socket. [root@server-1 tmp]# systemctl status syslog-ng.service ● syslog-ng.service - System Logger Daemon Loaded: loaded (/usr/lib/systemd/system/syslog-ng.service; enabled; vendor preset: enabled) Active: activating (start) since Tue 2017-07-25 11:44:20 GMT+7; 36s ago Docs: man:syslog-ng(8) Main PID: 21822 (syslog-ng) CGroup: /system.slice/syslog-ng.service └─21822 /usr/sbin/syslog-ng -F -p /var/run/syslogd.pid --fd-limit 50000 Jul 25 11:44:20 lvsinfosecsplunkcollector-1137760 systemd[1]: Starting System Logger Daemon... [root@server-1 tmp]# syslog-ng -V syslog-ng 3.9.1 Installer-Version: 3.9.1 Revision: Module-Directory: /usr/local/lib/syslog-ng Module-Path: /usr/local/lib/syslog-ng Available-Modules: date,json-plugin,syslogformat,add-contextual-data,affile,afprog,confgen,cryptofuncs,pseudofile,sdjournal,csvparser,graphite,cef,afmongodb,basicfuncs,system-source,afuser,dbparser,afstomp,afsocket,disk-buffer,kvformat,afamqp,linux-kmsg-format Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-IPv6: on Enable-Spoof-Source: off Enable-TCP-Wrapper: off Enable-Linux-Caps: off -- Thanks, Vijay Amrut.

2 2

Windows Events multi-line
by Smith, Paul (Sr. Admin-InfoSec) 25 Jul '17

25 Jul '17

Hi Nagy, Thanks for checking that. That was probably one of the reasons it did not work. That is too bad since Windows servers using the syslog-ng agent are the source. I would have thought that would be working since bot products are from Balabit. Windows logs also send a lot of garbage with Information lines that it would also be good to remove since that is just going to fill up the log files with junk. I am getting 8-10 gigs per day from a single Windows Domain Controller and I need to add in about 25 more. The log file would be immense. I tested out using the syslog driver and was poking around with the multi-line-prefix and multi-line-garbage and they don't give any syntax errors, as long as I don't use the multi-line-mode option. Does syslog-ng ignore these options if the multi-line-mode option is not specified? Basically, do all the multi-line-xxx options not work like this? It seems like the only option that works for syslog or network is the flags(no-multi-line) flag. Is that the case? Are there any other alternatives? I would hate to write the syslogs to a file and then parse that file again to get the logs I want. That is double work for the server. But that seems like that might be the only option I have. Hmmm...maybe I can log to a VM syslog-ng and have that one parse the data first using the file driver and then I can relay to my main systems. Paul C Smith -----Original Message----- From: syslog-ng [mailto:syslog-ng-bounces@lists.balabit.hu] On Behalf Of syslog-ng-request(a)lists.balabit.hu Sent: Thursday, July 13, 2017 7:00 AM To: syslog-ng(a)lists.balabit.hu Subject: syslog-ng Digest, Vol 147, Issue 14 Send syslog-ng mailing list submissions to syslog-ng(a)lists.balabit.hu To subscribe or unsubscribe via the World Wide Web, visit https://lists.balabit.hu/mailman/listinfo/syslog-ng or, via email, send a message with subject or body 'help' to syslog-ng-request(a)lists.balabit.hu You can reach the person managing the list at syslog-ng-owner(a)lists.balabit.hu When replying, please edit your Subject line so it is more specific than "Re: Contents of syslog-ng digest..." Today's Topics: 1. Re: Windows Events multi-line (Nagy) ---------------------------------------------------------------------- Message: 1 Date: Thu, 13 Jul 2017 10:50:03 +0200 From: Nagy, Gábor <gabor.nagy(a)balabit.com> To: "Syslog-ng users' and developers' mailing list" <syslog-ng(a)lists.balabit.hu> Subject: Re: [syslog-ng] Windows Events multi-line Message-ID: <CAETAYnCft3KAyWF_5pB3g4+jodrn6+SpXTYJqqyfnFoUFzzaEw(a)mail.gmail.com> Content-Type: text/plain; charset="utf-8" Hello, Checking your configuration I see that you are using multi-line mode for a network source driver. Unfortunately, currently this is not supported, only the file() and pipe() source drivers support multi-line messages. I have checked the documentation and found that although it is mentioned which source drivers support multi-line mode (under flags option "no-multi-line"), the multi-line-* options are included in the network source driver options page. This is a documentation bug, which will be fixed soon. Sorry for the inconveniences. Best Regards, Gabor On Mon, Jul 10, 2017 at 8:15 PM, Smith, Paul (Sr. Admin-InfoSec) < Paul.C.Smith(a)snapon.com> wrote: > Hello, > > > > I’m having problems getting Windows events on a single line on > syslog-ng OSE. I’ve scoured the interwebs and not found what I need to > get this exact. I am guessing this is not an uncommon problem but I > can’t seem to find quite what I need. I am guessing I am just missing > some simple thing here. > > > > Here are my details. > > > > Using syslog-ng OSE 3.9.1 > > > > Have syslog-ng Windows Agent 6.0.6 running on Windows 2012 server > > > > Have a tcp source and that writes direct to the log file. Works fine > with no options set. > > > > Getting multiple lines per event. > > > > I’ve added what I think are the correct settings for multi-line, it > does not work. I don’t think it is the regex syntax, but something else? > > > > ___________ > > Syslog.ng.conf > > > > source s_nettcp_win { network(ip(x.x.x.x) port(601) transport("tcp") > > multi-line-mode(regexp) > > multi-line-prefix("^[0-9]{3,5} > \s<[0-9]{2,3}>1\s([0-9]{4})-(0[1-9]|1[0-2])-(0[1-9]|[1-2][ > 0-9]|3[0-1])T([0-9]{2}:[0-9]{2}:[0-9]{2}-[0-9]{2}:[0-9]{2})") > > flags(no-parse)); }; > > > > ___________ > > Here is the error when trying to start syslog-ng or run syslog-ng –s: > > > > Error parsing afsocket, syntax error, unexpected LL_IDENTIFIER, > expecting KW_NORMALIZE_HOSTNAMES or KW_USE_DNS or KW_USE_FQDN or > KW_DNS_CACHE in /etc/syslog-ng/syslog-ng.conf at line 38, column 76: > > > > source s_nettcp_win { network(ip(x.x.x.x) port(601) transport("tcp") > multi-line-mode(regexp) multi-line-prefix("^[0-9]{3,5} > \s<[0-9]{2,3}>1\s([0-9]{4})-(0[1-9]|1[0-2])-(0[1-9]|[1-2][ > 0-9]|3[0-1])T([0-9]{2}:[0-9]{2}:[0-9]{2}-[0-9]{2}:[0-9]{2})") > flags(no-parse)); }; > > > ^^^^^^^^^^^^^^^ > > ___________ > > Sample of the event log part I am matching regex on: > > > > Jul 10 12:11:19 x.x.x.x 912 <133>1 2017-07-10T12:11:18-05:00 > computername Microsoft_Windows_security_auditing. 6260 - [win(a)18372.4 > EVENT_CATEGORY="Logoff" EVENT_FACILITY="16" EVENT_ID="4634" EVENT_LEVEL="0" > EVENT_NAME="Security" EVENT_REC_NUM="573516592" EVENT_SID="N/A" > EVENT_SOURCE="Microsoft Windows security auditing." EVENT_TASK="Logoff" > EVENT_TYPE="Success Audit" EVENT_USERNAME="domain\\userid"][meta > sequenceId="10817278" sysUpTime="-198876"] domain\userid: Security > Microsoft Windows security auditing.: [Success Audit] An account was > logged off. > > > > ____________________________________________________________ > __________________ > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng > Documentation: http://www.balabit.com/support/documentation/? > product=syslog-ng > FAQ: http://www.balabit.com/wiki/syslog-ng-faq > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.balabit.hu/pipermail/syslog-ng/attachments/20170713/826dd963/a…> ------------------------------ Subject: Digest Footer _______________________________________________ syslog-ng maillist - syslog-ng(a)lists.balabit.hu https://lists.balabit.hu/mailman/listinfo/syslog-ng ------------------------------ End of syslog-ng Digest, Vol 147, Issue 14 ******************************************

2 1

positional arguments for blocks
by Fabien Wernli 21 Jul '17

21 Jul '17

Hi, Many native blocks have positional arguments, e.g. network(): network( "127.0.0.1", … ) Is it possible to set positional arguments to custom blocks?

2 1

Re: [syslog-ng] syslog-ng 3.10 fedora/rhel/centos packages
by alex 19 Jul '17

19 Jul '17

Hello, this is taken from syslog-ng 3.8.1 change log: "Minimal version of libdbi is set to 0.9.0" Correct me if I am wrong, but latest version of libdbi on CentOS 6.X is 0.8.3. How is syslog-ng 3.9 the latest compatible version with older CentOS distro? *From:* syslog-ng [mailto:syslog-ng-bounces@lists.balabit.hu] *On Behalf Of *Czanik, Péter *Sent:* Wednesday, July 19, 2017 1:42 PM *To:* Syslog-ng users' and developers' mailing list < syslog-ng(a)lists.balabit.hu> *Subject:* Re: [syslog-ng] syslog-ng 3.10 fedora/rhel/centos packages Just tested, 3.10.1 does not compile on RHEL 6 (well, in practice the latest CentOS 6.X). It fails with: CCLD modules/dbparser/pdbtool/pdbtool ./lib/.libs/libsyslog-ng.so: undefined reference to `g_atomic_pointer_add' collect2: ld returned 1 exit status make[2]: *** [modules/dbparser/pdbtool/pdbtool] Error 1 make[1]: *** [all-recursive] Error 1 So, currently 3.9 is the latest syslog-ng compiling on old distros. Making sure, that syslog-ng compiles on old distros is on the roadmap of the team. Compiling on FreeBSD was fixed right after the 3.10 release, I hope RHEL 6 / SLES 11 will also compile again soon. Bye, Peter Czanik (CzP) <peter.czanik(a)balabit.com> Balabit / syslog-ng upstream https://www.balabit.com/blog/author/peterczanik/ https://twitter.com/PCzanik On Mon, Jul 17, 2017 at 6:47 PM, Czanik, Péter <peter.czanik(a)balabit.com> wrote: Hi, I'll check RHEL 6 later this week. Personally I tested on SLES 11, as (open)SUSE is my primary testing platform. RHEL 6 might work, at least the later versions, as glib was updated. See https://github.com/balabit/ syslog-ng/issues/1403 for some background info. Bye, Peter Czanik (CzP) <peter.czanik(a)balabit.com> Balabit / syslog-ng upstream https://www.balabit.com/blog/author/peterczanik/ https://twitter.com/PCzanik On Mon, Jul 17, 2017 at 5:12 PM, Evan Rempel <erempel(a)uvic.ca> wrote: RHEL 6 still has 2 years in the normal support lifecycle and even more if a extended support is purchased. IMHO syslog-ng should strive to have a current release that still compiles on these platforms until at least the end of standard support (2020). Evan. On 07/17/2017 07:36 AM, Czanik, Péter wrote: hi, Until now my unofficial syslog-ng 3.10 packages for Fedora / RHEL 7 / CentOS 7 were available from my "githead" repo. Today I created a dedicated 3.10 repository, as I plan to use the githead repo for creating builds from git HEAD again. It is available at https://copr.fedorainfracloud. org/coprs/czanik/syslog-ng310/ Bye, Peter Czanik (CzP) <peter.czanik(a)balabit.com> Balabit / syslog-ng upstream https://www.balabit.com/blog/author/peterczanik/ https://twitter.com/PCzanik

2 1

syslog-ng 3.10 fedora/rhel/centos packages
by Czanik, Péter 19 Jul '17

19 Jul '17

hi, Until now my unofficial syslog-ng 3.10 packages for Fedora / RHEL 7 / CentOS 7 were available from my "githead" repo. Today I created a dedicated 3.10 repository, as I plan to use the githead repo for creating builds from git HEAD again. It is available at https://copr.fedorainfracloud.org/coprs/czanik/syslog-ng310/ Bye, Peter Czanik (CzP) <peter.czanik(a)balabit.com> Balabit / syslog-ng upstream https://www.balabit.com/blog/author/peterczanik/ https://twitter.com/PCzanik

2 3

Java Stack traces and the dreaded multi-line
by Clayton Dukes 17 Jul '17

17 Jul '17

Does anyone have good config examples for parsing incoming Java events from both Network and Files? [cid:image001.png@01D2FEE8.2D247F00]

1 0

in-list syntax
by Damian Bell 17 Jul '17

17 Jul '17

Hello, I’ve previously had a very simple regex list configured for Cisco ASA devices to deliver email alerts on routing changes in our network, which was configured thus:- filter f_sev1 { match("%ASA-3-622001"); }; …which works just fine. Wanting to expand this list a bit, and aware that scaling regex matching up is going to incur performance issues, I’ve thought that an “in-list” expression file might have been a better approach, and as such have configured the following:- filter f_sev1 { in-list("/etc/syslog-ng/email-match-list.list", value("PROGRAM")); }; …where “PROGRAM” is (I believe) the applicable part of the code that matches to the Cisco "%ASA-3-622001" part of the message. The whitelist has the following entries (as an example):- %DUAL-5-NBRCHANGE %ASA-3-622001 %SPANTREE-5-TOPOTRAP %SPANTREE-5-ROOTCHANGE %ASA-5-111010 I am not getting any matches here, however. What am I missing? Thanks very much in advance, Damian Damian Bell Infrastructure Engineer | Support | H Clarkson & Co Ltd T: +44 20 7334 5483 Email: Damian.Bell(a)clarksons.com<mailto:Damian.Bell@clarksons.com> Group Email: infrastructure(a)clarksons.com Clarksons Platou TM Commodity Quay, St. Katharine Docks | London E1W 1BF | United Kingdom www.clarksons.com<http://www.clarksons.com> Please consider the environment before printing this e-mail ________________________________ This message is private and confidential. If you have received it in error, you are on notice of its status. Please notify us immediately by reply email and then delete this message from your system. Please do not copy it or use it for any purposes, or disclose its contents to any other person: to do so could be a breach of confidence. Emails may be monitored. Details of Clarkson group companies and their regulators (where applicable) can be found at this url: Disclosure<http://www.clarksons.com/disclosure/> ________________________________

3 2

rpm to do upgrade on isolated network centos6 server?
by craig bowser 17 Jul '17

17 Jul '17

I'm sure this has been asked before, but I have not yet found a reference or faq so if someone can point me there, I'd appreciate it... After some googling and searching the email list history, I have not found a way to download the actual rpm for syslog-ng so I can burn it to disc and install it on a server on an isolated network. The page below indicates that getting actual rpm files only come with the commercial version and other than that you need to download from the repo? https://www.balabit.com/blog/rpm-packages-from-syslog-ng-git-head/ Also, I didn't see any current rpms on on rpmfind.net for centOS6..... So I assume I have to download the source, move it over, then install from that source? If that's the case, is there a list of other dependencies I need? Did I miss a link or instruction? Thanks. Craig L Bowser ____________________________ This email is measured by size. Bits and bytes may have settled during transport.

4 4

Windows Events multi-line
by Smith, Paul (Sr. Admin-InfoSec) 13 Jul '17

13 Jul '17

Hello, I'm having problems getting Windows events on a single line on syslog-ng OSE. I've scoured the interwebs and not found what I need to get this exact. I am guessing this is not an uncommon problem but I can't seem to find quite what I need. I am guessing I am just missing some simple thing here. Here are my details. Using syslog-ng OSE 3.9.1 Have syslog-ng Windows Agent 6.0.6 running on Windows 2012 server Have a tcp source and that writes direct to the log file. Works fine with no options set. Getting multiple lines per event. I've added what I think are the correct settings for multi-line, it does not work. I don't think it is the regex syntax, but something else? ___________ Syslog.ng.conf source s_nettcp_win { network(ip(x.x.x.x) port(601) transport("tcp") multi-line-mode(regexp) multi-line-prefix("^[0-9]{3,5}\s<[0-9]{2,3}>1\s([0-9]{4})-(0[1-9]|1[0-2])-(0[1-9]|[1-2][0-9]|3[0-1])T([0-9]{2}:[0-9]{2}:[0-9]{2}-[0-9]{2}:[0-9]{2})") flags(no-parse)); }; ___________ Here is the error when trying to start syslog-ng or run syslog-ng -s: Error parsing afsocket, syntax error, unexpected LL_IDENTIFIER, expecting KW_NORMALIZE_HOSTNAMES or KW_USE_DNS or KW_USE_FQDN or KW_DNS_CACHE in /etc/syslog-ng/syslog-ng.conf at line 38, column 76: source s_nettcp_win { network(ip(x.x.x.x) port(601) transport("tcp") multi-line-mode(regexp) multi-line-prefix("^[0-9]{3,5}\s<[0-9]{2,3}>1\s([0-9]{4})-(0[1-9]|1[0-2])-(0[1-9]|[1-2][0-9]|3[0-1])T([0-9]{2}:[0-9]{2}:[0-9]{2}-[0-9]{2}:[0-9]{2})") flags(no-parse)); }; ^^^^^^^^^^^^^^^ ___________ Sample of the event log part I am matching regex on: Jul 10 12:11:19 x.x.x.x 912 <133>1 2017-07-10T12:11:18-05:00 computername Microsoft_Windows_security_auditing. 6260 - [win(a)18372.4 EVENT_CATEGORY="Logoff" EVENT_FACILITY="16" EVENT_ID="4634" EVENT_LEVEL="0" EVENT_NAME="Security" EVENT_REC_NUM="573516592" EVENT_SID="N/A" EVENT_SOURCE="Microsoft Windows security auditing." EVENT_TASK="Logoff" EVENT_TYPE="Success Audit" EVENT_USERNAME="domain\\userid"][meta sequenceId="10817278" sysUpTime="-198876"] domain\userid: Security Microsoft Windows security auditing.: [Success Audit] An account was logged off.

2 1

Problems with set() / condition() ?
by Jorge Pereira 13 Jul '17

13 Jul '17

Hi List, I am trying to use the below set() + condition() rewrite { set("$(* $(/ $R_MIN 5) 5)", value("min_slice")); set("0${min_slice}", value("min5"), condition("min_slice" <= "9")); set("${min_slice}", value("min5"), condition("min_slice" >= "10")); # final macro, e.g: www.mytest.com_20170417-1001PM-ng_lua.log.gz set("${YEAR}${MONTH}${DAY}-${HOUR12}${min5}${AMPM}", value("logfmt_file")); set("${YEAR}${MONTH}${DAY}/${HOUR12}${min5}${AMPM}", value("logfmt_dir")); }; But, does not work with R_MIN values around 1 ~ 9 then never execute the second set(... *condition("min_slice" <= "9")*) statement. Has somebody idea? p.s: I am using the syslog-ng 3.7.3 -- Jorge Pereira

2 3

how to preserve original filenames when getting input from wildcard source
by Wout Peeters 12 Jul '17

12 Jul '17

Hi all, New user, I have a very basic question about the wildcard source function that's been recently added to syslog-ng OSE: Suppose I'm getting input from the following: source s_files { wildcard-file( base-dir("/var/log/") filename-pattern("*.log") recursive(no) follow-freq(1) ); }; And in /var/log I have different *.log files: - 1.log - 2.log - 3.log ... How can I save the output on my syslog-ng server, so that input from 1.log would arrive in a file named 1.log on my server, input from 2.log in a file named 2.log etc.? Thanks and k ind regards, Wout

2 1

New user trying to filter / rewrite apache logs
by Filipe Cifali 11 Jul '17

11 Jul '17

Hi all, reading the docs I got into this config: source s_apache_access_log { file( "/var/logs/apache2/access_log" follow-freq(1) flags(no-parse) ); }; filter f_apache_access_log { match( '(.*) (.*) - - \[[0-9]{2}\/[A-Z][a-z]{2}\/[0-9]{4}:[0-9]{2}:[0-9]{2}:[0-9]{2} -0300\] \"(.*) (.*) (.*)\" (.*) (.*) \"-\" (.*)' type("pcre") flags("store-matches") ); }; rewrite r_apache_access_log { set("$1", value("DOMAIN") condition(filter(f_apache_access_log))); set("$2", value("IP") condition(filter(f_apache_access_log))); set("$3", value("HTTP_METHOD") condition(filter(f_apache_access_log))); set("$4", value("URI") condition(filter(f_apache_access_log))); set("$6", value("HTTP_STATUS") condition(filter(f_apache_access_log))); set("$7", value("SIZE") condition(filter(f_apache_access_log))); set("$8", value("USER_AGENT") condition(filter(f_apache_access_log))); }; destination d_apache_access_log { mongodb( # https://docs.mongodb.com/manual/reference/connection-string/ persist-name("apache-access-logs") uri("mongodb://$server_and_port/syslog?wtimeoutMS=60000&socketTimeoutMS=60000&connectTimeoutMS=60000") collection("logs") retries(3600) value-pairs( pair("HOST", "${HOST}") pair("SERVICE", "APACHE") pair("DATE", "${DAY}/${MONTH}/${YEAR}") pair("TIME", "${HOUR}:${MIN}") pair("MESSAGE", "${MESSAGE}") pair("DOMAIN", "${DOMAIN}") pair("HTTP_STATUS", "${HTTP_STATUS}") pair("HTTP_METHOD", "${HTTP_METHOD}") pair("USER_AGENT", "${USER_AGENT}") pair("SIZE", "${SIZE}") pair("URI", "${URI}") pair("IP", "${IP}") ) ); }; log { source(s_apache_access_log); filter(f_apache_access_log); rewrite(r_apache_access_log); destination(d_apache_access_log); }; but I think something is not ok, I'm not sure this is the right way to do it. This log produces an strange behavior: www.cifa.li 127.0.0.1 - - [11/Jul/2017:09:18:56 -0300] "GET / HTTP/1.1" 200 18652 "http://cifa.li/" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:54.0) Gecko/20100101 Firefox/54.0" but this one doesn't cifa.li 127.0.0.1 - - [11/Jul/2017:09:18:56 -0300] "GET / HTTP/1.1" 200 18652 "http://cifa.li/" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:54.0) Gecko/20100101 Firefox/54.0" The behavior is (only for subdomains): DOMAIN: ': www.cifa.li' corret one DOMAIN: 'www.cifa.li' The subdomain seems like it's adding stuff that I didn't (or want) to add. Am I missing something? Thanks in advance. -- [ ]'s Filipe Cifali Stangler

3 6

Syslog-ng OSE idle connections
by Alexandros Leventakis 10 Jul '17

10 Jul '17

Hello, I am using syslog-ng OSE for the last 5 years. I have a question regarding idle established connections. Is there a way to set a timeout on configuration to close such connections if interval is reached and no traffic is generated? so_keepalive is set to false. Regards, Alex [cid:image2393c0.PNG@781730d6.429f4045] Alexandros Leventakis Software Developer | ClearSkies 1, Lefkos Anastasiades Street, 2012 Strovolos, Nicosia Cyprus Tel.: +35722463600 Ext.:167 ALeventakis(a)odysseycs.com<mailto:ALeventakis@odysseycs.com> <https://www.odysseycs.com/Newsletter/15.OdysseyNewsletter/Odyssey_GDPR_Leaf…>[cid:imagef54931.PNG@71df97ec.44b00e28]<https://www.odysseycs.com/whoweare/contactus/> This e-mail contains proprietary information some or all of which may be legally privileged. It is for the intended recipient only. If an addressing or transmission error has misdirected this e-mail, please notify the author by replying to this e-mail. If you are not the intended recipient you must not use, disclose, distribute, copy, print or rely on this e-mail. The content of this email may contain private views and opinions, which do not constitute formal disclosure or commitment unless specifically stated. WWW.ODYSSEYCS.COM<http://www.odysseycs.com/> [cid:image74dc1c.JPG@3fe405e2.4bb745f1]

2 1

value-pairs and empty keys
by Fabien Wernli 10 Jul '17

10 Jul '17

Hi, How would you go about dropping keys with empty values? Or more generally, conditionally select keys depending on their values? I *think* something changed somewhere between 3.7 and 3.9 as I used to have no values containing "", and now I do. Cheers

3 3

Re: [syslog-ng] mark-mode
by Balazs Scheidler 03 Jul '17

03 Jul '17

Hi, Right now, there's no other means. Within the code theres a flag to indicate marks, but that does not cross the wire. On Jun 29, 2017 16:01, "Fabien Wernli" <wernli(a)in2p3.fr> wrote: Hi, What would be the best way to identify a message originating from mark-mode? I don't see any obvious way to do so, apart from matching the message for '-- MARK --'. Here's an example on how it looks: { "YEAR_DAY": "180", "YEAR": "2017", "WEEK_DAY_NAME": "Thursday", "WEEK_DAY_ABBREV": "Thu", "WEEK_DAY": "5", "WEEKDAY": "Thu", "WEEK": "26", "USEC": "233311", "UNIXTIME": "1498744609", "TZOFFSET": "+02:00", "TZ": "+02:00", "TAGS": ".source.#anon-source0", "TAG": "0d", "S_YEAR_DAY": "180", "S_YEAR": "2017", "S_WEEK_DAY_NAME": "Thursday", "S_WEEK_DAY_ABBREV": "Thu", "S_WEEK_DAY": "5", "S_WEEKDAY": "Thu", "S_WEEK": "26", "S_USEC": "233311", "S_UNIXTIME": "1498744609", "S_TZOFFSET": "+02:00", "S_TZ": "+02:00", "S_STAMP": "Jun 29 15:56:49", "S_SEC": "49", "S_MSEC": "233", "S_MONTH_WEEK": "4", "S_MONTH_NAME": "June", "S_MONTH_ABBREV": "Jun", "S_MONTH": "06", "S_MIN": "56", "S_ISODATE": "2017-06-29T15:56:49+02:00", "S_HOUR12": "03", "S_HOUR": "15", "S_FULLDATE": "2017 Jun 29 15:56:49", "S_DAY": "29", "S_DATE": "Jun 29 15:56:49", "S_AMPM": "PM", "SYSUPTIME": "207", "STAMP": "Jun 29 15:56:49", "SOURCEIP": "127.0.0.1", "SOURCE": "#anon-source0", "SEC": "49", "R_YEAR_DAY": "180", "R_YEAR": "2017", "R_WEEK_DAY_NAME": "Thursday", "R_WEEK_DAY_ABBREV": "Thu", "R_WEEK_DAY": "5", "R_WEEKDAY": "Thu", "R_WEEK": "26", "R_USEC": "233311", "R_UNIXTIME": "1498744609", "R_TZOFFSET": "+02:00", "R_TZ": "+02:00", "R_STAMP": "Jun 29 15:56:49", "R_SEC": "49", "R_MSEC": "233", "R_MONTH_WEEK": "4", "R_MONTH_NAME": "June", "R_MONTH_ABBREV": "Jun", "R_MONTH": "06", "R_MIN": "56", "R_ISODATE": "2017-06-29T15:56:49+02:00", "R_HOUR12": "03", "R_HOUR": "15", "R_FULLDATE": "2017 Jun 29 15:56:49", "R_DAY": "29", "R_DATE": "Jun 29 15:56:49", "R_AMPM": "PM", "RUNID": "1", "PRIORITY": "notice", "PRI": "13", "MSG": "dl", "MSEC": "233", "MONTH_WEEK": "4", "MONTH_NAME": "June", "MONTH_ABBREV": "Jun", "MONTH": "06", "MIN": "56", "MESSAGE": "dl", "LOGHOST": "localhost.localdomain", "LEVEL_NUM": "5", "LEVEL": "notice", "ISODATE": "2017-06-29T15:56:49+02:00", "HOUR12": "03", "HOUR": "15", "HOST_FROM": "localhost", "HOSTID": "abb0b0e5", "HOST": "localhost", "FULLDATE": "2017 Jun 29 15:56:49", "FILE_NAME": "/dev/stdin", "FACILITY_NUM": "1", "FACILITY": "user", "DAY": "29", "DATE": "Jun 29 15:56:49", "C_YEAR_DAY": "180", "C_YEAR": "2017", "C_WEEK_DAY_NAME": "Thursday", "C_WEEK_DAY_ABBREV": "Thu", "C_WEEK_DAY": "5", "C_WEEKDAY": "Thu", "C_WEEK": "26", "C_UNIXTIME": "1498744609", "C_TZOFFSET": "-00:00", "C_TZ": "-00:00", "C_STAMP": "Jun 29 13:56:48", "C_SEC": "48", "C_MONTH_WEEK": "4", "C_MONTH_NAME": "June", "C_MONTH_ABBREV": "Jun", "C_MONTH": "06", "C_MIN": "56", "C_ISODATE": "2017-06-29T13:56:48-00:00", "C_HOUR": "13", "C_FULLDATE": "2017 Jun 29 13:56:48", "C_DAY": "29", "C_DATE": "Jun 29 13:56:48", "BSDTAG": "5B", "AMPM": "PM" } ____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/? product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq

1 0

mark-mode
by Fabien Wernli 29 Jun '17

29 Jun '17

Hi, What would be the best way to identify a message originating from mark-mode? I don't see any obvious way to do so, apart from matching the message for '-- MARK --'. Here's an example on how it looks: { "YEAR_DAY": "180", "YEAR": "2017", "WEEK_DAY_NAME": "Thursday", "WEEK_DAY_ABBREV": "Thu", "WEEK_DAY": "5", "WEEKDAY": "Thu", "WEEK": "26", "USEC": "233311", "UNIXTIME": "1498744609", "TZOFFSET": "+02:00", "TZ": "+02:00", "TAGS": ".source.#anon-source0", "TAG": "0d", "S_YEAR_DAY": "180", "S_YEAR": "2017", "S_WEEK_DAY_NAME": "Thursday", "S_WEEK_DAY_ABBREV": "Thu", "S_WEEK_DAY": "5", "S_WEEKDAY": "Thu", "S_WEEK": "26", "S_USEC": "233311", "S_UNIXTIME": "1498744609", "S_TZOFFSET": "+02:00", "S_TZ": "+02:00", "S_STAMP": "Jun 29 15:56:49", "S_SEC": "49", "S_MSEC": "233", "S_MONTH_WEEK": "4", "S_MONTH_NAME": "June", "S_MONTH_ABBREV": "Jun", "S_MONTH": "06", "S_MIN": "56", "S_ISODATE": "2017-06-29T15:56:49+02:00", "S_HOUR12": "03", "S_HOUR": "15", "S_FULLDATE": "2017 Jun 29 15:56:49", "S_DAY": "29", "S_DATE": "Jun 29 15:56:49", "S_AMPM": "PM", "SYSUPTIME": "207", "STAMP": "Jun 29 15:56:49", "SOURCEIP": "127.0.0.1", "SOURCE": "#anon-source0", "SEC": "49", "R_YEAR_DAY": "180", "R_YEAR": "2017", "R_WEEK_DAY_NAME": "Thursday", "R_WEEK_DAY_ABBREV": "Thu", "R_WEEK_DAY": "5", "R_WEEKDAY": "Thu", "R_WEEK": "26", "R_USEC": "233311", "R_UNIXTIME": "1498744609", "R_TZOFFSET": "+02:00", "R_TZ": "+02:00", "R_STAMP": "Jun 29 15:56:49", "R_SEC": "49", "R_MSEC": "233", "R_MONTH_WEEK": "4", "R_MONTH_NAME": "June", "R_MONTH_ABBREV": "Jun", "R_MONTH": "06", "R_MIN": "56", "R_ISODATE": "2017-06-29T15:56:49+02:00", "R_HOUR12": "03", "R_HOUR": "15", "R_FULLDATE": "2017 Jun 29 15:56:49", "R_DAY": "29", "R_DATE": "Jun 29 15:56:49", "R_AMPM": "PM", "RUNID": "1", "PRIORITY": "notice", "PRI": "13", "MSG": "dl", "MSEC": "233", "MONTH_WEEK": "4", "MONTH_NAME": "June", "MONTH_ABBREV": "Jun", "MONTH": "06", "MIN": "56", "MESSAGE": "dl", "LOGHOST": "localhost.localdomain", "LEVEL_NUM": "5", "LEVEL": "notice", "ISODATE": "2017-06-29T15:56:49+02:00", "HOUR12": "03", "HOUR": "15", "HOST_FROM": "localhost", "HOSTID": "abb0b0e5", "HOST": "localhost", "FULLDATE": "2017 Jun 29 15:56:49", "FILE_NAME": "/dev/stdin", "FACILITY_NUM": "1", "FACILITY": "user", "DAY": "29", "DATE": "Jun 29 15:56:49", "C_YEAR_DAY": "180", "C_YEAR": "2017", "C_WEEK_DAY_NAME": "Thursday", "C_WEEK_DAY_ABBREV": "Thu", "C_WEEK_DAY": "5", "C_WEEKDAY": "Thu", "C_WEEK": "26", "C_UNIXTIME": "1498744609", "C_TZOFFSET": "-00:00", "C_TZ": "-00:00", "C_STAMP": "Jun 29 13:56:48", "C_SEC": "48", "C_MONTH_WEEK": "4", "C_MONTH_NAME": "June", "C_MONTH_ABBREV": "Jun", "C_MONTH": "06", "C_MIN": "56", "C_ISODATE": "2017-06-29T13:56:48-00:00", "C_HOUR": "13", "C_FULLDATE": "2017 Jun 29 13:56:48", "C_DAY": "29", "C_DATE": "Jun 29 13:56:48", "BSDTAG": "5B", "AMPM": "PM" }

1 0

Syslog-ng 3.9
by PENLAND,MATTHEW & DANA 28 Jun '17

28 Jun '17

I have several systems that are sending there log files to syslog-ng, without issue. I having an issue getting syslog-ng to work with logs from our Brocade MLX’s. I run wireshark on the syslog server and see the file hit it. However, syslog-ng does not put it in its folder or any where else. Sections of the config file relevant to the Brocade. destination d_bro { file("/home/hosts/brocade/$YEAR$MONTH$DAY-$HOST.log"); }; ##################### # BROCADE ##################### filter f_bro { not level(debug) and ( host( "BROCADE-BG-MLX" ) or host( "BROCADE-BG-FESX" ) or host( "BROCADE-BR-MLX" ) or host( "BROCADE-DUCK-FESX" ) or host( "BROCADE-ELJY-MLX" ) or host( "BROCADE-JSPR-MLX" ) or host( "BROCADE-STUDIO-FESX" ));}; log { source(s_net); filter(f_bro); destination(d_bro); }; Wireshark on the server - Capture.PNG

3 2

syslog-ng poll: containers
by Czanik, Péter 28 Jun '17

28 Jun '17

Hi, Yesterday I created another poll, this time about containers: https://twitter.com/PCzanik/status/879704447743533056 Please vote and also re-tweet it, so more people are reached. Thanks, Peter Czanik (CzP) <peter.czanik(a)balabit.com> Balabit / syslog-ng upstream https://www.balabit.com/blog/author/peterczanik/ https://twitter.com/PCzanik

1 0

java module crashes on recent kernels addressing CVE-2017-1000364
by Czanik, Péter 26 Jun '17

26 Jun '17

Hi, Due to an incompatible fix in recent kernels (to resolve CVE-2017-1000364), the Java module in syslog-ng crashes. This is used for example by the Elasticsearch destination. Right now the workaround is to use a kernel without the fix. Follow https://github.com/balabit/syslog-ng/issues/1562 to see any updates on the issue. Bye, Peter Czanik (CzP) <peter.czanik(a)balabit.com> Balabit / syslog-ng upstream https://www.balabit.com/blog/author/peterczanik/ https://twitter.com/PCzanik

1 0

java destinations
by Czanik, Péter 26 Jun '17

26 Jun '17

Hi, I wanted to send some logs to Elasticsearch, but I did not succeed. Syslog-ng crashed. Actually it seems to be something JVM related. A bit too many things changed recently in my test environment: - a new laptop - a new version of vmware workstation - a new version of glibc and openjdk inside the vm-s (I always start with a security update...) The problem appeared both with syslog-ng 3.9 and 3.10 on both openSUSE Leap and CentOS 7. Have you ever seen anything similar? Bye, CzP This one is from CentOS7 where I already tried rolling back glibc and openjdk: [2017-06-23T13:25:22.816610] Compiling d_elastic reference [destination] at [/etc/syslog-ng/conf.d/es2.conf:27:3] [2017-06-23T13:25:22.816968] Seeking the journal to the start position; [2017-06-23T13:25:22.817808] Processing the time zone file (32bit part); filename='/usr/share/zoneinfo/UTC' # # A fatal error has been detected by the Java Runtime Environment: # # SIGBUS (0x7) at pc=0x00007f5ecc511c18, pid=1651, tid=0x00007f5ee3949b40 # # JRE version: (8.0_121-b13) (build ) # Java VM: OpenJDK 64-Bit Server VM (25.121-b13 mixed mode linux-amd64 compressed oops) # Problematic frame: # j java.lang.Object.<clinit>()V+0 # # Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again # # An error report file with more information is saved as: # /tmp/hs_err_pid1651.log # # If you would like to submit a bug report, please visit: # http://bugreport.java.com/bugreport/crash.jsp # Aborted (core dumped) This one is from openSUSE, where I had gdb installed... linux-pzl9:/var/lib/systemd/coredump # gdb -c core.syslog-ng.0.46a0d0e824ce4fa29a644ef713064044.2015.1498211596000000 /usr/sbin/syslog-ng GNU gdb (GDB; openSUSE Leap 42.2) 7.11.1 Copyright (C) 2016 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html > This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-suse-linux". Type "show configuration" for configuration details. For bug reporting instructions, please see: <http://bugs.opensuse.org/>. Find the GDB manual and other documentation resources online at: <http://www.gnu.org/software/gdb/documentation/>. For help, type "help". Type "apropos word" to search for commands related to "word"... Reading symbols from /usr/sbin/syslog-ng...(no debugging symbols found)...done. [New LWP 2015] [New LWP 2016] [New LWP 2017] [New LWP 2018] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". Core was generated by `syslog-ng -Fvde'. Program terminated with signal SIGSEGV, Segmentation fault. #0 _expand_stack_to (bottom=0x7ffe43f70fff "", bottom@entry=0x7ffe43f70990 "") at /usr/src/debug/icedtea-3.4.0/openjdk/hotspot/src/os/linux/vm/os_linux.cpp:673 673 /usr/src/debug/icedtea-3.4.0/openjdk/hotspot/src/os/linux/vm/os_linux.cpp: No such file or directory. [Current thread is 1 (Thread 0x7ff2ab992b40 (LWP 2015))] Missing separate debuginfos, use: zypper install syslog-ng-debuginfo-3.10.1-2.3.x86_64 (gdb) bt full #0 _expand_stack_to (bottom=0x7ffe43f70fff "", bottom@entry=0x7ffe43f70990 "") at /usr/src/debug/icedtea-3.4.0/openjdk/hotspot/src/os/linux/vm/os_linux.cpp:673 sp = 0x7ffe43f7a120 "" size = 37153 p = 0x7ffe43f70ff0 "" #1 0x00007ff2a5f6f184 in os::Linux::manually_expand_stack (t=t@entry=0x19e9000, addr=0x7ffe43f70990 "") at /usr/src/debug/icedtea-3.4.0/openjdk/hotspot/src/os/linux/vm/os_linux.cpp:686 mask_all = {__val = {18446744067267100671, 18446744073709551615 <repeats 15 times>}} old_sigset = {__val = {18446744067266838271, 140680142549761, 140730038723104, 140680138113208, 1140303776, 210453397503, 140730038723232, 140730038724512, 1, 27171840, 18446744073709551615, 4, 140730038723200, 140680140758410, 140730038723208, 18446744069414584320}} #2 0x00007ff2a5f77f7d in JVM_handle_linux_signal (sig=11, info=0x7ffe43f7a530, ucVoid=0x7ffe43f7a400, abort_if_unrecognized=1) at /usr/src/debug/icedtea-3.4.0/openjdk/hotspot/src/os_cpu/linux_x86/vm/os_linux_x86.cpp:330 addr = <optimized out> uc = 0x7ffe43f7a400 thread = 0x19e9000 stub = 0x0 newset = {__val = {0, 0, 18446744073709551615, 2, 27670976, 140679847562464, 3, 27171760, 27171728, 140679847562559, 140679847562515, 0, 140730038723488, 140680143714222, 3, 140730038724512}} err = {<StackObj> = {<No data fields>}, _id = 1, _message = 0x19e9c00 "", _detail_msg = 0xffffffffffffffff <error: Cannot access memory at address 0xffffffffffffffff>, _thread = 0x4, _pc = 0x7ffe43f7a740 "\320\v\177\212\362\177", _siginfo = 0x7ff2a60051f8 <SharedRuntime::generate_i2c2i_adapters(MacroAssembler*, int, int, BasicType const*, VMRegPair const*, AdapterFingerPrint*)+3464>, _context = 0x3, _filename = 0xffffffffffffffff <error: Cannot access memory at address 0xffffffffffffffff>, _lineno = -1, _current_step = 48, _current_step_info = 0x7ff2a65619f0 <vtable for Relocation+16> "\240栥\362\177", _verbose = 0, static first_error = 0x0, static first_error_tid = -1, static coredump_status = false, static coredump_message = '\000' <repeats 1999 times>, _size = 140680227341344, static out = {<outputStream> = {<ResourceObj> = {<No data fields>}, _vptr.outputStream = 0x7ff2a655fc50 <vtable for fdStream+16>, _indentation = 0, _width = 80, _position = 0, _newlines = 0, _precount = 0, _stamp = {_counter = 0}}, _fd = 1, _need_close = false}, static log = {<outputStream> = {<ResourceObj> = {<No data fields>}, _vptr.outputStream = 0x7ff2a655fc50 <vtable for fdStream+16>, _indentation = 0, _width = 80, _position = 0, _newlines = 0, _precount = 0, _stamp = {_counter = 0}}, _fd = -1, _need_close = false}} t = 0x19e9000 shm = {<StackObj> = {<No data fields>}, _thread = 0x19e9000} vmthread = 0x0 pc = <optimized out> #3 0x00007ff2a5f6e3d8 in signalHandler (sig=11, info=0x7ffe43f7a530, uc=0x7ffe43f7a400) at /usr/src/debug/icedtea-3.4.0/openjdk/hotspot/src/os/linux/vm/os_linux.cpp:4354 orig_errno = 2 #4 <signal handler called> No symbol table info available. #5 0x00007ff29453dc26 in ?? () No symbol table info available. #6 0x00007ffe43f7a990 in ?? () No symbol table info available. #7 0x00007ff28a7d0d50 in ?? () No symbol table info available. #8 0x00007ffe43f7a9d8 in ?? () No symbol table info available. #9 0x00007ff28a892ff8 in ?? () No symbol table info available. #10 0x0000000000000000 in ?? () No symbol table info available. (gdb) Peter Czanik (CzP) <peter.czanik(a)balabit.com> Balabit / syslog-ng upstream https://www.balabit.com/blog/author/peterczanik/ https://twitter.com/PCzanik

5 9

linux kernel tuning
by Jim Hendrick 23 Jun '17

23 Jun '17

Hi all, What are your experiences with tuning the kernel for UDP, TCP, etc. How long does it take to see results from a single change? We are working issues with UDP drops, and have made fairly radical increases in buffer sizes with no apparent improvement, only to re-visit the system in a few days and seeing it cleaned up. "intuitively" I have always thought it should happen right away - but that is not what we are seeing here now. Thoughts? Thanks in advance! Jim

2 1

wildcard-file() source
by Czanik, Péter 22 Jun '17

22 Jun '17

Hi, Last week I ran a poll on twitter about the new features of syslog/ng 3.10 and the wildcard-file() source was voted the most. Starting with version 3.10, syslog-ng can collect messages from multiple text files. You do not have to specify file names one by one, just use a wildcard to select which files to read. This is especially useful when you do not know the file names by the time syslog-ng is started. This is often the case with web servers with multiple virtual hosts. From this blog you can learn how to get started using the wildcard-file() source. https://www.balabit.com/blog/reading-multiple-files-wildcard-file-source-sy… Bye, Peter Czanik (CzP) <peter.czanik(a)balabit.com> Balabit / syslog-ng upstream https://www.balabit.com/blog/author/peterczanik/ https://twitter.com/PCzanik

1 0

question Syslog-NG Set Thread Name ?
by rmkml 20 Jun '17

20 Jun '17

Hello, First, Big Thx everyone for very good Syslog-NG product! During middle/high load, it's always interesting for known what Jobs load Syslog-NG process, but if I think correctly, top -H not show what Thread Name is used ? Do you known it's possible Syslog-NG supporting Set Thread Name like pthread_setname_np ? Best Regards @Rmkml

1 0

template() parameter for apache-accesslog-parser
by Czanik, Péter 20 Jun '17

20 Jun '17

Hi, I ran into a situation, that an apache access log message arrives as a field in a CSV file. Rigth now the apache-accesslog-parser is always run on the MESSAGE template, but in this case I'd need to run this parser on csv.apache instead of MESSAGE. What do you think about the following change? It would keep the default behavior but enable me to use a different a template. Bye, CzP linux-pzl9:/usr/share/syslog-ng/include/scl/apache # diff -u /root/syslog-ng/scl/apache/apache.conf apache.conf --- /root/syslog-ng/scl/apache/apache.conf 2017-06-19 13:47:20.044547152 +0200 +++ apache.conf 2017-06-20 16:51:06.740528980 +0200 @@ -19,7 +19,7 @@ # COPYING for details. # ############################################################################# -block parser apache-accesslog-parser(prefix(".apache.")) { +block parser apache-accesslog-parser(prefix(".apache.") template("${MESSAGE}")) { channel { @@ -29,6 +29,7 @@ dialect(escape-double-char) flags(strip-whitespace) delimiters(" ") + template(`template`) quote-pairs('""[]') # field names match of that of Logstash columns("clientip", "ident", "auth", Peter Czanik (CzP) <peter.czanik(a)balabit.com> Balabit / syslog-ng upstream https://www.balabit.com/blog/author/peterczanik/ https://twitter.com/PCzanik

2 1

tags not working
by Sandbox 19 Jun '17

19 Jun '17

Hi, I would like to transfer/store apache logs, but for some reason i cant tag them: I tried: source s_access { pipe("/var/log/apache2/pipe_access.log" tags("testtag"));. }; or source s_access { pipe("/var/log/apache2/pipe_access.log" program-override("testtag"));. }; Because it doesn't work with 'tags' i used the apache to format the message, i put the choosen word to the log format so it stores the correct place the log on the client and also on the server. Part of the log: - [19/Jun/2017:13:46:29 +0200] acc001 127.0.0.1 - - "GET /server-status?auto HTTP/1.1" 200 1781 If it possible i want to use 'tags' instead of manipulating apache logformats. I tried to read the pipes directly, but the above settings doesnt show, just the above log. I tried to tcpdump the traffic from client -> server (client side) but doesn't show any of the settings. It looks like for me, syslog-ng wont tag my logs. And i tried to read the stored log with file() and tag it, that doesn't work neither. What i am missing? Thanks, Robert

2 4