July 2017 - syslog-ng - lists.balabit.hu

Java Stack traces and the dreaded multi-line
by Clayton Dukes 17 Jul '17

17 Jul '17

Does anyone have good config examples for parsing incoming Java events from both Network and Files? [cid:image001.png@01D2FEE8.2D247F00]

1 0

in-list syntax
by Damian Bell 17 Jul '17

17 Jul '17

Hello, I’ve previously had a very simple regex list configured for Cisco ASA devices to deliver email alerts on routing changes in our network, which was configured thus:- filter f_sev1 { match("%ASA-3-622001"); }; …which works just fine. Wanting to expand this list a bit, and aware that scaling regex matching up is going to incur performance issues, I’ve thought that an “in-list” expression file might have been a better approach, and as such have configured the following:- filter f_sev1 { in-list("/etc/syslog-ng/email-match-list.list", value("PROGRAM")); }; …where “PROGRAM” is (I believe) the applicable part of the code that matches to the Cisco "%ASA-3-622001" part of the message. The whitelist has the following entries (as an example):- %DUAL-5-NBRCHANGE %ASA-3-622001 %SPANTREE-5-TOPOTRAP %SPANTREE-5-ROOTCHANGE %ASA-5-111010 I am not getting any matches here, however. What am I missing? Thanks very much in advance, Damian Damian Bell Infrastructure Engineer | Support | H Clarkson & Co Ltd T: +44 20 7334 5483 Email: Damian.Bell(a)clarksons.com<mailto:Damian.Bell@clarksons.com> Group Email: infrastructure(a)clarksons.com Clarksons Platou TM Commodity Quay, St. Katharine Docks | London E1W 1BF | United Kingdom www.clarksons.com<http://www.clarksons.com> Please consider the environment before printing this e-mail ________________________________ This message is private and confidential. If you have received it in error, you are on notice of its status. Please notify us immediately by reply email and then delete this message from your system. Please do not copy it or use it for any purposes, or disclose its contents to any other person: to do so could be a breach of confidence. Emails may be monitored. Details of Clarkson group companies and their regulators (where applicable) can be found at this url: Disclosure<http://www.clarksons.com/disclosure/> ________________________________

3 2

rpm to do upgrade on isolated network centos6 server?
by craig bowser 17 Jul '17

17 Jul '17

I'm sure this has been asked before, but I have not yet found a reference or faq so if someone can point me there, I'd appreciate it... After some googling and searching the email list history, I have not found a way to download the actual rpm for syslog-ng so I can burn it to disc and install it on a server on an isolated network. The page below indicates that getting actual rpm files only come with the commercial version and other than that you need to download from the repo? https://www.balabit.com/blog/rpm-packages-from-syslog-ng-git-head/ Also, I didn't see any current rpms on on rpmfind.net for centOS6..... So I assume I have to download the source, move it over, then install from that source? If that's the case, is there a list of other dependencies I need? Did I miss a link or instruction? Thanks. Craig L Bowser ____________________________ This email is measured by size. Bits and bytes may have settled during transport.

4 4

Windows Events multi-line
by Smith, Paul (Sr. Admin-InfoSec) 13 Jul '17

13 Jul '17

Hello, I'm having problems getting Windows events on a single line on syslog-ng OSE. I've scoured the interwebs and not found what I need to get this exact. I am guessing this is not an uncommon problem but I can't seem to find quite what I need. I am guessing I am just missing some simple thing here. Here are my details. Using syslog-ng OSE 3.9.1 Have syslog-ng Windows Agent 6.0.6 running on Windows 2012 server Have a tcp source and that writes direct to the log file. Works fine with no options set. Getting multiple lines per event. I've added what I think are the correct settings for multi-line, it does not work. I don't think it is the regex syntax, but something else? ___________ Syslog.ng.conf source s_nettcp_win { network(ip(x.x.x.x) port(601) transport("tcp") multi-line-mode(regexp) multi-line-prefix("^[0-9]{3,5}\s<[0-9]{2,3}>1\s([0-9]{4})-(0[1-9]|1[0-2])-(0[1-9]|[1-2][0-9]|3[0-1])T([0-9]{2}:[0-9]{2}:[0-9]{2}-[0-9]{2}:[0-9]{2})") flags(no-parse)); }; ___________ Here is the error when trying to start syslog-ng or run syslog-ng -s: Error parsing afsocket, syntax error, unexpected LL_IDENTIFIER, expecting KW_NORMALIZE_HOSTNAMES or KW_USE_DNS or KW_USE_FQDN or KW_DNS_CACHE in /etc/syslog-ng/syslog-ng.conf at line 38, column 76: source s_nettcp_win { network(ip(x.x.x.x) port(601) transport("tcp") multi-line-mode(regexp) multi-line-prefix("^[0-9]{3,5}\s<[0-9]{2,3}>1\s([0-9]{4})-(0[1-9]|1[0-2])-(0[1-9]|[1-2][0-9]|3[0-1])T([0-9]{2}:[0-9]{2}:[0-9]{2}-[0-9]{2}:[0-9]{2})") flags(no-parse)); }; ^^^^^^^^^^^^^^^ ___________ Sample of the event log part I am matching regex on: Jul 10 12:11:19 x.x.x.x 912 <133>1 2017-07-10T12:11:18-05:00 computername Microsoft_Windows_security_auditing. 6260 - [win(a)18372.4 EVENT_CATEGORY="Logoff" EVENT_FACILITY="16" EVENT_ID="4634" EVENT_LEVEL="0" EVENT_NAME="Security" EVENT_REC_NUM="573516592" EVENT_SID="N/A" EVENT_SOURCE="Microsoft Windows security auditing." EVENT_TASK="Logoff" EVENT_TYPE="Success Audit" EVENT_USERNAME="domain\\userid"][meta sequenceId="10817278" sysUpTime="-198876"] domain\userid: Security Microsoft Windows security auditing.: [Success Audit] An account was logged off.

2 1

Problems with set() / condition() ?
by Jorge Pereira 13 Jul '17

13 Jul '17

Hi List, I am trying to use the below set() + condition() rewrite { set("$(* $(/ $R_MIN 5) 5)", value("min_slice")); set("0${min_slice}", value("min5"), condition("min_slice" <= "9")); set("${min_slice}", value("min5"), condition("min_slice" >= "10")); # final macro, e.g: www.mytest.com_20170417-1001PM-ng_lua.log.gz set("${YEAR}${MONTH}${DAY}-${HOUR12}${min5}${AMPM}", value("logfmt_file")); set("${YEAR}${MONTH}${DAY}/${HOUR12}${min5}${AMPM}", value("logfmt_dir")); }; But, does not work with R_MIN values around 1 ~ 9 then never execute the second set(... *condition("min_slice" <= "9")*) statement. Has somebody idea? p.s: I am using the syslog-ng 3.7.3 -- Jorge Pereira

2 3

how to preserve original filenames when getting input from wildcard source
by Wout Peeters 12 Jul '17

12 Jul '17

Hi all, New user, I have a very basic question about the wildcard source function that's been recently added to syslog-ng OSE: Suppose I'm getting input from the following: source s_files { wildcard-file( base-dir("/var/log/") filename-pattern("*.log") recursive(no) follow-freq(1) ); }; And in /var/log I have different *.log files: - 1.log - 2.log - 3.log ... How can I save the output on my syslog-ng server, so that input from 1.log would arrive in a file named 1.log on my server, input from 2.log in a file named 2.log etc.? Thanks and k ind regards, Wout

2 1

New user trying to filter / rewrite apache logs
by Filipe Cifali 11 Jul '17

11 Jul '17

Hi all, reading the docs I got into this config: source s_apache_access_log { file( "/var/logs/apache2/access_log" follow-freq(1) flags(no-parse) ); }; filter f_apache_access_log { match( '(.*) (.*) - - \[[0-9]{2}\/[A-Z][a-z]{2}\/[0-9]{4}:[0-9]{2}:[0-9]{2}:[0-9]{2} -0300\] \"(.*) (.*) (.*)\" (.*) (.*) \"-\" (.*)' type("pcre") flags("store-matches") ); }; rewrite r_apache_access_log { set("$1", value("DOMAIN") condition(filter(f_apache_access_log))); set("$2", value("IP") condition(filter(f_apache_access_log))); set("$3", value("HTTP_METHOD") condition(filter(f_apache_access_log))); set("$4", value("URI") condition(filter(f_apache_access_log))); set("$6", value("HTTP_STATUS") condition(filter(f_apache_access_log))); set("$7", value("SIZE") condition(filter(f_apache_access_log))); set("$8", value("USER_AGENT") condition(filter(f_apache_access_log))); }; destination d_apache_access_log { mongodb( # https://docs.mongodb.com/manual/reference/connection-string/ persist-name("apache-access-logs") uri("mongodb://$server_and_port/syslog?wtimeoutMS=60000&socketTimeoutMS=60000&connectTimeoutMS=60000") collection("logs") retries(3600) value-pairs( pair("HOST", "${HOST}") pair("SERVICE", "APACHE") pair("DATE", "${DAY}/${MONTH}/${YEAR}") pair("TIME", "${HOUR}:${MIN}") pair("MESSAGE", "${MESSAGE}") pair("DOMAIN", "${DOMAIN}") pair("HTTP_STATUS", "${HTTP_STATUS}") pair("HTTP_METHOD", "${HTTP_METHOD}") pair("USER_AGENT", "${USER_AGENT}") pair("SIZE", "${SIZE}") pair("URI", "${URI}") pair("IP", "${IP}") ) ); }; log { source(s_apache_access_log); filter(f_apache_access_log); rewrite(r_apache_access_log); destination(d_apache_access_log); }; but I think something is not ok, I'm not sure this is the right way to do it. This log produces an strange behavior: www.cifa.li 127.0.0.1 - - [11/Jul/2017:09:18:56 -0300] "GET / HTTP/1.1" 200 18652 "http://cifa.li/" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:54.0) Gecko/20100101 Firefox/54.0" but this one doesn't cifa.li 127.0.0.1 - - [11/Jul/2017:09:18:56 -0300] "GET / HTTP/1.1" 200 18652 "http://cifa.li/" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:54.0) Gecko/20100101 Firefox/54.0" The behavior is (only for subdomains): DOMAIN: ': www.cifa.li' corret one DOMAIN: 'www.cifa.li' The subdomain seems like it's adding stuff that I didn't (or want) to add. Am I missing something? Thanks in advance. -- [ ]'s Filipe Cifali Stangler

3 6

Syslog-ng OSE idle connections
by Alexandros Leventakis 10 Jul '17

10 Jul '17

Hello, I am using syslog-ng OSE for the last 5 years. I have a question regarding idle established connections. Is there a way to set a timeout on configuration to close such connections if interval is reached and no traffic is generated? so_keepalive is set to false. Regards, Alex [cid:image2393c0.PNG@781730d6.429f4045] Alexandros Leventakis Software Developer | ClearSkies 1, Lefkos Anastasiades Street, 2012 Strovolos, Nicosia Cyprus Tel.: +35722463600 Ext.:167 ALeventakis(a)odysseycs.com<mailto:ALeventakis@odysseycs.com> <https://www.odysseycs.com/Newsletter/15.OdysseyNewsletter/Odyssey_GDPR_Leaf…>[cid:imagef54931.PNG@71df97ec.44b00e28]<https://www.odysseycs.com/whoweare/contactus/> This e-mail contains proprietary information some or all of which may be legally privileged. It is for the intended recipient only. If an addressing or transmission error has misdirected this e-mail, please notify the author by replying to this e-mail. If you are not the intended recipient you must not use, disclose, distribute, copy, print or rely on this e-mail. The content of this email may contain private views and opinions, which do not constitute formal disclosure or commitment unless specifically stated. WWW.ODYSSEYCS.COM<http://www.odysseycs.com/> [cid:image74dc1c.JPG@3fe405e2.4bb745f1]

2 1

value-pairs and empty keys
by Fabien Wernli 10 Jul '17

10 Jul '17

Hi, How would you go about dropping keys with empty values? Or more generally, conditionally select keys depending on their values? I *think* something changed somewhere between 3.7 and 3.9 as I used to have no values containing "", and now I do. Cheers

3 3

Re: [syslog-ng] mark-mode
by Balazs Scheidler 03 Jul '17

03 Jul '17

Hi, Right now, there's no other means. Within the code theres a flag to indicate marks, but that does not cross the wire. On Jun 29, 2017 16:01, "Fabien Wernli" <wernli(a)in2p3.fr> wrote: Hi, What would be the best way to identify a message originating from mark-mode? I don't see any obvious way to do so, apart from matching the message for '-- MARK --'. Here's an example on how it looks: { "YEAR_DAY": "180", "YEAR": "2017", "WEEK_DAY_NAME": "Thursday", "WEEK_DAY_ABBREV": "Thu", "WEEK_DAY": "5", "WEEKDAY": "Thu", "WEEK": "26", "USEC": "233311", "UNIXTIME": "1498744609", "TZOFFSET": "+02:00", "TZ": "+02:00", "TAGS": ".source.#anon-source0", "TAG": "0d", "S_YEAR_DAY": "180", "S_YEAR": "2017", "S_WEEK_DAY_NAME": "Thursday", "S_WEEK_DAY_ABBREV": "Thu", "S_WEEK_DAY": "5", "S_WEEKDAY": "Thu", "S_WEEK": "26", "S_USEC": "233311", "S_UNIXTIME": "1498744609", "S_TZOFFSET": "+02:00", "S_TZ": "+02:00", "S_STAMP": "Jun 29 15:56:49", "S_SEC": "49", "S_MSEC": "233", "S_MONTH_WEEK": "4", "S_MONTH_NAME": "June", "S_MONTH_ABBREV": "Jun", "S_MONTH": "06", "S_MIN": "56", "S_ISODATE": "2017-06-29T15:56:49+02:00", "S_HOUR12": "03", "S_HOUR": "15", "S_FULLDATE": "2017 Jun 29 15:56:49", "S_DAY": "29", "S_DATE": "Jun 29 15:56:49", "S_AMPM": "PM", "SYSUPTIME": "207", "STAMP": "Jun 29 15:56:49", "SOURCEIP": "127.0.0.1", "SOURCE": "#anon-source0", "SEC": "49", "R_YEAR_DAY": "180", "R_YEAR": "2017", "R_WEEK_DAY_NAME": "Thursday", "R_WEEK_DAY_ABBREV": "Thu", "R_WEEK_DAY": "5", "R_WEEKDAY": "Thu", "R_WEEK": "26", "R_USEC": "233311", "R_UNIXTIME": "1498744609", "R_TZOFFSET": "+02:00", "R_TZ": "+02:00", "R_STAMP": "Jun 29 15:56:49", "R_SEC": "49", "R_MSEC": "233", "R_MONTH_WEEK": "4", "R_MONTH_NAME": "June", "R_MONTH_ABBREV": "Jun", "R_MONTH": "06", "R_MIN": "56", "R_ISODATE": "2017-06-29T15:56:49+02:00", "R_HOUR12": "03", "R_HOUR": "15", "R_FULLDATE": "2017 Jun 29 15:56:49", "R_DAY": "29", "R_DATE": "Jun 29 15:56:49", "R_AMPM": "PM", "RUNID": "1", "PRIORITY": "notice", "PRI": "13", "MSG": "dl", "MSEC": "233", "MONTH_WEEK": "4", "MONTH_NAME": "June", "MONTH_ABBREV": "Jun", "MONTH": "06", "MIN": "56", "MESSAGE": "dl", "LOGHOST": "localhost.localdomain", "LEVEL_NUM": "5", "LEVEL": "notice", "ISODATE": "2017-06-29T15:56:49+02:00", "HOUR12": "03", "HOUR": "15", "HOST_FROM": "localhost", "HOSTID": "abb0b0e5", "HOST": "localhost", "FULLDATE": "2017 Jun 29 15:56:49", "FILE_NAME": "/dev/stdin", "FACILITY_NUM": "1", "FACILITY": "user", "DAY": "29", "DATE": "Jun 29 15:56:49", "C_YEAR_DAY": "180", "C_YEAR": "2017", "C_WEEK_DAY_NAME": "Thursday", "C_WEEK_DAY_ABBREV": "Thu", "C_WEEK_DAY": "5", "C_WEEKDAY": "Thu", "C_WEEK": "26", "C_UNIXTIME": "1498744609", "C_TZOFFSET": "-00:00", "C_TZ": "-00:00", "C_STAMP": "Jun 29 13:56:48", "C_SEC": "48", "C_MONTH_WEEK": "4", "C_MONTH_NAME": "June", "C_MONTH_ABBREV": "Jun", "C_MONTH": "06", "C_MIN": "56", "C_ISODATE": "2017-06-29T13:56:48-00:00", "C_HOUR": "13", "C_FULLDATE": "2017 Jun 29 13:56:48", "C_DAY": "29", "C_DATE": "Jun 29 13:56:48", "BSDTAG": "5B", "AMPM": "PM" } ____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/? product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq

1 0