- syslog-ng - lists.balabit.hu

Is the option flush_lines valid now?
by 伊藤宏平 16 Oct '15

16 Oct '15

I tried to use syslog-ng. At first I used a syslog-ng 3.5.3 on ubuntu14.04.But I can not use flush_line option. I asked question in stack overflow.Mr.argus said that " There is a lack of condition"linux - syslog-ng の flush_lines　オプションが効かない - スタック・オーバーフロー Specifically function "log_writer_flush" should be like that. Before: gboolean log_writer_flush(LogWriter *self, LogWriterFlushMode flush_mode) { : if (flush_mode >= LW_FLUSH_BUFFER || count == 0) { if (log_proto_client_flush(proto) == LPS_ERROR) return FALSE; }After: LogProtoFileWriter *w = (LogProtoFileWriter *)proto; if ( (w->buf_count >= w->buf_size) && (flush_mode >= LW_FLUSH_BUFFER || count == 0) ) { if (log_proto_client_flush(proto) == LPS_ERROR) return FALSE; } And now I use a latest syslog-ng ose 3.7.1, hoping the bug has been fixed.But I can not use flush_line option either. Is the option flush_lines valid now? try (動) 試みる、努める、努力する、試す、挑む、見る、審理する tried (形) 検査済みの、証明済みの、試験済みの Powered by iKnow!

3 5

Upon syslog-ng reload /restart a flood of logs is received at a tcp port (destination)
by Shruthi Rao 14 Oct '15

14 Oct '15

I have configured a destination in syslog to a tcp port on localhost options { long_hostnames(off); sync(0); perm(0640); stats(3600); }; source src { internal(); unix-dgram("/dev/log"); udp(); }; destination mgagent{ tcp("127.0.0.1" port(5009) template("$TAG $HOST $PROGRAM $MSGONLY\n")); }; I have a tcp server n the localhost ,receiving logs on this port and it does further processing . Whenever syslog-ng is reloaded or restarted and te tcp server is running , a flood of the logs is sent on this tcp port ( which is handled by the tcp server) , but this is not wanted. The logs which were already sent before restart is sent repeatedly . Is there any way to stop or prevent this . syslog version is syslog-ng 2.0.9 on SLES 11 SP3 -Thanks

2 1

AMQP supported RPM for el5
by Pankaj Sain 09 Oct '15

09 Oct '15

Hi, Please could you help to know, if syslog-ng-2.1.4-9.el5 supports AMQP. If not, could you suggest me the AMQP supported version(*RPM*) for *el5 linux* variants. Regards, *Pankaj Sain** | Netprophets Cyberworks Pvt. Ltd.* Please do not print this mail unless it is necessary.

2 5

insider 2015-10: monitor syslog-ng with Android app; documentation; docker; big data; SUSECon;
by Czanik, Péter 08 Oct '15

08 Oct '15

Dear syslog-ng users, This is the 43th issue of the syslog-ng Insider, a monthly newsletter that brings you syslog-ng-related news. NEWS Monitor syslog-ng with Android app ---------------------------------- During the last Google Summer of Code a new Android application was developed by Krishna Kannan, a student at National University of Singapore, which can monitor syslog-ng. It can check, if syslog-ng is alive and show its configuration and statistics on the screen of your Android device. For details, check https://pzolee.blogs.balabit.com/2015/09/monitor-syslog-ng-with-android-app/ Documentation on github ----------------------- We are pleased to announce that the source code of the syslog-ng Open Source Edition Administrator Guide (https://www.balabit.com/sites/default/files/documents/syslog-ng-ose-latest-… now publicly available on GitHub. This means that if you find an annoying typo, an error in a configuration example, or a bug in the documentation, you can help us correct it by submitting a pull request. Also, if you have a better example to illustrate a feature, or a clearer description about how an option works, let us know. For details about accessing the source XML files and on contributing, see our GitHub page at https://github.com/balabit/syslog-ng-ose-guides/ Docker images for syslog-ng --------------------------- There are three docker images available for syslog-ng OSE at https://hub.docker.com/u/balabit/ The “syslog-ng” image targets end users, the other two are for developers, who would like to set up an isolated development environment for syslog-ng quickly. If you would like to modify the images, the sources are available at https://github.com/balabit/syslog-ng-docker Big data -------- syslog-ng delivers log data directly to Hadoop, Elasticsearch, MongoDB and Kafka, so supports practically all major “big data” systems as a destination. Learn how this simplifies your data pipeline and makes your data routing more flexible and efficient: https://jluby.blogs.balabit.com/2015/09/17/got-big-data-ingestion-problems-… syslog-ng @ SUSECon ------------------- SUSE was one of the first major Linux distributions to include syslog-ng. For a long time syslog-ng was the default syslog implementation in both SLES (SUSE Linux Enterprise Server) and openSUSE, and many SLES and openSUSE users are still enthusiastic about syslog-ng. This year, BalaBit (the company behind syslog-ng) participates at the annual SUSE conference, SUSECon, in Amsterdam, during the first week of November. https://czanik.blogs.balabit.com/2015/09/balabits-syslog-ng-at-the-annual-s… Your feedback and news tips about the next issue is welcome at documentation(a)balabit.com To read this newsletter on-line, visit: http://insider.blogs.balabit.com/ Peter Czanik (CzP) <peter.czanik(a)balabit.com> Balabit / syslog-ng upstream http://czanik.blogs.balabit.com/ https://twitter.com/PCzanik

1 0

Question about compiling 3.7.1 on CentOS 7
by Tech Support 05 Oct '15

05 Oct '15

All; I'm trying to compile 3.7.1 on CentOS 7 with Elasticsearch support which I'm *assuming* requires Java support. However, when I run 'configure', it doesn't enable the Java support. I must be missing a Java dependency somewhere. Does anyone know what dependencies I need for this? Any insight at all would be greatly appreciated. Thanks; John

2 1

Elasticsearch destination and time-zone info
by Evan Rempel 05 Oct '15

05 Oct '15

We are in the process of integrating our logging infrastructure into elasticsearch with Kibana, but have a slight challenge regarding time zones. Some of our equipment will only log in UTC time. This is not an issue because Kibana does time presentation in localtime. Most of our hosts log with our local timezone. This all works out fine due to the way kibana displays the logs - localtime. The problem is that the elasticsearch indexes roll over based on some template (XXXX-YYYY-MM-DD) or such, and this template will an incorrect set of messages. For example, I live in time zone -7:00. This means that any messages after 17:00 (17:00 + 7:00 = 00:00 the next day) that were logged with UTC will go into the index for the next day. So, is there any way to set the time-zone option for the elasticsearch destination? Alternatively, are there any date templates where I could do something like $(timezone UTC $ISODATE) $(timezone UTC $HOUR) in my template? Any help would be appreciated. -- Evan Rempel erempel(a)uvic.ca Senior Systems Administrator 250.721.7691 Data Centre Services, University Systems, University of Victoria

4 9

Re: [syslog-ng] How to unsubscribe from this mailers
by Mayur Saxena (maysaxen) 02 Oct '15

02 Oct '15

2 1

TCP Resets
by Michael Starks 02 Oct '15

02 Oct '15

I am running syslog-ng 3.4.7 as distributed by ELSA. I have a problem whereby nxlog agents (v2.9.1347) making a new TCP connection to the syslog-ng server repeatedly show lines like this in nxlog.log: 2015-09-28 14:28:25 INFO reconnecting in 1 seconds Many connections are working, but it especially has a problem with new agents coming online. Sometimes when I restart syslog-ng it helps, but not always. Other than the standard ELSA config, I am simply also writing to a filesystem destination. If I remove the elsa.pl destination, I still have problems. The problem isn't necessarily related to just nxlog agents. I also have a Cisco proxy that is using TCP to send logs, and that is also getting RSTs sent to it. I tried v3.6.3 and that didn't seem to help. A packet capture shows the SYN from the agent, a SYN ACK, another SYN, then a FIN *prior* to the data packet being received. Immediately after the data packet, the log server sends a RST. As a result, I have a bunch of TIME_WAITs open. No host firewall is running. I am currently only bursting to about 350 EPS and I have syslog-ng running in the foreground looking for dropped packets like so (there haven't been any): /usr/local/syslog-ng-3.4.7/sbin/syslog-ng -Ftvd 2>&1 | grep "dropping messages" Any assistance would be greatly appreciated! Currently loaded syslog-ng.conf: # cat /usr/local/syslog-ng/etc/syslog-ng.conf @version: 3.2 options { log_fifo_size(30000); #threaded(yes); # enable if using Syslog-NG 3.3.x use_dns (no); # IMPORTANT! A busy syslog server can become unresponsive if DNS lookups fail, so we disable them. use_fqdn (no); keep_hostname (yes); }; source s_network { # no-multi-line means parse message as-is, newlines included, to allow for multi-line messages tcp(flags(no-multi-line)); udp(flags(no-multi-line)); }; source s_internal { internal(); }; source s_realtime { pipe("/data/elsa/tmp/realtime" flags(expect-hostname)); }; source s_import { pipe("/data/elsa/tmp/import" flags(expect-hostname syslog-protocol)); }; source s_ops { unix-dgram("/data/elsa/tmp/ops" flags(expect-hostname syslog-protocol no-multi-line)); }; parser p_db { db-parser(file("/usr/local/elsa/node/conf/merged.xml")); }; filter f_rewrite_cisco_program { match('^(%[A-Z\_]+\-\d\-[0-9A-Z\_]+): ([^\n]+)' value("MSGONLY") type("pcre") flags("store-matches" "nobackref")); }; filter f_rewrite_cisco_program_2 { match('^[\*\.]?(?:Jan|Feb|Mar|Apr|May|Jun|Jul|Aug|Sep|Oct|Nov|Dec)\s+\d{1,2}\s\d{1,2}:\d{1,2}:\d{1,2}(?:\.\d+)?(?: [A-Z]{3})?: (%[^:]+): ([^\n]+)' value("MSGONLY") type("pcre") flags("store-matches" "nobackref")); }; filter f_rewrite_cisco_program_3 { match('^\d+[ywdh]\d+[ywdh]: (%[^:]+): ([^\n]+)' value("MSGONLY") type("pcre") flags("store-matches" "nobackref")); }; filter f_rewrite_cisco_program_4 { match('^\d{6}: [\*\.]?(?:Jan|Feb|Mar|Apr|May|Jun|Jul|Aug|Sep|Oct|Nov|Dec)\s+\d{1,2}\s\d{1,2}:\d{1,2}:\d{1,2}(?:\.\d+)?(?: [A-Z]{3})?: (%[^:]+): ([^\n]+)' value("MSGONLY") type("pcre") flags("store-matches" "nobackref")); }; filter f_unclassified { "${.classifier.class}" eq "unknown" }; rewrite r_unparsed { set("98", value("MSGID")); }; rewrite r_cisco_program { set("$1", value("PROGRAM") condition(filter(f_rewrite_cisco_program) or filter(f_rewrite_cisco_program_2) or filter(f_rewrite_cisco_program_3) or filter(f_rewrite_cisco_program_4))); set("$2", value("MESSAGE") condition(filter(f_rewrite_cisco_program) or filter(f_rewrite_cisco_program_2) or filter(f_rewrite_cisco_program_3) or filter(f_rewrite_cisco_program_4))); }; rewrite r_snare { subst("MSWinEventLog.+(Security|Application|System).+", "$1", value("PROGRAM") flags(global)); }; # URL encode any literal pipes so the tab rewriter doesn't introduce ambiguity rewrite r_from_pipes { subst('\|', "%7C", value("MESSAGE") flags(global) condition(program("bro_*" type(glob)))); }; #rewrite r_from_pipes { subst('\|', "%7C", value("MESSAGE") flags(global)); }; rewrite r_pipes { subst("\t", "|", value("MESSAGE") flags(global)); }; rewrite r_host { set("$SOURCEIP", value("HOST")); }; rewrite r_extracted_host { set("$pdb_extracted_sourceip", value("HOST") condition("$pdb_extracted_sourceip" != "")); }; rewrite r_extracted_timestamp { set("$S_UNIXTIME", value("SOURCE")); set("$pdb_extracted_timestamp", value("SOURCE") condition("$pdb_extracted_timestamp" != "")); }; template t_db_parsed { template("$R_UNIXTIME\t$HOST\t$PROGRAM\t${.classifier.class}\t$MSGONLY\t${i0}\t${i1}\t${i2}\t${i3}\t${i4}\t${i5}\t${s0}\t${s1}\t${s2}\t${s3}\t${s4}\t${s5}\n"); }; template t_db_parsed_import { template("$MSGID\t$SOURCE\t$HOST\t$PROGRAM\t${.classifier.class}\t$MSGONLY\t${i0}\t${i1}\t${i2}\t${i3}\t${i4}\t${i5}\t${s0}\t${s1}\t${s2}\t${s3}\t${s4}\t${s5}\n"); }; template t_ops { template("$S_UNIXTIME\t$HOST\t$PROGRAM\t$MSGID\t$MSGONLY\t${.SDATA.elsa@32473.line_number}\t${.SDATA.elsa@32473.pid}\t${.SDATA.elsa@32473.client}\t${.SDATA.elsa@32473.qid}\t\t\t${.SDATA.elsa@32473.priority}\t${.SDATA.elsa@32473.file}\t${.SDATA.elsa@32473.method}\t${.SDATA.elsa@32473.hostname}\t\t\n"); }; template t_unparsed { template("Unparsed host=$HOST program=$PROGRAM msg=$MSGONLY\n"); }; destination d_elsa { program("perl /usr/local/elsa/node/elsa.pl -c /etc/elsa_node.conf" template(t_db_parsed)); }; destination d_elsa_import { program("perl /usr/local/elsa/node/elsa.pl -c /etc/elsa_node.conf -f __IMPORT__" template(t_db_parsed_import)); }; destination d_elsa_ops { program("perl /usr/local/elsa/node/elsa.pl -c /etc/elsa_node.conf -f __OPS__" template(t_ops)); }; destination d_unclassified { unix-dgram("/data/elsa/tmp/ops" flags(syslog-protocol) template(t_unparsed)); }; destination d_debug { unix-dgram("/data/elsa/tmp/debug" template(t_db_parsed)); }; # Custom destination -Mike destination d_hosts { file("/data/logs/$HOST/$HOST.log"); }; include /etc/elsa_syslog-ng.conf; #log { # source(s_ops); # destination(d_elsa_ops); # flags(final); #}; #log { # source(s_network); # source(s_realtime); # rewrite(r_host); # rewrite(r_cisco_program); # rewrite(r_snare); # rewrite(r_from_pipes); # rewrite(r_pipes); # parser(p_db); # rewrite(r_extracted_host); # ###FILTER_UNPARSED###log { filter(f_unclassified); rewrite(r_unparsed); destination(d_unclassified); flags(final); }; # log { destination(d_elsa); }; # log { destination(d_debug); }; #}; #log { # source(s_import); # rewrite(r_cisco_program); # rewrite(r_snare); # rewrite(r_from_pipes); # rewrite(r_pipes); # parser(p_db); # rewrite(r_extracted_host); # rewrite(r_extracted_timestamp); # destination(d_elsa_import); #}; # cat /etc/elsa_syslog-ng.conf ################################## #### Global Options ######### ################################## options { check_hostname(yes); create_dirs(yes); dir_group(log_analyst); dir_owner(root); dir_perm(0550); group(log_analyst); owner(root); perm(0640); # Let's not be vulnerable to DNS attacks and general problems dns_cache(no); use_dns(no); # Keep the original hostname (good for relay servers) keep_hostname(yes); # Keep original timestamp (not received time) - this could be forged, so use caution during forensincs examinations keep_timestamp(yes); }; ################################## ########### Sources ######### ################################## # Source for local system logs source s_local { # message generated by Syslog-NG internal(); # standard Linux log source (this is the default place for the syslog() # function to send logs to) unix-stream("/dev/log"); # messages from the kernel file("/proc/kmsg" program_override("kernel: ")); }; ################################## ######### Destinations ########## ################################## # This is the directory structure that will be automatically created as logs come in destination d_hosts { file("/data/logs/$HOST/$HOST.log"); };

2 2

Syslog ng and Mysql
by Arash Shams 01 Oct '15

01 Oct '15

Hello ELK solution is not suitable for us to collect all logs in one server . i prefer to use syslog-ng to collect all logs in Mysql database. is there possible to send all servers log to one syslog-ng server and that collect them to database ? anyone can give me a good configuration file or some examples in this case ? thank you

7 20

Issues with lowercase hostname on 3.2.5
by Kay N. Avila 25 Sep '15

25 Sep '15

Hello, I'm running syslog-ng 3.2.5 from EPEL on RHEL and having an issue with the $HOST variable. My destination is destination device_log { file("/<snipped>/$HOST.log"); }; and that is working, except the file is created with an uppercase name if the host reports it that way. I'd like them to all be consistently lowercase. I can't use a DNS look up instead since not all of the hosts have reverse DNS set for their IPs. I've tried normalize-hostnames(yes); in the global options but it doesn't seem to make a difference. Also I don't believe $(lowercase ...) is available to me in 3.2.5. I've read through the documentation and have come up empty. What am I missing? Apologies if this has been asked before; I searched the list but couldn't find anything. Thanks for any pointers, Kay Avila

3 4

kafka destination of syslog-ng 3.7.1
by Szabó, István 22 Sep '15

22 Sep '15

Hi, kafka destination using the official apache kafka java libraries have been release in syslog-ng 3.7.1. May I ask if anyone is already using it and can share any experience with using it? Thanks and best regards, Istvan

1 0

Re: [syslog-ng] unofficial syslog-ng 3.7.1 Debian packages
by Budai, László 22 Sep '15

22 Sep '15

Hi, I think this is fixed by https://github.com/balabit/syslog-ng/pull/684 . Debian packages will be updated soon. L. On Tue, Sep 22, 2015 at 11:27 AM, Németh, Balázs <balazs.nemeth(a)balabit.com> wrote: > Hello, > > Which package should provide the following file? > /usr/share/syslog-ng/xsd/patterndb-4.xsd > > dpkg -S /usr/share/syslog-ng/xsd/patterndb-4.xsd > Returns nothing. > > Without that I cannot validate my patterndb sets. > > warning: failed to load external entity > "/usr/share/syslog-ng/xsd/patterndb-4.xsd" > Schemas parser error : Failed to locate the main schema resource at > '/usr/share/syslog-ng/xsd/patterndb-4.xsd'. > WXS schema /usr/share/syslog-ng/xsd/patterndb-4.xsd failed to compile > /tmp/O1tb4DMkvj.xml: xmllint returned an error, the executed command was: > xmllint --noout --nonet --schema /usr/share/syslog-ng/xsd/patterndb-4.xsd > /tmp/O1tb4DMkvj.xml > > dpkg -l | grep syslog-ng >> ii syslog-ng-core 3.7.1-1 >> amd64 Enhanced system logging daemon (core) >> ii syslog-ng-mod-journal 3.7.1-1 >> >> lsb_release -a >> No LSB modules are available. >> Distributor ID: Debian >> Description: Debian GNU/Linux 8.2 (jessie) >> Release: 8.2 >> Codename: jessie >> > > Thanks in advance, > > 2015-08-17 16:46 GMT+02:00 Budai, László <laszlo.budai(a)balabit.com>: > >> Hi, >> >> I've just created my unofficial OBS repo[1] for syslog-ng 3.7.1 series >> (current version: 3.7.1-1 ). >> >> List of supported OSs (i386/amd64): >> * Debian 7.0 >> * Debian 8.0 >> * Ubuntu 12.04 >> * Ubuntu 14.04 >> * Ubuntu 14.10 >> * Ubuntu 15.04 >> >> >> Install >> ------- >> >> example: Debian 8.0 >> >> 1. get release key >> wget -qO - >> http://download.opensuse.org/repositories/home:/laszlo_budai:/syslog-ng/Deb… >> | sudo apt-key add - >> >> 2. add repo to APT sources >> eg.: /etc/apt/sources.list.d/syslog-ng-obs.list >> deb >> http://download.opensuse.org/repositories/home:/laszlo_budai:/syslog-ng/Deb… >> ./ >> >> Then `apt-get update` and `apt-get install syslog-ng-core=3.7.1-1` >> >> You can replace Debian_8.0 to >> * Debian_7.0 >> * xUbuntu_12.04, >> * xUbuntu_14.04 >> * xUbuntu_14.10 >> * xUbuntu_15.04 >> >> (`x` before Ubuntu is not a typo ;-) ) >> >> Available packages: >> * syslog-ng-core >> * syslog-ng-dbg >> * syslog-ng-dev >> * syslog-ng-mod-stomp >> * syslog-ng-mod-amqp >> * syslog-ng-mod-elastic >> * syslog-ng-mod-geoip >> * syslog-ng-mod-graphite >> * syslog-ng-mod-hdfs >> * syslog-ng-mod-http >> * syslog-ng-mod-java >> * syslog-ng-mod-java-common-lib >> * syslog-ng-mod-journal >> * syslog-ng-mod-json >> * syslog-ng-mod-kafka >> * syslog-ng-mod-mongodb >> * syslog-ng-mod-python >> * syslog-ng-mod-redis >> * syslog-ng-mod-riemann >> * syslog-ng-mod-smtp >> * syslog-ng-mod-sql >> * syslog-ng-mod-stomp >> >> >> Notes to the java destinations >> ------------------------------ >> >> About the >> >> ``` >> Error opening plugin module; module='mod-java', error='libjvm.so: cannot >> open shared object file: No such file or directory' >> ``` >> >> problem. >> >> libmod-java.so is linked to the libjvm.so library. >> If your java installation doesn't create a symlink in /usr/lib directory >> to libjvm.so then >> you have to modify your ld.so.conf or expand your LD_LIBRARY_PATH with >> the containing library >> before starting syslog-ng. >> >> eg.: >> >> LD_LIBRARY_PATH=/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/amd64/server:$LD_LIBRARY_PATH >> >> >> Links >> -------- >> [1] https://build.opensuse.org/project/show/home:laszlo_budai:syslog-ng >> [2] https://github.com/balabit/syslog-ng/releases/tag/syslog-ng-3.7.1 >> >> regards, >> Laszlo Budai >> >> > > > -- > > Balázs NÉMETH > System Engineer > > e-mail: balazs.nemeth(a)balabit.com > phone: +36-1-880-6740 > > BalaBit-Europe Kft.www.balabit.com > H-1117 Alíz str. 2. Budapest > > This Communication is Confidential. We only send and receive email on > the basis of the term set out at http://www.balabit.com/disclaimer/. > >

1 0

question on elasticsearch & indexint structured data fields
by jrhendri＠roadrunner.com 11 Sep '15

11 Sep '15

Hi, I am testing elasticsearch with the 3.7.1 ose build on ubuntu 14.04 and have some questions regarding how to get elasticsearch & kibana to "see" the individual fields within a structured syslog message. I have tried a few different formats but all the >key>=<value> pairs appear within the MESSAGE part. For example: MESSAGE 2015-09-09T17:00:06.775 0055-inet-fw-node0 RT_FLOW - RT_FLOW_SESSION_CREATE_LS [junos(a)2636.1.1.1.2.35 logical-system-name="internetVR" source-address="143.115.190.50" source-port="42241" destination-address="70.39.233.137" destination-port="53" service-name="junos-dns-udp" nat-source-address="143.115.190.50" nat-source-port="42241" nat-destination-address="70.39.233.137" nat-destination-port="53" src-nat-rule-name="None" dst-nat-rule-name="None" protocol-id="17" policy-name="Device-Zone-903" source-zone-name="dns-b2b" destination-zone-name="internet" session-id-32="80968105" username="N/A" roles="N/A" packet-incoming-interface="reth3.120" application="UNKNOWN" nested-application="UNKNOWN" encrypted="UNKNOWN"] The "Available Fields" I see in kibana are: @timestamp tDATE tFACILITY tHOST tMESSAGE tPRIORITY tPROGRAM t_id t_index t_type I would like to be able to have each field recognized separately so that kibana could search for specific things like "source-address", etc. I thought these would be available under the .SDATA. set, but apparently I missed something. Is this possible (and I am just lacking understanding) or am I expecting too much? These are the last three tests I have used within the elasticsearch destination and they all essentially result in the same thing. #!# option( "message_template", "$(format-json --scope nv_pairs)\n") #!# option( "message_template", "$(format-json --scope rfc5424 --exclude DATE --key ISODATE @timestamp=${ISODATE})\n") option( "message_template", "$(format-json --scope rfc5424 @timestamp=${ISODATE} --key .SDATA.* ) \n" ) Thanks for any help or guidance! Jim

3 3

Fields don't appear on kibana.
by Jacek Drewniak 10 Sep '15

10 Sep '15

Hello, When I am putting new fields to elasticsearch for example using rewrite, they don't appear on kibana. But when I prefix name this fields by ".SDATA.meta" - they appear. For example: http://pastebin.com/8teCTC0x -- fields dosn't appears - kibana thinks thats are empty: http://pastebin.com/ikXMesWj -- fields appears. This same with geoip. fields. This is my kibana fields: http://i.imgur.com/pWIhsHf.png?1 Can You help me with the problem? Jacek -- *Jacek Drewniak* R&D *email*: jacek.drewniak(a)oort.in *mobile*: *+**48 696 151 670* *website*: www.oort.in AWARDS Bluetooth Breakthrough Award Finalist CES 2015 Envisioneering Innovation & Design Award Winner Tech Trailblazers Awards Winner Most exciting company at Bluetooth Media Event in New York 2014 Polish Agency for Enterprise Development Award Winner

4 14

About unofficial Debian/Ubuntu packages
by Gergely Nagy 09 Sep '15

09 Sep '15

Hi! As some of you may remember, I used to build and host unofficial Debian and Ubuntu packages[1] for syslog-ng, but I stopped doing so for a number of reasons a while ago. [1]: https://asylum.madhouse-project.org/projects/debian/ Recently, I felt the need to use a more recent (3.7.1) syslog-ng, so I started to update the packaging. Right now, the results are on GitHub[2], but the package repositories are not updated. Before I do that, I'd like to evaluate if it is worth the effort, because there's a huge gap between building packages for my own use, and building them for others too. For one, I only care about Debian stable on armhf, which is, well, far from the most popular architecture. [2]: https://github.com/algernon/syslog-ng-debian So, if you or anyone you know would benefit from the continuation of my unofficial (but high quality, and supported) Debian/Ubuntu packages, please let me know off-list, I'd like to talk. Thanks in advance! -- |8]

1 0

Parsing message in unstructured rows
by Giovanni Mancuso 08 Sep '15

08 Sep '15

Hi, I have an application that log date in every rows. The problem is that the string isn't in specific part of MESSAGE, but it could be the first element or the last element, or in the middle :-) :-) For example (only MESSAGE): User: user1(a)example.com *Date: 12/12/2014* Status: OK User: user2(a)example.com ID: 1234 Status: DEL ....... *Date: 03/05/2014 Date: 05/08/2015 *User:**user3@example.com ....... Stauts: OK* *......... I want split the log in more files arranged in different path builded by the date information, for example /LOGS/YYYY/MM/DD/mylog.log. I do: Konsole output rewrite r_rewrite_set { set("$(python get_data)", value("APP.DATE")); }; python{ import re def get_data(logmsg): out=None vars(logmsg) out = re.findall(" Date: (\d\d/\d\d/\d\d\d\d) ", logmsg.MESSAGE) if len(out) == 1: return out[0] else: raise Exception("Invalid match") }; In this way i have in APP.DATE the date. Now i have a some questions: 1) Is there another way to do this without python? 2) In this way for every message, syslog-ng forks and exec a python interpreter? 3) Is there a way to add custom SDATA field from python? Or is there a way to create APP.DATE from python without rewrite rule? 4) Is there a documentation about python{}? I only found a post in a blog. Thanks

2 3

Re: [syslog-ng] Parsing message in unstructured rows
by Jim Hendrick 08 Sep '15

08 Sep '15

Would patterndb be an option?Thinking one to match each log line type each parsing out the date and whatever else you want. Jim Sent from my Verizon Wireless 4G LTE smartphone -------- Original message -------- From: Giovanni Mancuso <giovanni.mancuso(a)par-tec.it> Date: 09/07/2015 3:26 PM (GMT-05:00) To: "Scheidler, Balázs" <balazs.scheidler(a)balabit.com>, Syslog-ng users' and developers' mailing list <syslog-ng(a)lists.balabit.hu> Subject: Re: [syslog-ng] Parsing message in unstructured rows Il 07 settembre 2015 21:10:59 "Scheidler, Balázs" <balazs.scheidler(a)balabit.com> ha scritto: > On Sep 7, 2015 6:11 PM, "Giovanni Mancuso" <giovanni.mancuso(a)par-tec.it> > wrote: > > > > Hi, > > > > I have an application that log date in every rows. The problem is that > the string isn't in specific part of MESSAGE, but it could be the first > element or the last element, or in the middle :-) :-) > > > > For example (only MESSAGE): > > User: user1(a)example.com Date: 12/12/2014 Status: OK > > User: user2(a)example.com ID: 1234 Status: DEL ....... Date: 03/05/2014 > > Date: 05/08/2015 User: user3(a)example.com ....... Stauts: OK > > ......... > > > > I want split the log in more files arranged in different path builded by > the date information, for example /LOGS/YYYY/MM/DD/mylog.log. > > > > I do: > > rewrite r_rewrite_set { > > set("$(python get_data)", value("APP.DATE")); > > }; > > > > python{ > > import re > > def get_data(logmsg): > > out=None > > vars(logmsg) > > out = re.findall(" Date: (\d\d/\d\d/\d\d\d\d) ", logmsg.MESSAGE) > > if len(out) == 1: > > return out[0] > > else: > > raise Exception("Invalid match") > > }; > > > > In this way i have in APP.DATE the date. > > Now i have a some questions: > > 1) Is there another way to do this without python? > > Well, this should be possible with a simple regexp filter. I try to use a filter regexp, but i don't find the solution. Have you an example? > > > 2) In this way for every message, syslog-ng forks and exec a python > interpreter? > > No, it embeds a Python interpreter. > > > 3) Is there a way to add custom SDATA field from python? Or is there a > way to create APP.DATE from python without rewrite rule? > > Not right now. > > > 4) Is there a documentation about python{}? I only found a post in a blog. > > It's being prepared by the tech writer team in BalaBit > > > > > Thanks > > > > > > > ______________________________________________________________________________ > > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng > > Documentation: > http://www.balabit.com/support/documentation/?product=syslog-ng > > FAQ: http://www.balabit.com/wiki/syslog-ng-faq > > > > Inviato con AquaMail per Android http://www.aqua-mail.com

2 1

Problem with a pattern in 3.6.4
by C. L. Martinez 04 Sep '15

04 Sep '15

Hi all, I am trying to configure a pattern for the following log entry in syslog-ng 3.6.4: idpsnort01 09/03-13:18:41.935109 [**] [3:19187:6] PROTOCOL-DNS TMG Firewall Client long host entry exploit attempt [**] [Classification: Attempted User Privilege Gain] [Priority: 1] [AppID: dns] {UDP} 80.58.61.250:53 -> 10.196.0.67:60941 My pattern is: <pattern>@ESTRING:s3: @@ESTRING:: @@ESTRING:: [**]@ @QSTRING:s0:[]@ @ESTRING:s1: [**] [@Classification:@QSTRING:s2: ]@ [Priority : @NUMBER:i0:@] @@[AppID: @QSTRING:s4: ] @QSTRING:i1:{}@ @IPv4:i2:@:@NUMBER:i3:@ -> @IPv4:i4:@:@NUMBER:i5:@</pattern> If you try it, you can see it doesn't works. Problem is with the following part of the message: [Priority: 1] [AppID: dns] I need to escape "] [AppID:" and catch "dns" field, but I have tried some configs withut luck. Any idea?? Many thanks.

2 2

insider 2015-09: syslog-ng 3.7.1 released; ARM; configuring Java; Python trick;
by Czanik, Péter 03 Sep '15

03 Sep '15

Dear syslog-ng users, This is the 43th issue of the syslog-ng Insider, a monthly newsletter that brings you syslog-ng-related news. NEWS syslog-ng Open Source Edition 3.7.1 released -------------------------------------------- The first syslog-ng 3.7 release is now available together with a matching incubator 0.5 release. Major new features include Java and Python destinations migrated from the incubator, and a collection of Java based “big data” destinations: Elasticsearch, Hadoop and Kafka. For a complete list of new features, check the release notes available at https://github.com/balabit/syslog-ng/releases/tag/syslog-ng-3.7.1 syslog-ng on ARM ---------------------- As syslog-ng is in many Linux distributions, it is also available on ARM. Recently we had a chance to evaluate syslog-ng on a 64bit ARM machine. The results are mixed, but very promising. If you already use syslog-ng on ARM or plan to use it, contact me and share your use case and experiences! https://czanik.blogs.balabit.com/2015/07/syslog-ng-on-64bit-arm/ Getting started with the new Java based destinations ---------------------------------------------------- Installing the new Java based destination drivers requires a bit more preparation than installing the syslog-ng package and writing syslog-ng.conf, as was previously. This “HowTo” helps you to get started with Java based destinations on popular Linux distributions and FreeBSD: https://czanik.blogs.balabit.com/2015/08/getting-started-with-syslog-ng-3-7… A nice Python trick ------------------- syslog-ng Open Source Edition 3.7 allows you to process your log messages in Python, making it easy to solve a thousand use-cases. Here is one of the many possibilities: https://guest.blogs.balabit.com/2015/09/processing-log-messages-with-python… NEW RELEASES syslog-ng OSE 3.7.1: https://github.com/balabit/syslog-ng/releases/tag/syslog-ng-3.7.1 syslog-ng PE 5F4: https://jluby.blogs.balabit.com/2015/07/23/syslog-ng-premium-edition-5f4-re… syslog-ng Store Box 4F2: https://jluby.blogs.balabit.com/2015/08/10/syslog-ng-store-box-4f2-released/ Your feedback and news tips about the next issue is welcome at documentation(a)balabit.com To read this newsletter on-line, visit: http://insider.blogs.balabit.com/ Peter Czanik (CzP) <peter.czanik(a)balabit.com> BalaBit IT Security / syslog-ng upstream http://czanik.blogs.balabit.com/ https://twitter.com/PCzanik

1 0

Advice on the right destination
by Giovanni Mancuso 31 Aug '15

31 Aug '15

Hi, I am writing to ask your advice on a solution I'm thinking. I have 12 servers with postfix, amavisd-new and other custom software that manage the e-mail system and I was working in a web interface to analysis the logs and correlation with the ability to search for certain fields (from, to, message-id , date). All applications send the logs to a centralized syslog-ng, and I was trying to understand which type of "destination" is better to use to ensure the rapid search. I was analyzing the possibility of using elasticsearch, but I don't know neither it or its performances. The quantity of data is very high, about 3TB of data monthly or each machine, with 2 years of retention. What do you think about? Have you any suggestions? Thanks -- *Giovanni Mancuso* System Architect *T* 06.9826.9600 *M* +39.340.65.80.739 *F* 06.9826.9680 P.zza S.Benedetto da Norcia, 33 - 00071 Pomezia (RM) Par-Tec S.p.A. <http://www.par-tec.it> Web Site <http://www.par-tec.it> info(a)par-tec.it <mailto:info@par-tec.it> Pagina Facebook <https://www.facebook.com/ParTecSpA> Profilo Twitter <https://twitter.com/partecspa> Pagina LinkedIn <https://www.linkedin.com/company/par-tec/> Canale YouTube <https://www.youtube.com/user/ParTecSpA> CONFIDENZIALE: Questo messaggio ed i suoi allegati sono di carattere confidenziale per i destinatari in indirizzo. È vietato l'inoltro non autorizzato a destinatari diversi da quelli indicati nel messaggio originale. Se ricevuto per errore, l'uso del contenuto è proibito; si prega di comunicarlo al mittente e cancellarlo immediatamente.

2 1

Can't get syslog-ng to start...
by Mason, Tron 31 Aug '15

31 Aug '15

All, I keep getting the following when attempting to start syslog-ng: /usr/local/etc/syslog-ng/syslog-ng.conf: line 10: @version:: command not found /usr/local/etc/syslog-ng/syslog-ng.conf: line 12: syntax error near unexpected token `(' /usr/local/etc/syslog-ng/syslog-ng.conf: line 12: `options { use_dns(yes);' Not sure why I'm getting these messages when I'm running a later version: bash-3.2# /usr/local/sbin/syslog-ng -V syslog-ng 3.2.4 Installer-Version: 3.2.4 Revision: ssh+git://bazsi@git.balabit//var/scm/git/syslog-ng/syslog-ng-ose--mainline--3.2#master#ef7b91e4a1b1f9628c66138b4ae83de7e4c697c6 Compile-Date: Jun 9 2011 05:24:53 Enable-Threads: off Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-Sun-STREAMS: on Enable-IPv6: on Enable-Spoof-Source: on Enable-TCP-Wrapper: off Enable-SSL: on Enable-SQL: off Enable-Linux-Caps: off Enable-Pcre: on Enable-Pacct: off bash-3.2# more /usr/local/etc/syslog-ng/syslog-ng.conf # Syslog-ng example configuration file for Solaris # # Copyright (c) 1999 Balazs Scheidler # $Id: syslog-ng.conf.solaris,v 1.2 1999/11/15 12:30:41 bazsi Exp $ # Current syslog-ng binary version 1.5.16 # # Solaris 2.5.1 and below uses the STREAMS driver, above extends it # with doors. For 2.5.1 remove the door() option from the source declaration. # @version: 3.2.4 options { use_dns(yes); # use_time_recvd(yes); chain_hostnames(off); log_fifo_size(1000); }; #source local { sun-streams("/dev/log" door("/var/run/syslog_door")); }; source local { sun-streams("/dev/log" door("/etc/.syslog_door")); internal(); }; #source s_tcp { tcp(port(5014) max-connections(100); }; #source s_tcp { tcp(ip(0.0.0.0) port(5014)); }; source s_udp { udp(); }; # source s_udp { udp(ip(0.0.0.0) port(514)); }; ..........

3 3

Some development notes on 3.7
by Budai, László 26 Aug '15

26 Aug '15

Hi, after the 3.7 is out we would had to create a new 3.7/master branch, but I would postpone this step at a later point. Why? Because I know that there are some fixes in the queue what we would have to forward port to 3.8 and some of them to 3.6. This branching will be a lazy one: we will branch it when we have to, but not earlier. L.

1 0

Forwarding system startup messages
by Saurabh Shukla 25 Aug '15

25 Aug '15

I am running syslog-ng 3.6.4 and I have the following destination and log path configured that forwards all messages to the destination: destination remote { network("remote.example.com" port(514) transport(tcp) log_fifo_size(2048)); }; log { source(s_all); destination(remote); flags(flow-control);}; When the system reboots, I see that startup messages from the kernel are logged into /var/log/syslog. syslog-ng establishes a connection to the remote destination around 10 sec after the first message was logged into /var/log/syslog. However, it fails to forward any message that was logged into /var/log/syslog during the first 10 seconds even though I have the output buffer and flow control configured. Is this a bug in syslog-ng or am I missing some configuration steps? Thanks, -- Saurabh

2 8

"Error resolving hostname" for UDP Destination
by David Hauck 24 Aug '15

24 Aug '15

Hi, I've experienced problems starting syslog-ng related to hostname lookup on UDP destinations for a while now and am finally getting back to looking into some kind of mitigation for the exceptions. Here's what I see: if I have a destination configured with an endpoint that requires name lookup and the name lookup fails, I see the following error when attempting to start (or reload) syslog-ng: Starting syslog-ng: Error resolving hostname; host='test.nacc.netacquire.dom' Error initializing message pipeline; Unfortunately, this results in the entire process failing to start. If syslog-ng is otherwise running and I attempt to reload the configuration (with the problematic name present) I don't see any errors logged, but I do see that the process is now defunct (and is no longer running): root 18674 1 0 10:03 ? 00:00:00 supervising syslog-ng root 18701 18674 0 10:04 ? 00:00:00 [syslog-ng] <defunct> Here's the simple destination in question: destination d_NAaudit_Prio { file("/var/log/netacquire/audit_log" template(t_NAFormat_Prio)); udp("test.nacc.netacquire.dom" port(514) template(t_NAFormat_Prio)); }; I'm wondering if there might be some way to mitigate this exception. For example: 1. Log the error and allow syslog-ng to otherwise continue to start/function without the destination being configured/used. 2. As above, and suspend the destination but allow it to continue to attempt to probe the name lookup so that if the resolution does eventually succeed the destination can be resumed. Are either of the above options feasible/available? Are there any other ideas for how to mitigate this exception so that syslog-ng can be started/reloaded successfully (other than detecting this event a priori and removing the destination from the configuration before attempts to start/reload)? Thanks in advance! -David PS: Here's my version information: # syslog-ng --version syslog-ng 3.5.4.1 Installer-Version: 3.5.4.1 Revision: ssh+git://algernon@git.balabit/var/scm/git/syslog-ng/syslog-ng-ose--mainline--3.5#master#4090ee62163780ae68a0c83cfdc23998c904fe97 Compile-Date: May 19 2015 17:04:36 Available-Modules: confgen,afstomp,afmongodb,linux-kmsg-format,cryptofuncs,afsocket,csvparser,dbparser,afamqp,afsocket-tls,syslogformat,afsocket-notls,afprog,basicfuncs,afuser,affile,system-source Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-IPv6: on Enable-Spoof-Source: off Enable-TCP-Wrapper: on Enable-Linux-Caps: off Enable-Pcre: on

3 13

How to disable DNS resolution in syslog-ng?
by Peter Volkov 24 Aug '15

24 Aug '15

Hi guys. Whenever syslog-ng start without networking it hangs trying to resolve hostname it runs on (checked with tcpdump). Now I've tried to add use_dns(no); into configuration file but delay still in place: options { use_dns(no); use_fqdn(no); } I've tried to google, but still failed to find answer: Is there any way to disable this hostname resolution delay? Thank in advance, -- Peter.

2 1