September 2015 - syslog-ng - lists.balabit.hu

Fields don't appear on kibana.
by Jacek Drewniak 10 Sep '15

10 Sep '15

Hello, When I am putting new fields to elasticsearch for example using rewrite, they don't appear on kibana. But when I prefix name this fields by ".SDATA.meta" - they appear. For example: http://pastebin.com/8teCTC0x -- fields dosn't appears - kibana thinks thats are empty: http://pastebin.com/ikXMesWj -- fields appears. This same with geoip. fields. This is my kibana fields: http://i.imgur.com/pWIhsHf.png?1 Can You help me with the problem? Jacek -- *Jacek Drewniak* R&D *email*: jacek.drewniak(a)oort.in *mobile*: *+**48 696 151 670* *website*: www.oort.in AWARDS Bluetooth Breakthrough Award Finalist CES 2015 Envisioneering Innovation & Design Award Winner Tech Trailblazers Awards Winner Most exciting company at Bluetooth Media Event in New York 2014 Polish Agency for Enterprise Development Award Winner

4 14

About unofficial Debian/Ubuntu packages
by Gergely Nagy 09 Sep '15

09 Sep '15

Hi! As some of you may remember, I used to build and host unofficial Debian and Ubuntu packages[1] for syslog-ng, but I stopped doing so for a number of reasons a while ago. [1]: https://asylum.madhouse-project.org/projects/debian/ Recently, I felt the need to use a more recent (3.7.1) syslog-ng, so I started to update the packaging. Right now, the results are on GitHub[2], but the package repositories are not updated. Before I do that, I'd like to evaluate if it is worth the effort, because there's a huge gap between building packages for my own use, and building them for others too. For one, I only care about Debian stable on armhf, which is, well, far from the most popular architecture. [2]: https://github.com/algernon/syslog-ng-debian So, if you or anyone you know would benefit from the continuation of my unofficial (but high quality, and supported) Debian/Ubuntu packages, please let me know off-list, I'd like to talk. Thanks in advance! -- |8]

1 0

Parsing message in unstructured rows
by Giovanni Mancuso 08 Sep '15

08 Sep '15

Hi, I have an application that log date in every rows. The problem is that the string isn't in specific part of MESSAGE, but it could be the first element or the last element, or in the middle :-) :-) For example (only MESSAGE): User: user1(a)example.com *Date: 12/12/2014* Status: OK User: user2(a)example.com ID: 1234 Status: DEL ....... *Date: 03/05/2014 Date: 05/08/2015 *User:**user3@example.com ....... Stauts: OK* *......... I want split the log in more files arranged in different path builded by the date information, for example /LOGS/YYYY/MM/DD/mylog.log. I do: Konsole output rewrite r_rewrite_set { set("$(python get_data)", value("APP.DATE")); }; python{ import re def get_data(logmsg): out=None vars(logmsg) out = re.findall(" Date: (\d\d/\d\d/\d\d\d\d) ", logmsg.MESSAGE) if len(out) == 1: return out[0] else: raise Exception("Invalid match") }; In this way i have in APP.DATE the date. Now i have a some questions: 1) Is there another way to do this without python? 2) In this way for every message, syslog-ng forks and exec a python interpreter? 3) Is there a way to add custom SDATA field from python? Or is there a way to create APP.DATE from python without rewrite rule? 4) Is there a documentation about python{}? I only found a post in a blog. Thanks

2 3

Re: [syslog-ng] Parsing message in unstructured rows
by Jim Hendrick 08 Sep '15

08 Sep '15

Would patterndb be an option?Thinking one to match each log line type each parsing out the date and whatever else you want. Jim Sent from my Verizon Wireless 4G LTE smartphone -------- Original message -------- From: Giovanni Mancuso <giovanni.mancuso(a)par-tec.it> Date: 09/07/2015 3:26 PM (GMT-05:00) To: "Scheidler, Balázs" <balazs.scheidler(a)balabit.com>, Syslog-ng users' and developers' mailing list <syslog-ng(a)lists.balabit.hu> Subject: Re: [syslog-ng] Parsing message in unstructured rows Il 07 settembre 2015 21:10:59 "Scheidler, Balázs" <balazs.scheidler(a)balabit.com> ha scritto: > On Sep 7, 2015 6:11 PM, "Giovanni Mancuso" <giovanni.mancuso(a)par-tec.it> > wrote: > > > > Hi, > > > > I have an application that log date in every rows. The problem is that > the string isn't in specific part of MESSAGE, but it could be the first > element or the last element, or in the middle :-) :-) > > > > For example (only MESSAGE): > > User: user1(a)example.com Date: 12/12/2014 Status: OK > > User: user2(a)example.com ID: 1234 Status: DEL ....... Date: 03/05/2014 > > Date: 05/08/2015 User: user3(a)example.com ....... Stauts: OK > > ......... > > > > I want split the log in more files arranged in different path builded by > the date information, for example /LOGS/YYYY/MM/DD/mylog.log. > > > > I do: > > rewrite r_rewrite_set { > > set("$(python get_data)", value("APP.DATE")); > > }; > > > > python{ > > import re > > def get_data(logmsg): > > out=None > > vars(logmsg) > > out = re.findall(" Date: (\d\d/\d\d/\d\d\d\d) ", logmsg.MESSAGE) > > if len(out) == 1: > > return out[0] > > else: > > raise Exception("Invalid match") > > }; > > > > In this way i have in APP.DATE the date. > > Now i have a some questions: > > 1) Is there another way to do this without python? > > Well, this should be possible with a simple regexp filter. I try to use a filter regexp, but i don't find the solution. Have you an example? > > > 2) In this way for every message, syslog-ng forks and exec a python > interpreter? > > No, it embeds a Python interpreter. > > > 3) Is there a way to add custom SDATA field from python? Or is there a > way to create APP.DATE from python without rewrite rule? > > Not right now. > > > 4) Is there a documentation about python{}? I only found a post in a blog. > > It's being prepared by the tech writer team in BalaBit > > > > > Thanks > > > > > > > ______________________________________________________________________________ > > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng > > Documentation: > http://www.balabit.com/support/documentation/?product=syslog-ng > > FAQ: http://www.balabit.com/wiki/syslog-ng-faq > > > > Inviato con AquaMail per Android http://www.aqua-mail.com

2 1

Problem with a pattern in 3.6.4
by C. L. Martinez 04 Sep '15

04 Sep '15

Hi all, I am trying to configure a pattern for the following log entry in syslog-ng 3.6.4: idpsnort01 09/03-13:18:41.935109 [**] [3:19187:6] PROTOCOL-DNS TMG Firewall Client long host entry exploit attempt [**] [Classification: Attempted User Privilege Gain] [Priority: 1] [AppID: dns] {UDP} 80.58.61.250:53 -> 10.196.0.67:60941 My pattern is: <pattern>@ESTRING:s3: @@ESTRING:: @@ESTRING:: [**]@ @QSTRING:s0:[]@ @ESTRING:s1: [**] [@Classification:@QSTRING:s2: ]@ [Priority : @NUMBER:i0:@] @@[AppID: @QSTRING:s4: ] @QSTRING:i1:{}@ @IPv4:i2:@:@NUMBER:i3:@ -> @IPv4:i4:@:@NUMBER:i5:@</pattern> If you try it, you can see it doesn't works. Problem is with the following part of the message: [Priority: 1] [AppID: dns] I need to escape "] [AppID:" and catch "dns" field, but I have tried some configs withut luck. Any idea?? Many thanks.

2 2

insider 2015-09: syslog-ng 3.7.1 released; ARM; configuring Java; Python trick;
by Czanik, Péter 03 Sep '15

03 Sep '15

Dear syslog-ng users, This is the 43th issue of the syslog-ng Insider, a monthly newsletter that brings you syslog-ng-related news. NEWS syslog-ng Open Source Edition 3.7.1 released -------------------------------------------- The first syslog-ng 3.7 release is now available together with a matching incubator 0.5 release. Major new features include Java and Python destinations migrated from the incubator, and a collection of Java based “big data” destinations: Elasticsearch, Hadoop and Kafka. For a complete list of new features, check the release notes available at https://github.com/balabit/syslog-ng/releases/tag/syslog-ng-3.7.1 syslog-ng on ARM ---------------------- As syslog-ng is in many Linux distributions, it is also available on ARM. Recently we had a chance to evaluate syslog-ng on a 64bit ARM machine. The results are mixed, but very promising. If you already use syslog-ng on ARM or plan to use it, contact me and share your use case and experiences! https://czanik.blogs.balabit.com/2015/07/syslog-ng-on-64bit-arm/ Getting started with the new Java based destinations ---------------------------------------------------- Installing the new Java based destination drivers requires a bit more preparation than installing the syslog-ng package and writing syslog-ng.conf, as was previously. This “HowTo” helps you to get started with Java based destinations on popular Linux distributions and FreeBSD: https://czanik.blogs.balabit.com/2015/08/getting-started-with-syslog-ng-3-7… A nice Python trick ------------------- syslog-ng Open Source Edition 3.7 allows you to process your log messages in Python, making it easy to solve a thousand use-cases. Here is one of the many possibilities: https://guest.blogs.balabit.com/2015/09/processing-log-messages-with-python… NEW RELEASES syslog-ng OSE 3.7.1: https://github.com/balabit/syslog-ng/releases/tag/syslog-ng-3.7.1 syslog-ng PE 5F4: https://jluby.blogs.balabit.com/2015/07/23/syslog-ng-premium-edition-5f4-re… syslog-ng Store Box 4F2: https://jluby.blogs.balabit.com/2015/08/10/syslog-ng-store-box-4f2-released/ Your feedback and news tips about the next issue is welcome at documentation(a)balabit.com To read this newsletter on-line, visit: http://insider.blogs.balabit.com/ Peter Czanik (CzP) <peter.czanik(a)balabit.com> BalaBit IT Security / syslog-ng upstream http://czanik.blogs.balabit.com/ https://twitter.com/PCzanik

1 0