- syslog-ng - lists.balabit.hu

How to use parser
by 大町靖 25 Feb '12

25 Feb '12

Hello Robert, Thanks for your effort. I rewrite the configurtion below. parser p1{ csv-parser(columns("Msg1","Msg2") delimiters("#AA#") # template ("$Msg1\n$Msg2"); }; destination d_mesge { file("/var/log/messages" template("$MSG1\nMSG2\n")); }; log { source(s_sys); parser(p1); destination(d_mesg); }; But the result isn't unexpected. I want to divide by "#AA#". ("#AA#" is a string.) It seems that the result is divided by the '#' or 'A' . (character). For example, the messages is "12A456 #AA# 789". The result is MSG1 = 12 MSG2=456. I expect that MSG1=12A456 MSG2=789. What should I do ? Thanks, Yasu === Hi, remove the template() option from the parser definition and add it to the definition of the destination. Regards, Robert On 02/07/2012 03:50 PM, 大町靖 wrote: > Hello Experts, > > Please let me know how to use parser setting. > > This is the syslog message. > Msg1 #AA# Msg2 > > I want to divide this message. > I expect the below. > Msg1 > Msg2 > > But It doesn't work well , though I wrote the configuration below. > parser p1{ > csv-parser(columns("Msg1","Msg2") > delimiters("#AA#") > template ("$Msg1\n$Msg2"); > }; > log { source(s_sys); parser(p1); destination(d_mesg); }; > > I check the output. The result wasn't unexpected. > Msg1 #AA# Msg2 > > If my configuration is wrong, please let me know. > > Thanks, > Yasu > > ______________________________________________________________________________ > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng > Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng > FAQ: http://www.balabit.com/wiki/syslog-ng-faq > >

2 1

Re: [syslog-ng] Installing syslog-ng 3.3.4 OSS on multiple Centos servers
by Srinivasan Subramanian 25 Feb '12

25 Feb '12

What about Mageia ? I see the latest binaries and source RPMS in the Mageia repository? Looks like the latest will always be available there. Srini > Date: Fri, 24 Feb 2012 14:22:46 +0100 > From: Gergely Nagy <algernon(a)balabit.hu> > Subject: Re: [syslog-ng] Installing syslog-ng 3.3.4 OSS on multiple > Centos servers > To: syslog-ng(a)lists.balabit.hu > Message-ID: <87pqd49xi1.fsf(a)luthien.mhp> > Content-Type: text/plain > > Srinivasan Subramanian <ssrini_vasan(a)hotmail.com> writes: > > > I have a need to install syslog-ng 3.3.4 on multiple Centos servers. > > Unfortunately i am unable to find RPMs for the newer versions and only > > find the source downloads. > > I don't know of any Centos RPMs, but you might try the sRPMs from > SuSE[1]. At worst, they'll be a reasonable starting point. > > [1]: http://software.opensuse.org/search/download?base=openSUSE%3AFactory&file=B… > > --

1 0

3.3.4 anyone else notice a memory leak
by Evan Rempel 25 Feb '12

25 Feb '12

I run a number of syslog-ng processes, and I am noticing that the processes that read from a pipe, and write to a program destination all seem to have a memory leak in them. The rate of leak seems to be dependent on the volume of log lines (not surprising). Has anyone else noticed this. My config is; @version: 3.3 # # UVic syslog-ng configuration options { log_fifo_size(100000); use_fqdn(yes); keep_hostname(yes); chain_hostnames(yes); time_reap(60); time_reopen(5); flush_lines(1000); flush_timeout(1000); }; source local { pipe("/var/log/syslog.pipes/syslogstats" log_iw_size(90000) log_fetch_limit(500) flags(no-parse)); }; source int { internal(); }; destination localsyslog { tcp("localhost" port(1514) localip(localhost) log_fifo_size(50000) template("<$PRI>$S_DATE syslogstats@$HOST syslogstats: $MSGONLY\n") template_escape(no) ); }; destination syslog_stats { program("/home/sysprog/syslog-apps/syslog_stats/syslogstats" template("$MESSAGE\n") template_escape(no) ); }; log { source(local); destination(syslog_stats); flags(flow-control); }; log { source(int); destination(localsyslog); };

6 12

syslog-ng Insider - February 2012
by Peter Czanik 23 Feb '12

23 Feb '12

Dear syslog-ng users, This is the 11th issue of the syslog-ng Insider, a monthly newsletter that brings you syslog-ng related news. Your feedback and news tips about the next issue is welcome at documentation(a)balabit.com <mailto:documentation@balabit.com> FEATURED NEWS Brno: Fedora, CEE, journal and syslog-ng ---------------------------------------- Last week the Brno Red Hat office hosted two conferences, one small about logging and the Fedora Developer conference. The logging miniconf covered some very hot topics: CEE, journal, auditd and some lesser known projects, like ELAPI. After the formal program, we had some very good discussions about the future of logging. You can check the diagram drawn up as conclusion here: https://czanik.blogs.balabit.com/2012/02/brno-fedora-cee-journal-and-syslog… And read more about how syslog-ng supports CEE: http://algernon.blogs.balabit.com/2012/02/cee-handling-with-syslog-ng/ BalaBit has just released the latest version of its leading log management tool, syslog-ng 4 F2 ------------------------------- Adding to the existing, rich feature set which includes high-performance multi-thread processing, encrypted and timestamped log files, disk-based buffering, direct database access, native TLS support, the syslog-ng 4 F2 now supports Application-levelAcknowledgement via Reliable Log Transport Protocol (RLTP)^TM , a new transport protocol that prevents message loss during connection breaks. In addition, the latest version of syslog-ng can now natively collect and process log messages from SQL databases enabling users to easily manage log messages from a wide variety of enterprise software and custom applications. syslog-ng 3.3.4 is released --------------------------- It is a bugfix release, which fixes all previously known problems in the 3.3 series. There is only one change in 3.3 sources since the last release: manual pages were put under the GPL, and XML sources are now also available, so that the entire source code of syslog-ng is free from this point onwards. Sources are available at http://www.balabit.com/network- <http://www.balabit.com/network-security/syslog-ng/opensource-logging-system…>security/syslog-ng/opensource-logging-system/downloads/download <http://www.balabit.com/network-security/syslog-ng/opensource-logging-system…> Packages for some distributions are available from http://www.balabit.com/network-security/syslog-ng/opensource-logging-system… Detailed changelog is available at http://www.balabit.com/files/syslog-ng/open-source-edition/3.3.4/changelog-… Documentation was also updated: http://www.balabit.com/support/documentation/documents/syslog-ng-ose-3.3-gu… EU Data Protection Directive – How a single regulation could boost the transparency in IT security? -------------------------------- Personal opinion from Balázs Scheidler, CEO of BalaBit Overall, the *EU Data Protection Directive can be a milestone in boosting the transparency of IT security at* organizations – similarly to the regulatory compliance regulations after the Enron case. If adopted, the new directive could bring about a change in *the implementation of IT security policies so that the current focus on audits could shift to the deeper integration of IT security processes into business processes. *As logging and log management are the base of every monitoring method, *technologies with high-speed and zero message loss capabilities, like syslog-ng, will come to the front.* Encrypting log files, in which companies store user names, passwords and other sensitive company data is also key to prevent data loss. http://bscheidler.blogs.balabit.com/2012/02/eu-data-protection-directive-ho… OTHER SHORT NEWS * What makes the upcoming v3.4 even more flexible? It's summarized by Bazsi, lead developer of syslog-ng at http://bazsi.blogs.balabit.com/2012/01/syslog-ng-flexibility-improvements/ Watch out for an upcoming Alpha version! * There are still many syslog-ng v2.0 users. For those, here is a teaser, what changed since that release: http://czanik.blogs.balabit.com/2012/02/whats-new-in-syslog-ng-3-x/ * A syslog-ng presentation for FOSDEM: http://czanik.blogs.balabit.com/2012/02/fosdem-syslog-ng-as-upstream/ * ELSA, the high performance web GUI for syslog-ng, received a very useful new feature, post processing of search results: http://ossectools.blogspot.com/2012/01/hunting-with-elsa-transforms.html NEW RELEASES: * syslog-ng OSE 3.3.4: https://lists.balabit.hu/pipermail/syslog-ng-announce/2012-January/000133.h… * syslog-ng PE 4F2 ARCHIVE http://insider.blogs.balabit.com/ -- Peter Czanik (CzP)<czanik(a)balabit.hu> BalaBit IT Security / syslog-ng upstream http://czanik.blogs.balabit.com/

1 0

Extract portion of SDATA file name?
by Andika Daud 21 Feb '12

21 Feb '12

Hi, I'm a noobie and just about beginning to try out syslog-ng at my company. I have the following configuration on the client: source s_apache { file("/apps/log/apache/store-dev03/*" program_override("apache") flags(no-parse) ); }; destination d_remote { syslog("10.5.76.125" transport("tcp") port(514) # template("${MSGONLY}") ); }; log { source(s_apache); destination(d_remote); }; And something like this on the server side: source s_network { syslog(ip(0.0.0.0) port(514) transport("tcp") ); }; destination d_local { file("/apps/log/syslong-ng/$PROGRAM/$HOST/$YEAR$MONTH$DAY.log" # ignore the use of template below, I just wanted to prove the server side could # see the value of the filepath from the client template("${.SDATA.file(a)18372.4.name} - ${MSGONLY}\n") ); }; log { source(s_network); destination(d_local); }; My ("${.SDATA.file(a)18372.4.name} value can be something like this: '/apps/log/apache/store-dev03/access.log'. What I want to be able to do is to parse that path. That is to discard the begining of the path, /apps/log/apache. Get the rest of the value, especially the file name (access.log) to be used to construct destination path. Is this possible? Thank you for your kind help.

2 2

[PATCH (3.3)] doc: Fix EXTRA_DIST
by Gergely Nagy 21 Feb '12

21 Feb '12

There is a \ missing from the end of the second line of EXTRA_DIST, which results in a couple of files being left out of make dist, which in turn makes distcheck and building from dist tarballs fail. This patch adds the missing \. Signed-off-by: Gergely Nagy <algernon(a)balabit.hu> --- doc/Makefile.am | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/doc/Makefile.am b/doc/Makefile.am index 33b113c..6c6c1c2 100644 --- a/doc/Makefile.am +++ b/doc/Makefile.am @@ -1,7 +1,7 @@ man_MANS = man/syslog-ng.8 man/syslog-ng.conf.5 man/pdbtool.1 man/loggen.1 man/syslog-ng-ctl.1 EXTRA_DIST = man/syslog-ng.8.xml man/syslog-ng.conf.5.xml man/pdbtool.1.xml man/loggen.1.xml man/syslog-ng-ctl.1.xml \ - $(man_MANS) + $(man_MANS) \ security/bof-2002-09-27.txt \ security/dos-2000-11-22.txt \ xsd/patterndb-1.xsd \ -- 1.7.9

1 0

[Bug 152] New: Vers 3.3.3 pipe source does not parse host from input line
by bugzilla＠bugzilla.balabit.com 14 Feb '12

14 Feb '12

https://bugzilla.balabit.com/show_bug.cgi?id=152 Summary: Vers 3.3.3 pipe source does not parse host from input line Product: syslog-ng Version: 3.3.x Platform: PC OS/Version: Linux Status: NEW Severity: normal Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: erempel(a)uvic.ca Type of the Report: --- Estimated Hours: 0.0 Try as I might, I can not get a pipe source to maintain the hostname. Syslog-ng includes the hostname as the $MSGHDR (or the $PROGRAM). my config # ----------------------------------- source patterndb { pipe("/var/log/syslog.pipes/patterndb" log_fetch_limit(500) log_iw_size(100000) ); }; source int { internal(); }; template t_standardfile { template("$S_ISODATE $FULLHOST $FACILITY.$LEVEL $MSGHDR][$PROGRAM][$MESSAGE\n"); template_escape(no); }; destination d_var_patterndb { file("/var/log/patterndb.$R_YEAR$R_MONTH$R_DAY.000000" owner("root") group("syslogs") perm(0640) template(t_standardfile)); }; log { source(patterndb); destination(d_var_patterndb); }; # ----------------------------------- data I write tothe pipe # ----------------------------------- <12>2011-11-25T00:00:30-08:00 somehost.uvic.ca mmfs: Fri Nov 25 00:00:29.618 2011: Accepted and connected to 172.20.102.38 hermes0080 <c0n350> <12>2011-11-25T00:00:30-08:00 somehost.uvic.ca mmfs: Fri Nov 25 00:00:29.620 2011: Connecting to 172.20.107.23 nestor0167 <c0n200> <12>2011-11-25T00:00:30-08:00 somehost.uvic.ca mmfs: Fri Nov 25 00:00:29.621 2011: Connected to 172.20.107.23 nestor0167 <c0n200> # ----------------------------------- The output file # ----------------------------------- 2011-11-25T00:00:30-08:00 patterndb(a)catamount.comp.uvic.ca user.warning somehost.uvic.ca ][somehost.uvic.ca][mmfs: Fri Nov 25 00:00:29.618 2011: Accepted and connected to 172.20.102.38 hermes0080 <c0n350> 2011-11-25T00:00:30-08:00 patterndb(a)catamount.comp.uvic.ca user.warning somehost.uvic.ca ][somehost.uvic.ca][mmfs: Fri Nov 25 00:00:29.620 2011: Connecting to 172.20.107.23 nestor0167 <c0n200> 2011-11-25T00:00:30-08:00 patterndb(a)catamount.comp.uvic.ca user.warning somehost.uvic.ca ][somehost.uvic.ca][mmfs: Fri Nov 25 00:00:29.621 2011: Connected to 172.20.107.23 nestor0167 <c0n200> # ----------------------------------- I would expect the output file to read # ----------------------------------- 2011-11-25T00:00:30-08:00 somehost.uvic.ca user.warning mmfs: ][mmfs][Fri Nov 25 00:00:29.618 2011: Accepted and connected to 172.20.102.38 hermes0080 <c0n350> 2011-11-25T00:00:30-08:00 somehost.uvic.ca user.warning mmfs: ][mmfs][Fri Nov 25 00:00:29.620 2011: Connecting to 172.20.107.23 nestor0167 <c0n200> 2011-11-25T00:00:30-08:00 somehost.uvic.ca user.warning mmfs: ][mmfs][Fri Nov 25 00:00:29.621 2011: Connected to 172.20.107.23 nestor0167 <c0n200> # ----------------------------------- -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 2

Network compression
by Rory Toma 14 Feb '12

14 Feb '12

I notice that network compression is listed as a feature for syslog-ng. I can't find any documentation on how to enable this anywhere, however. How do I tell a syslog-ng receiver to accept compressed traffic?

2 3

Documentation questions
by Chris Johnson 13 Feb '12

13 Feb '12

I was reading through the documentation, and noticed that in some examples the log statements are named: log demo_filteredlog { source(s1); filter(demo_filter); destination(d1);}; When I tried using this method in my config file, syslog-ng fails to start. Is this method supported or is this just an oversight with the documentation? Also, the links to the sample pattern database (http://www.balabit.com/downloads/files/patterndb-snapshot/?utm_source=syslo…) apprears to be incorrect. Is this the same pattern database at /downloads/files?path=/patterndb-snapshot? or is there a more recent one available to download? Thanks Chris

2 1

Cant get pattern matching to work
by Chris Johnson 12 Feb '12

12 Feb '12

Sorry for bombarding the mailing list with questions. I have been working through the documentation but just seem to keep hitting obstacles. I have compared this to the examples in the manual and some samples I was able to track down, but Im not seeing where I am going wrong. For this question, I am working with the following data: "MESSAGE" : "RT_FLOW_SESSION_CLOSE: session closed unset: 192.168.199.253/55189->8.8.8.8/53 junos-dns-udp X.X.X.X/60836->8.8.8.8/53 source-nat-rule None 17 trust-to-untrust trust untrust 30455 1(83) 1(83) 2 N/A N/A N/A(N/A) vlan.0", "PROGRAM" : "RT_FLOW" Based on this , I have generated the following pattern database and config just to see if I am heading down the right path. Unfortunately, what I thought would generate a pattern match on the logs and generate the appropriate macros, are not retuning anything. I have also tried taking out the $PROGRAM patter, but that did not seem to help. I have also validated this with pdbtool and it succeeds. Any help is greatly appreciated Chris patterndb.xml <patterndb version='3' pub_date='2011-02-11'> <ruleset name='session_close' id='123456678'> <rules> <rule provider='cj' id='182437592347598' class='session'> <patterns> <pattern> @ESTRING:TEST1:::@ @ANYSTRING:TEST2@ </pattern> </patterns> </rule> </rules> </ruleset> </patterndb> syslog-ng.conf #####Destinations##### destination d_mongodb { mongodb( value-pairs( key("TEST1") key("TEST2") scope("base") ) ); }; #####Parser##### parser pattern_db { db_parser( file("/usr/local/etc/patterndb.xml") ); }; #####Log##### log { source(s_network); parser(pattern_db); destination(d_mongodb); }; Unfortunately, what I thought would pattern match and generate a macro for

2 6