April 2015 - syslog-ng - lists.balabit.hu

Create Pattern-DB rules
by Justin Kala 20 Oct '15

20 Oct '15

Hello I am trying to create Pattern-DB for the following Authorization messages coming from O/S. Can you help on creating a matching rule . Note: I do not have PATTERN-DB parser utility.I am going to create the db-parser.xml manually and put this rules inside the file. 2014-09-28T14:12:44-04:00 abcdef01-app/abcdef01-app sshd[11019]: [ID 800047 auth.notice] Failed password for root from 100.200.255.01 port 54438 ssh2 2014-09-28T14:03:46-04:00 abcdef01-app/abcdef01-app sshd[27420]: [ID 800047 auth.notice] Failed publickey for root from 100.200.255.02 port 59219 ssh2 2014-09-28T14:08:28-04:00 abcdef01-app/abcdef01-app sshd[3954]: [ID 800047 auth.notice] Failed keyboard-interactive for root from 100.200.255.03 port 65410 ssh2 2014-09-28T14:10:11-04:00 abcdef01-app/abcdef01-app sshd[5222]: [ID 293258 auth.error] libsldap: Status: 49 Mesg: openConnection: simple bind failed - Invalid credentials Thanks & Regards Kaladhar

4 16

Solaris & Linux (RH6) Console and SSH Use Logging
by wiskbroom＠hotmail.com 08 Jun '15

08 Jun '15

Greetings; This is question is slightly OT, but I can't imagine a better place to ask; so please, no flames. I would like to begin logging all attempts (succesful, failed, no password given, etc) to login to Sparc/Solaris machines, as well as RedHat 6 Linux boxes. My clients are using stock syslog, but my server is running syslog-NG; my second goal is to redirect all login type logs to just one file for ALL of my Solaris & Linux servers. Does anyone have a known good syslog config file for both Solaris, and Linux? Also, a good syslog-NG entry to force just the console and ssh data into a separate file? Thank you! .vp

2 2

threading question
by Jim Hendrick 21 May '15

21 May '15

Hi, Could someone help clarify something for me? I am running with threading enabled, and I have separate redis destinations (three currently) all pointing to the same server. My thought was that each destination would use a separate connection and thread. I then filter log sources and send (roughly) 1/3 to each destination. My question is this. Does each destination get a separate thread? (reading the documentation it left room for doubt). Also - more generally is this a reasonable way to get better throughput to a redis data store? (note: both syslog-ng and redis are running locally, so the connections use 'localhost') Thanks! Jim

4 3

Flag "no-multiline" not working on Syslog-ng
by Alan Sam 20 May '15

20 May '15

Hello community, I have the following diagram between some PE and Syslog-ng: Cisco devices -> Syslog-ng (running on Solaris) Syslog-ng version: o NTPSYSLOG# syslog-ng -V o syslog-ng 3.0.4 o Revision: ssh+git://bazsi@git.balabit //var/scm/git/syslog-ng/syslog-ng-ose--mainline--3.0#master#1b5d618e301ad94aa20e692ffba16469dece8d10 o Compile-Date: Sep 2 2009 06:15:53 o Enable-Threads: off o Enable-Debug: off o Enable-GProf: off o Enable-Memtrace: off o Enable-Sun-STREAMS: on o Enable-Sun-Door: on o Enable-IPv6: on o Enable-Spoof-Source: on o Enable-TCP-Wrapper: off o Enable-SSL: on o Enable-SQL: off o Enable-Linux-Caps: off o Enable-Pcre: on One of the cisco devices sends a particular log line that is splited in two lines (there is a line-break in between): *Mar 13 10:33:13: %BGP-3-INVALID_MPLS: Invalid MPLS label (1)* * received in update for prefix XXXX:XXXX:XXX.XXX.XXX.X/XXX from X.X.X.X* When the log reaches the Syslog-ng on Solaris server, it is logged like this: *Mar 13 10:33:14 PE06PVAL01 1182434: Mar 13 10:33:13: %BGP-3-INVALID_MPLS: Invalid MPLS label (1)* *Mar 13 10:33:14 PE06PVAL01 1182435: received in update for prefix XXXX:XXXX:XXX.XXX.XXX.X/XXX from X.X.X.X* The log is transfered by UDP from the cisco device to the Solaris server (where the syslog-ng runs). This is the configuration used in syslog-ng: *ntpsyslog> more /etc/syslog-ng/syslog-ng.conf* *@version: 3.0* *#* *# syslog-ng configuration file.* *#* *# See syslog-ng(8) and syslog-ng.conf(5) for more information.* *#* *options {* * stats_freq (0);* * flush_lines (0);* * time_reopen (10);* * log_fifo_size (1000);* * long_hostnames(off);* * use_dns (yes);* * use_fqdn (no);* * create_dirs (no);* * keep_hostname (yes);* * perm(0640);* *};* *source s_sys { sun-streams ("/dev/log" door("/etc/.syslog_door")); internal(); udp(flags("no-multi-line")); };* *destination d_cons { file("/dev/console"); };* *destination d_mesg { file("/var/adm/messages"); };* *destination d_mail { file("/var/log/syslog"); };* *destination d_auth { file("/var/log/authlog"); };* *destination d_mlop { usertty("operator"); };* *destination d_mlrt { usertty("root"); };* *destination d_mlal { usertty("*"); };* *destination cisco { file("/respaldo/syslog/cisco/cisco.log"); };* *#----------------------------------------------------------------------* *# Forward to a nisip server* *#* *destination cnc-cisco { udp("X.X.X.X" port(X)); };* *#----------------------------------------------------------------------* *filter f_filter1 { level(err) or* * (level(notice) and facility (auth, kern)); };* *filter f_filter2 { level(err) or* * (facility(kern) and level(notice)) or* * (facility(daemon) and level(notice)) or* * (facility(mail) and level(crit)); };* *filter f_filter3 { level(alert) or* * (facility(kern) and level(err)) or* * (facility(daemon) and level(err)); };* *filter f_filter4 { level(alert); };* *filter f_filter5 { level(emerg); };* *filter f_filter6 { facility(kern) and level(notice); };* *filter f_filter7 { facility(mail) and level(debug); };* *#filter f_filter10 { level(alert); };* *filter f_filter9 { facility(user) and level(alert); };* *filter f_cisco { facility(local2); };* *# Alternativa* *log { source(s_sys_cisco); filter(f_cisco); destination(cisco); };* *# Alternativa* *source s_juniper { file("/respaldo/syslog/juniper/juniper.log"); };* *destination d_juniper_tcp { tcp("X.X.X.X" port(X)); };* *filter f_juniper_tcp {not match("TOPO|/kernel:|snmpd|trace_*|PING_*|BGP_*|bgp_*|repeated|task|task_connect|EVENT|received iff message|rshd|cron" value("MESSAGE* *")); };* *log { source(s_juniper); filter(f_juniper_tcp); destination(d_juniper_tcp); };* *source s_cisco { file("/respaldo/syslog/cisco/cisco.log"); };* *destination d_cisco_tcp { tcp("X.X.X.X" port(X)); };* *log { source(s_cisco); destination(d_cisco_tcp); };* *##################################* *# FWD from Syslog to CNC Cisco* *##################################* *source s_cisco { file("/respaldo/syslog/cisco/cisco.log"); };* *log {source(s_cisco); destination(cnc-cisco); };* *ntpsyslog>* I have tried different configurations in order to make the "no-multi-line" flag work. However, none of them have worked: destination cisco { file("/respaldo/syslog/cisco/cisco.log" flags(no-multi-line)); }; source s_sys { sun-streams ("/dev/log" door("/etc/.syslog_door")); internal(); udp(flags("no-multi-line")); }; If more information is required, please do not hesitate to ask for it. Thank you beforehand for your help. Alan Sam

5 10

insider 2015-04: 3.7 beta1; RedHat Summit; Hadoop; GSoC; RFC5424;
by Czanik, Péter 30 Apr '15

30 Apr '15

Dear syslog-ng users, This is the 41th issue of the syslog-ng Insider, a monthly newsletter that brings you syslog-ng-related news. NEWS syslog-ng 3.7 beta1 released ---------------------------- In the first beta of syslog-ng OSE 3.7, the Java and Python destinations were migrated from the syslog-ng incubator to the syslog-ng core. Also, syslog-ng now includes an interactive syslog-ng.conf debugger. Read about known problems and how to get started at https://czanik.blogs.balabit.com/2015/04/first-beta-of-syslog-ng-3-7-releas… syslog-ng at RedHat Summit -------------------------- BalaBit is a general sponsor of the Red Hat Summit. The event will be in Boston, June 23-26. http://www.redhat.com/summit/ Visit the BalaBit booth to talk about syslog-ng and log management! syslog-ng Hadoop support ------------------------ With the release of syslog-ng PE 5F3, support for Hadoop arrived. It will also be part of the upcoming syslog-ng OSE 3.7 release. Read more about how syslog-ng can send logs to a HDFS destination: https://jluby.blogs.balabit.com/2015/04/08/sending-logs-to-hadoop-using-sys… With this solution BalaBit became a certified MapR technology partner: https://www.mapr.com/partners/partner/balabit-it-security syslog-ng in Google Summer of Code 2015 --------------------------------------- This year we participate in GSoC again. The syslog-ng team received many good proposals, it will be a tough choice to decide on the finalists. We plan to have an IRC meeting with the participation of the winning GSoC students and members of the syslog-ng community. A separate e-mail will announce the exact time. The project and idea list is available at https://github.com/balabit/syslog-ng/wiki/GSoC2015-idea-&-project-list Forwarding name value pairs using RFC5424 ----------------------------------------- Recently many users asked how to forward the name of a file source over the network. It can easily be done using the RFC5424 syslog protocol. The same method can also be used to create and forward name value pairs between servers in general. Read more about it at https://czanik.blogs.balabit.com/2015/03/using-rfc5424-syslog-to-forward-fi… syslog-ng @ LOADays 2015 ------------------------ This year syslog-ng was presented again at LOADays, a small but very exciting Linux admin conference in Antwerp. The basics of syslog-ng, the importance of name value pairs and the latest developments in syslog-ng were presented. For details, check: https://czanik.blogs.balabit.com/2015/04/loadays-2015-logging-elasticsearch… NEW RELEASES syslog-ng 3.7.0beta1: https://github.com/balabit/syslog-ng/releases/tag/syslog-ng-3.7.0beta1 Your feedback and news tips about the next issue is welcome at documentation(a)balabit.com To read this newsletter on-line, visit: http://insider.blogs.balabit.com/ Peter Czanik (CzP) <peter.czanik(a)balabit.com> BalaBit IT Security / syslog-ng upstream http://czanik.blogs.balabit.com/ https://twitter.com/PCzanik

1 0

syslog-ng memory leak ? valgrind attached
by Jim Hendrick 29 Apr '15

29 Apr '15

From 3.6.2 OSE built with json & redis Seeing steady memory growth (> 10 GB after ~ a day at ~7000 EPS load) This was from a few minutes of runtime. (~5) Is this a known issue or did I trip over something? Thanks! Jim

5 4

Re: [syslog-ng] syslog-ng memory leak ? valgrind attached
by Jim Hendrick 29 Apr '15

29 Apr '15

I'll try this tomorrow. Took today off to get some work done on the house. Thanks,Jim Sent from my Verizon Wireless 4G LTE smartphone -------- Original message -------- From: "Czanik, Péter" <peter.czanik(a)balabit.com> Date: 04/29/2015 9:57 AM (GMT-05:00) To: Syslog-ng users' and developers' mailing list <syslog-ng(a)lists.balabit.hu> Subject: Re: [syslog-ng] syslog-ng memory leak ? valgrind attached For the redis destination it's a know problem, was quickly fixed after 3.6.2 release: https://github.com/balabit/syslog-ng/commit/53606941b1b6b6726c4945f3560984a… If you use my unofficial syslog-ng packages, let me know, and I can add this patch. Bye, Peter Czanik (CzP) <peter.czanik(a)balabit.com> BalaBit IT Security / syslog-ng upstream http://czanik.blogs.balabit.com/ https://twitter.com/PCzanik On Wed, Apr 29, 2015 at 3:49 PM, Jim Hendrick <jrhendri(a)roadrunner.com> wrote: From 3.6.2 OSE built with json & redis Seeing steady memory growth (> 10 GB after ~ a day at ~7000 EPS load) This was from a few minutes of runtime. (~5) Is this a known issue or did I trip over something? Thanks! Jim ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq

1 0

Re: [syslog-ng] syslog-ng memory leak ? valgrind attached
by Jim Hendrick 29 Apr '15

29 Apr '15

Queues are often full. (separate issue ). Sent from my Verizon Wireless 4G LTE smartphone -------- Original message -------- From: Fabien Wernli <wernli(a)in2p3.fr> Date: 04/29/2015 9:55 AM (GMT-05:00) To: syslog-ng(a)lists.balabit.hu Subject: Re: [syslog-ng] syslog-ng memory leak ? valgrind attached Hi Jim, On Wed, Apr 29, 2015 at 09:49:05AM -0400, Jim Hendrick wrote: > From 3.6.2 OSE built with json & redis > > Seeing steady memory growth (> 10 GB after ~ a day at ~7000 EPS load) > > This was from a few minutes of runtime. (~5) > > Is this a known issue or did I trip over something? We're having similar issues when the workload tickles your figures. One possible explanation seems to be that the memory from queues is never released [1]. Are your queues always empty or do they do regular work? Oh, then there was this one too [2] [1] https://github.com/balabit/syslog-ng/issues/287 [2] https://github.com/balabit/syslog-ng/issues/446 ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq

1 0

syslog-ng.conf configuration for log files every 15 minutes
by Akshay Surve 28 Apr '15

28 Apr '15

Hi, Currently, I have used the syslog-ng.conf to setup hourly files using the following config - destination d_clicks {file("/var/log/local-pid$PID-click-$YEAR-$MONTH-$DAY-$HOUR.log");}; Is there a way I could get files every 15 minutes? Best, Akshay

2 1

my fedora/epel packages update to use riemann-c-client 1.6.1
by Czanik, Péter 27 Apr '15

27 Apr '15

Hi, I just finished updating my fedora/epel packages on copr to riemann-c-client 1.6.1. This version brings gnutls support to riemann-c-client. As personally I don't have access to a riemann server, please test it and let me know if it works for you (I'm also interested in success stories, not just failures :-) ). https://copr.fedoraproject.org/coprs/czanik/syslog-ng36epel6/ https://copr.fedoraproject.org/coprs/czanik/syslog-ng36/ https://copr.fedoraproject.org/coprs/czanik/syslog-ng37/ Bye, Peter Czanik (CzP) <peter.czanik(a)balabit.com> BalaBit IT Security / syslog-ng upstream http://czanik.blogs.balabit.com/ https://twitter.com/PCzanik

1 0