- syslog-ng - lists.balabit.hu

Apache Kafka output module
by Pierre-Yves Ritschard 11 Jul '14

11 Jul '14

Hi list, I submitted a PR for an Apache Kafka [1] output module. This is my first syslog-ng module, so it might need a bit of guidance. This provides the ability to create a partitioned stream of logs based on a message field or static key. A typical destination configuration will look like: destination d_kafka { kafka( properties(metadata.broker.list("localhost:9092")) topic("syslog") payload("$(format-json --scope all-nv-pairs --scope core)") partition("static-key") # partition(random) # partition(field("$PROGRAM")) ); }; For reference, this allows us to work on logs locally with syslog, extract fields, then publish on a queue whose consumers will index logs in our data stores. [1]. http://kafka.apache.org

2 1

patterndb and context - access fields from initial message
by Atom2 11 Jul '14

11 Jul '14

Hi guys, I did have a conversation today on IRC with algernon (again many thanks to him) about the problem I am faced with and we agreed that I should send my question to the list. My aim is to consolidate a bunch of related syslog messages into a single message once the final message has come in (or a timeout has occured for whatever reason). To do this I use patterndb together with a defined context linking related messages to the context. Unfortunately the length of the context (i.e. the number of messages within the context) is unknown from the outset. Basically it (mainly) depends on how long the context is active for - which can be any time and is completely unforseeable. Now my problem is as follows: From the initial message I parse (using patterndb) certain fields into a number of named MACROS. This is all solved and works very well. When the final message comes in (or in case a timeout occurs) I would like to trigger an action (triggered by match or timeout respectively) to generate and subsequently inject a message about the full context using certain fields, mainly from the final message, but also including a number of fields from the inital message of the context. Generally a field is available using the syntax ${MACRO}@# where @# denotes a reference to a previous message within the same context - so ${MACRO}@5 would use the value of MACRO from a message with a distance of 5 from the curren message (where the distance of the current message to itself is 1 - so @5 references the 4th previous message). That also works very well but obviously requires that one knows which message to refer to using the distance value. The problem comes when the context-length is unknown and thus the distance of the initial message to the final message is not a fixed, pre-determined number but rather varies. Although there seems to be a variable/macro indicating/storing the context-length which (according to the manual) can be accessed by $(context-length) [NOTE: the paranthesis instead of the curly braces] the combination of the two does not seem to work out: I was expecting that ${MACRO}@$(context-length) would provide me with the required information - that is: the value of the parsed field from the initial message - but my construct unfortunately fails. Is there anything I am doing wrong here (i.e. is there any quoting/esacping required for certain characters), is my thought process flawed or does this not work/is it a bug. In case it does not work, how would I be able to achieve my goal. Many thanks and best regards, Atom2 P.S. The version of syslog-ng I am using is 3.4.7, compiled from sources, using the latest stable gentoo ebuild.

4 6

Syslog-ng completely unusable after upgrade
by James Lay 10 Jul '14

10 Jul '14

Well shoot...here's what I got...I'm using the stock syslog-ng.conf installed with 3.5.4.1 just for testing: ############################################################################# # Default syslog-ng.conf file which collects all local logs into a # single file called /var/log/messages. # @version: 3.3 @include "scl.conf" source s_local { system(); internal(); }; source s_network { udp(); }; destination d_local { file("/var/log/messages"); }; log { source(s_local); # uncomment this line to open port 514 to receive messages #source(s_network); destination(d_local); }; Running strace here's the last bit: /usr/sbin/syslog-ng -t -d -v -F -f /usr/etc/syslog-ng.conf open("/usr/lib/libsyslog-ng-3.3.11.so", O_RDONLY) = 10 read(10, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0@0\1\000"..., 512) = 512 fstat64(10, {st_mode=S_IFREG|0755, st_size=1427313, ...}) = 0 mmap2(NULL, 447784, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 10, 0) = 0xb79d7000 mmap2(0xb7a40000, 16384, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 10, 0x68) = 0xb7a40000 mmap2(0xb7a44000, 1320, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0xb7a44000 close(10) = 0 write(2, "iv_tls_user_register: called aft"..., 43iv_tls_user_register: called after iv_init ) = 43 rt_sigprocmask(SIG_UNBLOCK, [ABRT], NULL, 8) = 0 tgkill(5172, 5172, SIGABRT) = 0 --- SIGABRT (Aborted) @ 0 (0) --- +++ killed by SIGABRT +++ Process 5172 detached And that's it....I'm glad I'm doing this on a virtual machine. I've not been able to anything on this online...thoughts or help are appreciated. James

2 2

Configure option --disable-json doesn't seem to function
by James Lay 10 Jul '14

10 Jul '14

Topic says it: syslog-ng-3.5.4.1 ./configure --prefix=/usr --enable-dynamic-linking --disable-ipv6 --enable-spoof-source --disable-json checking python module: json... no configure: error: failed to find required module json configure: error: ./configure.gnu failed for modules/afamqp/rabbitmq-c It appears to still be looking. Thank you. James

2 9

syslog problem
by Riyas Ahamed 09 Jul '14

09 Jul '14

Hi, I have configured syslog-ng but I cannot able to see logs of network devices in syslog-ng server. Please help me to sort-out the problem. Regards, N.B.Riaz Ahmed http://www.csscorp.com/common/email-disclaimer.php

4 4

Recommended version
by James Lay 08 Jul '14

08 Jul '14

All, As I'm working with it here, what's the recommended version folks are running? Currently I'm on 3.3.11, and I figure I might as well upgrade..thanks for any pointers/tips. James

2 3

[Bug 280] New: Memory leak in value-pairs.c
by bugzilla＠bugzilla.balabit.com 08 Jul '14

08 Jul '14

https://bugzilla.balabit.com/show_bug.cgi?id=280 Summary: Memory leak in value-pairs.c Product: syslog-ng Version: 3.5.x Platform: PC OS/Version: Linux Status: NEW Severity: normal Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: sgounaro(a)optonline.net Type of the Report: bug Estimated Hours: 0.0 Hi , it looks like vp_walker_name_value_split_add_name_token has a memory leak in g_strndup call. Running syslog-ng with a lot of messages coming in about ~800 per sec , starts with 139m of memory and ends up with 9gig of memory in a day being used. -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 5

Properly forwarding syslog messages
by James Lay 07 Jul '14

07 Jul '14

Hey all, So I've looked around and I'm just not sure what the best method is for forwarding syslog messages. I have a current setup that is working well. I'd like to try and get all the messages forward to another machine using standard udp 514. I'd like the messages to appear as if they are coming from the originating machine, not the syslog server. I did see a couple sites that say I have to compile with spoofing. Can anyone point me to some resources to do this? Thank you. James

2 3

Optional Parser Type Match
by Michael Starks 07 Jul '14

07 Jul '14

Can a parser type be made to optionally match? For example, given the message of 'DROP IN=vlan2 OUT= MAC=48:5b:39:e8:44:c5:00:1d:5a:1c:37:b9:08:00:45:00:00:28 SRC=1.2.3.4 DST=172.16.0.1 LEN=40 TOS=0x00 PREC=0x00 TTL=45 ID=61318 PROTO=TCP SPT=443 DPT=45872 SEQ=1548679084 ACK=0 WINDOW=0 RES=0x00 RST URGP' OUT= may or may not be defined. I would like to match it if it exists with a parser and maybe just set it to nul if it doesn't exist. Similarly, in this message: 'DROP IN=vlan2 OUT= MAC=48:5b:39:e8:44:c5:00:1d:5a:1c:37:b9:08:00:45:00:00:28 SRC=1.2.3.4 DST=172.16.0.2 LEN=40 TOS=0x00 PREC=0x00 TTL=54 ID=3265 DF PROTO=TCP SPT=443 DPT=44616 SEQ=880418731 ACK=0 WINDOW=0 RES=0x00 RST UR' the DF field sometimes exists and sometimes doesn't. Whenever I define something like 'DROP IN=@ESTRING:s1: @OUT=@ESTRING:s0: @' the OUT= string doesn't match if there is no value.

2 2

No PatternDB Matches
by Michael Starks 02 Jul '14

02 Jul '14

x-posted from ELSA mailing list; no replies there so please excuse the noise. I am trying to add an additional ASA log message that is unclassified in ELSA. To make a long story short, *nothing" seems to be matching with pdbtool. Here is the line I am trying to work with: Jun 23 00:00:05 192.168.1.4 %ASA-3-106100: access-list INOUT denied tcp inside/1.2.3.4(64576) -> outside/5.6.7.8(80) hit-cnt 1 first hit [0x59bca63e, 0x620e5b55] Putting that aside for the moment since I can't get it to work, I went straight to the syslog-ng docs (https://www.balabit.com/sites/default/files/documents/syslog-ng-ose-3.4-gui…) and used the example. I created /etc/elsa/patterns.d/test.xml with this content: <patterndb version='4' pub_date='2010-10-17'> <ruleset name='ssh' id='123456678'> <pattern>ssh</pattern> <rules> <rule provider='me' id='182437592347598' class='system'> <patterns> <pattern>Accepted @QSTRING:SSH.AUTH_METHOD: @ for@QSTRING:SSH_USERNAME: @from\ @QSTRING:SSH_CLIENT_ADDRESS: @port @NUMBER:SSH_PORT_NUMBER:@ ssh2</pattern> </patterns> </rule> </rules> </ruleset> </patterndb> I then tested with this line: /usr/local/syslog-ng/bin/pdbtool match -p /etc/elsa/patterns.d/test.xml -M "Accepted password for sampleuser from 10.50.0.247 port 42156 ssh2" The result is this: MESSAGE=Accepted password for sampleuser from 10.50.0.247 port 42156 ssh2 .classifier.class=unknown Details are: [root@hostname elsa]# /usr/local/syslog-ng-3.2.4/sbin/syslog-ng -V syslog-ng 3.2.4 Installer-Version: 3.2.4 Revision: ssh+git://bazsi@git.balabit//var/scm/git/syslog-ng/syslog-ng-ose--mainline--3.2#master#ef7b91e4a1b1f9628c66138b4ae83de7e4c697c6 Compile-Date: May 23 2012 09:58:14 Enable-Threads: off Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-Sun-STREAMS: off Enable-IPv6: on Enable-Spoof-Source: off Enable-TCP-Wrapper: off Enable-SSL: on Enable-SQL: off Enable-Linux-Caps: on Enable-Pcre: on Enable-Pacct: off [root@hostname elsa]# cat /etc/redhat-release Oracle Linux Server release 6.5 Thank you in advance.

2 3