- syslog-ng - lists.balabit.hu

stale submodules for ivykis and libmongo-client in 3.5.x
by Nix 04 Apr '14

04 Apr '14

spindle 1469 /usr/src/syslog-ng/syslog-ng% git submodule update fatal: reference is not a tree: ec9066dcc66b33e3767b4729f86844549f004657 fatal: reference is not a tree: cf44084c01a4228244f0dc079c43f94f67462537 Unable to checkout 'ec9066dcc66b33e3767b4729f86844549f004657' in submodule path 'lib/ivykis' Unable to checkout 'cf44084c01a4228244f0dc079c43f94f67462537' in submodule path 'modules/afmongodb/libmongo-client' That this has been true for a month makes me wonder how many people are actually building syslog-ng from git... -- NULL && (void)

2 2

syslog-ng: lua, monitoring
by Peter Czanik 04 Apr '14

04 Apr '14

Hi, As not everyone is subscribed to the BalaBit blog rss feed, here are a couple of interesting blogs how to use syslog-ng in even more situations: Extending syslog-ng with lua scripts as destination: https://talien.blogs.balabit.com/2014/03/lua-destination-examples/ Monitoring with syslog-ng: https://talien.blogs.balabit.com/2014/03/monitoring-with-syslog-ng/ Bye, -- Peter Czanik (CzP) <czanik(a)balabit.hu> BalaBit IT Security / syslog-ng upstream http://czanik.blogs.balabit.com/ https://twitter.com/PCzanik

2 1

SQL destination - multiple destinations (tables) within one database ?
by Tomáš Novosad 03 Apr '14

03 Apr '14

Hello, i'd like to ask, if it si possible, somehow, to share one database connection across mutliple SQL destinations? Consider this setup (shortened): destination table_a { sql( type(psql) host("localhost") username("my_login") password("secret") database("my_db") table("table_a") columns(...) values (...) ); }; destination table_b { sql( type(psql) host("localhost") username("my_login") password("secret") database("my_db") table("table_b") columns(...) values (...) ); }; destination table_c { sql( type(psql) host("localhost") username("my_login") password("secret") database("my_db") table("table_c") columns(...) values (...) ); }; when i'm using this setup, syslog open 3 separate connections to database, 1 connection for each destination. Is it possible to configure syslog-ng to use only 1 connection, as database and login are the same for all 3 destinations ? If there would be, let's say 50 different tables in single DB, there would be 50 separate connections, which is a waste of resources, as Postgres assign memory for each connection. Also the config in syslog would be much simpler and easier to maintain, if the connection would be described in some section, and destination would define table name and point to some previously defined DB connection. Thanks for helping -- Tomáš Novosad LinuxBox.cz, s.r.o. 28. října 168, 709 00 Ostrava tel.: +420 591 166 221 mobil: +420 737 238 655 email: tomas.novosad(a)linuxbox.cz jabber: novosad(a)linuxbox.cz www.linuxbox.cz mobil servis: +420 737 238 656 email servis: servis(a)linuxbox.cz

2 3

syslog-ng not reconnecting when using tcp
by prasad padiyar 02 Apr '14

02 Apr '14

Hi, we use syslog-ng on one of our nodes as client and rsyslogd as server. We have configured tcp to connect to server. What we see is when the reboot the complete cluster (has node where the server runs and other node where the client runs,) only the server node is rebooted. The client node where syslog-ng runs is not rebooted. The issue is after we reboot the syslog-ng connection is not re-established and hence we are not getting any logs on the server. After going through some of the posts related to the problem and the admin guide, it mentions about time_reopen() value to solve this issue. In my case, i have configured time_reopen(40). So it means at 40s interval the tcp connection is tired to be re-established. But i see that this is not happening. Can you please help me resolve this issue. Also can you let me know how many times and at what interval this connection re-establishment is tried. Below are some of the logs that could be helpful. syslog-ng.conf: cat /etc/syslog-ng/syslog-ng.conf # syslog-ng configuration file. # # This should behave pretty much like the original syslog on RedHat. But # it could be configured a lot smarter. # # See syslog-ng(8) and syslog-ng.conf(5) for more information. # # 20000925 gb(a)sysfive.com # # Updated by Frank Crawford (<Frank.Crawford(a)ac3.com.au>) - 10 Aug 2002 # - for Red Hat 7.3 # - totally do away with klogd # - add message "kernel:" as is done with klogd. # # Updated by Frank Crawford (<Frank.Crawford(a)ac3.com.au>) - 22 Aug 2002 # - use the log_prefix option as per Balazs Scheidler's email # # Updated by Jose Pedro Oliveira (<jpo at di.uminho.pt>) - 05 Apr 2003 # - corrected filters 'f_filter2' and 'f_filter6' # these filters were only allowing messages of one specific # priority level; they should be allowing messages from that # priority and upper levels. # # Updated by Jose Pedro Oliveira (<jpo at di.uminho.pt>) - 25 Jan 2005 # - Don't sync the d_mail destination # # Updated by Jose Pedro Oliveira (<jpo at di.uminho.pt>) - 01 Feb 2005 # - /proc/kmsg is a file not a pipe. # ( https://lists.balabit.hu/pipermail/syslog-ng/2005-February/006963.html) # options { sync (0); time_reopen (40); log_fifo_size (1000); stats(86400); long_hostnames (off); use_dns (no); use_fqdn (no); create_dirs (no); keep_hostname (yes); }; source s_sys { file ("/proc/kmsg" log_prefix("kernel: ")); unix-stream ("/dev/log"); internal(); # udp(ip(0.0.0.0) port(514)); }; destination d_cons { file("/dev/console"); }; #destination d_mesg { file("/var/log/messages"); }; #destination d_auth { file("/var/log/secure"); }; destination d_mail { file("/var/log/maillog" sync(10)); }; destination d_spol { file("/var/log/spooler"); }; destination d_boot { file("/var/log/boot.log"); }; destination d_cron { file("/var/log/cron"); }; destination d_mlal { usertty("*"); }; destination tcp-to-master { tcp("169.254.1.82" localip("169.254.1.66") localport(601) port(601) ); }; #filter f_filter1 { facility(kern); }; filter f_filter2 { level(info..emerg) and not facility(mail,authpriv,cron) and not match("Connection broken to AF_INET") and not match("Error connecting to remote host AF_INET"); }; filter f_filter3 { facility(authpriv); }; filter f_filter4 { facility(mail); }; filter f_filter5 { level(emerg); }; filter f_filter6 { facility(uucp) or (facility(news) and level(crit..emerg)); }; filter f_filter7 { facility(local7); }; filter f_filter8 { facility(cron); }; log { source(s_sys); filter(f_filter2); destination(tcp-to-master); }; #log { source(s_sys); filter(f_filter1); destination(d_cons); }; # Redirecting the logs to /var/log/ of CLA, to avoid log files filling up AHUB3-A file system #log { source(s_sys); filter(f_filter2); destination(d_mesg); }; #log { source(s_sys); filter(f_filter3); destination(d_auth); }; log { source(s_sys); filter(f_filter4); destination(d_mail); }; log { source(s_sys); filter(f_filter5); destination(d_mlal); }; log { source(s_sys); filter(f_filter6); destination(d_spol); }; log { source(s_sys); filter(f_filter7); destination(d_boot); }; log { source(s_sys); filter(f_filter8); destination(d_cron); }; netstat output when the cluster was rebooted: Transition from ESTABLISHED to CLOSED_WAIT tcp 0 0 169.254.1.67:601 169.254.1.82:601 ESTABLISHED 3771/syslog-ng 10.34.37.082224000 tcp 1 0 169.254.1.67:601 169.254.1.82:601 CLOSE_WAIT 3771/syslog-ng CLOSED_WAIT to LAST_ACK tcp 1 0 169.254.1.67:601 169.254.1.82:601 CLOSE_WAIT 3771/syslog-ng 10.34.43.555017000 tcp 1 1 169.254.1.67:601 169.254.1.82:601 LAST_ACK - 10.34.43.644520000 LAST_ACK to CLOSED: tcp 1 1 169.254.1.67:601 169.254.1.82:601 LAST_ACK - 10.36.29.754533000 tcp 1 1 169.254.1.67:601 169.254.1.82:601 LAST_ACK - 10.36.29.801733000 tcp 1 1 169.254.1.67:601 169.254.1.82:601 LAST_ACK - 10.36.29.846228000 10.36.29.892427000 10.36.29.931181000 SYN_SENT: 10.36.43.192808000 tcp 0 1 169.254.1.67:601 169.254.1.82:601 SYN_SENT 3771/syslog-ng 10.36.43.238811000 tcp 0 1 169.254.1.67:601 169.254.1.82:601 SYN_SENT 3771/syslog-ng 10.36.43.279561000 tcp 0 1 169.254.1.67:601 169.254.1.82:601 SYN_SENT 3771/syslog-ng continues... 10.36.46.180840000 tcp 0 1 169.254.1.67:601 169.254.1.82:601 SYN_SENT 3771/syslog-ng 10.36.46.225891000 10.36.46.275090000 10.36.46.318485000 10.36.46.359169000 The server node comes up and it start rsyslogd after some more time. But when this starts, there is no syn packets received by rsyslogd and hence the connection is not established. Can you please let me know where exactly 40s(configured time_reopen value) coming into picture here in the netstat output? syslog-ng version: syslog-ng 1.6.12 ----------------------------------------------------------------------------------------------------------------------------------------------------- I got a reply from one of the balabit guys that i have to use version 3.4. I am currently trying to use it. But i still have some questions below. 1. How many number of times the connection is retried. 2. when exactly the connection retry happens (when will it get to know when it has to retry) 3. time_reopen(40), what exactly is 40seconds here. -- Regards, Prasad

2 2

SQL destination - inserting data into ARRAY column
by Tomáš Novosad 02 Apr '14

02 Apr '14

Hello everyone, i'm using syslog-ng to parse/store some logs into PostgreSQL database. I've got full control of the data being logged, because it is produced in my perl code. Data are logged into syslog and written unchanged into file, and in separate log statement, data are parsed into macros and inserted into database. There is a column which hold a list of email addresses. The column in database is defined as array of text (mail_to TEXT[]). This works fine, as the value is being parsed into macro, and this is enclosed in brackets. In detail log statement looks like: mail_to="addr1,addr2" which is parsed with pattern db into a macro ${mflog.to} then the macro is surrounded by brackets - this is done with set statement: set("{${mfstat.to}}", value("mfstat.to")); Syslog then insert this value into postgres:'{addr1,addr2}' Postgres successfuly parses the string and store it as two-element array [addr1,addr2]. This works just fine, but now to my problem. I also want to store another ARRAY colum, which is not of TEXT type, but a custom data type. The definition in database looks like: (parts part_data[]) where part_data is my custom type defined in postgres. Problems comes when i try to insert such data via syslog. My perl program produces this line which is sent to syslog: parts="(part1,20,bla),(part2,234,bli)" this is again parsed into macro and macro is again surronded by brackets. So syslog tries to insert this '{(part1,20,bla),(part2,234,bli)}' which fails because target column if of type part_data[], but postgres parses the string as text[]. Therfore i need to explicitly instruct postgres to do a data conversion. This can be easily achieved by adding the conversion to input string '{(part1,20,bla),(part2,234,bli)}'::part_data[] however i do no know how to achieve this, as the literal text '::part_data[]' is just outside the single qutes. I just do not know how to instruct syslog-ng to add extra text ::part_data[] just after the last single quote. All the inserted values are simple strings and i do not know how to add some extramodifications to that strings. Is there any way to do such thing? Similar thing would be for example running some function on inserted value. Let's say i would like the insert to look like this "INSERT into table (col1) VALUES (some_postgres_func(val1))" How do i set the sql destination in syslog-ng to achieve this? Many thanks for any ideas. -- Tomáš Novosad

2 2

Structured Data
by Ou, Jimmy 01 Apr '14

01 Apr '14

Hello, I am having problem removing the structured data in the syslog message. My test logs show the following: <113>1 2014-11-25T11:00:00+00:00 10.1.1.1 RedBox - - [meta sequenceId="16"] @cee:[{"host":"10.1.1.1"}] I want it to look like the following without the sequenceId tag: <113>1 2014-11-25T11:00:00+00:00 10.1.1.1 RedBox - - - @cee:[{"host":"10.1.1.1"}] Is there an option to turn this meta data off? Thanks, Jimmy

2 2

Filter out host from netmask range
by Ian Hasund 31 Mar '14

31 Mar '14

I am trying to remove a specific set of hosts from a network range using filters. for example, I have the following: filter f_home_subnet { ( netmask("192.168.0.0/24") ); }; I now want to exclude some specific hosts, let's say 192.168.0.10 and 192.168.0.20. My attempts of adding "not host" to the above doesn't seem to work. Any pointers / suggestions would be greatly appreciated. Thanks, Ian

3 4

pattern-db parser
by Justin B 31 Mar '14

31 Mar '14

Hello I have the Syslog-ng edition that came in built with Solaris Operating system and I did not find the pattern-db.xml or the default directory where it exists. Can I still create manually and use the Pattern-db parser?? I would want to use this on the operating system logs. Appreciate your response on this -- Kaladhar

2 1

Timestamp format
by Ou, Jimmy 28 Mar '14

28 Mar '14

Hello, I am having trouble using the timestamp formatter. The document says I can use ts_format() as rfc3164, bsd, rfc3339, or iso. I've tried multiple values and have gotten the same result. I've tried putting the ts_format(rfc3339) in the global options and in the destination's syslog() and have gotten the same result. My test logs show the following no matter what ts_format I set: <113>1 2014-11-25T11:00:00+00:00 10.1.1.1 RedBox - I want it to look like the following: <113>1 2014-11-25T11:00:00.000Z 10.1.1.1 RedBox - Am I missing something? Thanks, Jimmy Ou

2 1

[Bug 274] New: 3.5.4.1 memory leak
by bugzilla＠bugzilla.balabit.com 27 Mar '14

27 Mar '14

https://bugzilla.balabit.com/show_bug.cgi?id=274 Summary: 3.5.4.1 memory leak Product: syslog-ng Version: 3.5.x Platform: PC OS/Version: Linux Status: NEW Severity: normal Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: erempel(a)uvic.ca Type of the Report: --- Estimated Hours: 0.0 in our production environment our syslog process consumed 8GB of RAM while not queueing any messages. We take 10 minute syslog-ng-ctl stats snapshots and 15 minute process (ps -ef) style snapshots. There is definitely a memory leak there somewhere. We run multiple pattern databases, with multiple json output streams along side regular file destinations. What would you like from me to track it down? - valgrind - core dump - recompile with debuggin? If you provide the flags, I'll provide the data :-) -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 3