September 2016 - syslog-ng - lists.balabit.hu

Ubuntu 16 unmet dependencies - won't install
by Clayton Dukes 03 Oct '16

03 Oct '16

Hi Guys, Did something change recently with http://download.opensuse.org/repositories/home:/laszlo_budai:/syslog-ng/xUb… (or am I just overworked and missing something obvious?) I am unable to install now from a fresh OS: root@foo[~]: # apt-get install syslog-ng-core # (also tried syslog-ng syslog-ng-core here) Reading package lists... Done Building dependency tree Reading state information... Done Some packages could not be installed. This may mean that you have requested an impossible situation or if you are using the unstable distribution that some required packages have not yet been created or been moved out of Incoming. The following information may help to resolve the situation: The following packages have unmet dependencies: syslog-ng : Depends: syslog-ng-mod-sql but it is not going to be installed Depends: syslog-ng-mod-mongodb but it is not going to be installed Depends: syslog-ng-mod-json but it is not going to be installed Recommends: syslog-ng-mod-journal but it is not going to be installed Recommends: syslog-ng-mod-smtp but it is not going to be installed Recommends: syslog-ng-mod-amqp but it is not going to be installed Recommends: syslog-ng-mod-geoip but it is not going to be installed Recommends: syslog-ng-mod-redis but it is not going to be installed Recommends: syslog-ng-mod-stomp but it is not going to be installed Recommends: syslog-ng-mod-riemann but it is not going to be installed Recommends: syslog-ng-mod-graphite but it is not going to be installed Recommends: syslog-ng-mod-python but it is not going to be installed Recommends: syslog-ng-mod-java but it is not going to be installed Recommends: syslog-ng-mod-java-common-lib but it is not going to be installed Recommends: syslog-ng-mod-elastic but it is not going to be installed Recommends: syslog-ng-mod-hdfs but it is not going to be installed Recommends: syslog-ng-mod-kafka but it is not going to be installed Recommends: syslog-ng-mod-http but it is not going to be installed Recommends: syslog-ng-mod-curl but it is not going to be installed Recommends: syslog-ng-mod-kvformat but it is not going to be installed Recommends: syslog-ng-mod-add-contextual-data but it is not going to be installed syslog-ng-core : Depends: syslog-ng-mod-journal but it is not going to be installed E: Unable to correct problems, you have held broken packages. [cid:image001.png@01D21B13.4E86C770] Clayton Dukes / Founder and CTO http://www.logzilla.net<http://www.logzilla.net/>

3 4

ERROR:lib/dnscache.c:112:dns_cache_key_hash: code should not be reached
by Claus Albøge 01 Oct '16

01 Oct '16

Hi, I have noticed that syslog-ng exits from time to time, with the following error: ERROR:lib/dnscache.c:112:dns_cache_key_hash: code should not be reached Trying to run syslog-ng from gdb show the following: # gdb --args /usr/sbin/syslog-ng -F -p /var/run/syslogd.pid GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-80.el7 Copyright (C) 2013 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-redhat-linux-gnu". For bug reporting instructions, please see: <http://www.gnu.org/software/gdb/bugs/>... Reading symbols from /usr/sbin/syslog-ng...Reading symbols from /usr/lib/debug/usr/sbin/syslog-ng.debug...done. done. (gdb) run Starting program: /usr/sbin/syslog-ng -F -p /var/run/syslogd.pid [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". [New Thread 0x7ffff1e8a700 (LWP 18743)] [New Thread 0x7ffff1689700 (LWP 18748)] [New Thread 0x7ffff0e88700 (LWP 18893)] [New Thread 0x7fffe3fff700 (LWP 18956)] [New Thread 0x7fffe37fe700 (LWP 18959)] [New Thread 0x7fffe2ffd700 (LWP 18960)] [New Thread 0x7fffe27fc700 (LWP 18961)] [New Thread 0x7fffe1ffb700 (LWP 18962)] [Thread 0x7ffff1689700 (LWP 18748) exited] [Thread 0x7fffe27fc700 (LWP 18961) exited] [Thread 0x7fffe3fff700 (LWP 18956) exited] [New Thread 0x7fffe3fff700 (LWP 19114)] [New Thread 0x7fffe27fc700 (LWP 19136)] [New Thread 0x7ffff1689700 (LWP 19137)] [Thread 0x7fffe27fc700 (LWP 19136) exited] [Thread 0x7ffff1e8a700 (LWP 18743) exited] [New Thread 0x7ffff1e8a700 (LWP 19338)] [New Thread 0x7fffe27fc700 (LWP 19339)] [Thread 0x7fffe27fc700 (LWP 19339) exited] [Thread 0x7ffff1e8a700 (LWP 19338) exited] [New Thread 0x7ffff1e8a700 (LWP 19566)] [New Thread 0x7fffe27fc700 (LWP 19567)] [Thread 0x7ffff1e8a700 (LWP 19566) exited] [Thread 0x7fffe37fe700 (LWP 18959) exited] [New Thread 0x7ffff1e8a700 (LWP 19731)] [New Thread 0x7fffe37fe700 (LWP 19732)] [Thread 0x7fffe27fc700 (LWP 19567) exited] [Thread 0x7fffe1ffb700 (LWP 18962) exited] [New Thread 0x7fffe1ffb700 (LWP 19984)] [New Thread 0x7fffe27fc700 (LWP 19985)] ** ERROR:lib/dnscache.c:112:dns_cache_key_hash: code should not be reached Program received signal SIGABRT, Aborted. [Switching to Thread 0x7ffff1689700 (LWP 19137)] 0x00007ffff5fb05f7 in raise () from /lib64/libc.so.6 Missing separate debuginfos, use: debuginfo-install bzip2-libs-1.0.6-13.el7.x86_64 elfutils-libelf-0.163-3.el7.x86_64 elfutils-libs-0.163-3.el7.x86_64 eventlog-0.2.13-4.el7.x86_64 glib2-2.42.2-5.el7.x86_64 glibc-2.17-106.el7_2.8.x86_64 ivykis-0.36.2-2.el7.x86_64 json-c-0.11-4.el7_0.x86_64 keyutils-libs-1.5.8-3.el7.x86_64 krb5-libs-1.13.2-12.el7_2.x86_64 libattr-2.4.46-12.el7.x86_64 libcap-2.22-8.el7.x86_64 libcom_err-1.42.9-7.el7.x86_64 libgcc-4.8.5-4.el7.x86_64 libgcrypt-1.5.3-12.el7_1.1.x86_64 libgpg-error-1.12-3.el7.x86_64 libnet-1.1.6-7.el7.x86_64 libselinux-2.2.2-6.el7.x86_64 libuuid-2.23.2-26.el7_2.3.x86_64 openssl-libs-1.0.1e-51.el7_2.5.x86_64 pcre-8.32-15.el7_2.1.x86_64 systemd-libs-219-19.el7_2.13.x86_64 tcp_wrappers-libs-7.6-77.el7.x86_64 xz-libs-5.1.2-12alpha.el7.x86_64 zlib-1.2.7-15.el7.x86_64 (gdb) (gdb) bt #0 0x00007ffff5fb05f7 in raise () from /lib64/libc.so.6 #1 0x00007ffff5fb1ce8 in abort () from /lib64/libc.so.6 #2 0x00007ffff70426d5 in g_assertion_message () from /lib64/libglib-2.0.so.0 #3 0x00007ffff704276a in g_assertion_message_expr () from /lib64/libglib-2.0.so.0 #4 0x00007ffff7b62cc4 in dns_cache_key_hash (e=<optimized out>) at lib/dnscache.c:112 #5 0x00007ffff700c4fc in g_hash_table_remove_internal () from /lib64/libglib-2.0.so.0 #6 0x00007ffff7b67584 in resolve_sockaddr_to_inet_or_inet6_hostname (host_resolve_options=0x651754, saddr=0x7fffec3840d0, result_len=0x7ffff1686370) at lib/host-resolve.c:332 #7 resolve_sockaddr_to_hostname (result_len=result_len@entry=0x7ffff1686370, saddr=0x7fffec3840d0, host_resolve_options=0x651754) at lib/host-resolve.c:343 #8 0x00007ffff7b6c03a in log_source_mangle_hostname (self=self@entry=0x1cdf400, msg=msg@entry=0x7fffe4ec6da0) at lib/logsource.c:156 #9 0x00007ffff7b6c280 in log_source_queue (s=0x1cdf400, msg=0x7fffe4ec6da0, path_options=0x7ffff1686550, user_data=<optimized out>) at lib/logsource.c:266 #10 0x00007ffff7b6ca0a in log_pipe_queue (path_options=<optimized out>, msg=<optimized out>, s=<optimized out>) at lib/logpipe.h:339 #11 log_pipe_forward_msg (self=<optimized out>, path_options=<optimized out>, msg=<optimized out>) at lib/logpipe.h:302 #12 log_pipe_queue (path_options=<optimized out>, msg=<optimized out>, s=<optimized out>) at lib/logpipe.h:343 #13 log_pipe_forward_msg (self=<optimized out>, path_options=<optimized out>, msg=<optimized out>) at lib/logpipe.h:302 #14 log_pipe_queue (path_options=<optimized out>, msg=0x7fffe4ec6da0, s=0x7fffe4ec6da0) at lib/logpipe.h:343 #15 log_source_post (self=self@entry=0x1cdf400, msg=msg@entry=0x7fffe4ec6da0) at lib/logsource.c:249 #16 0x00007ffff7b6b179 in log_reader_handle_line (aux=0x7ffff1686600, length=<optimized out>, line=<optimized out>, self=0x1cdf400) at lib/logreader.c:320 #17 log_reader_fetch_log (self=self@entry=0x1cdf400) at lib/logreader.c:381 #18 0x00007ffff7b6b299 in log_reader_work_perform (s=0x1cdf400) at lib/logreader.c:88 #19 0x00007ffff7b7307d in _work (self=<optimized out>) at lib/mainloop-io-worker.c:52 #20 0x00007ffff655dc67 in iv_work_thread_do_work () from /lib64/libivykis.so.0 #21 0x00007ffff655cfc3 in iv_run_tasks () from /lib64/libivykis.so.0 #22 0x00007ffff655f7dc in iv_main () from /lib64/libivykis.so.0 #23 0x00007ffff655dab4 in iv_work_thread () from /lib64/libivykis.so.0 #24 0x00007ffff6560270 in iv_thread_handler () from /lib64/libivykis.so.0 #25 0x00007ffff6344dc5 in start_thread () from /lib64/libpthread.so.0 #26 0x00007ffff6071ced in clone () from /lib64/libc.so.6 # grep dns /etc/syslog-ng/syslog-ng.conf use_dns (no); Not sure what its trying to resolve. # syslog-ng -V syslog-ng 3.8.1 Installer-Version: 3.8.1 Revision: Module-Directory: //usr/lib64/syslog-ng Module-Path: //usr/lib64/syslog-ng Available-Modules: disk-buffer,sdjournal,afstomp,json-plugin,cryptofuncs,graphite,cef,kvformat,add-contextual-data,dbparser,pseudofile,csvparser,syslogformat,confgen,afsocket,afuser,date,linux-kmsg-format,system-source,basicfuncs,afamqp,affile,afprog Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-IPv6: on Enable-Spoof-Source: on Enable-TCP-Wrapper: on Enable-Linux-Caps: off Please let me know, if more info is needed. /Claus A

4 9

Converting multiline text input to concatanated single-line syslog format
by Newport, Brendan (Contractor - Security Operations - Development & Support) 30 Sep '16

30 Sep '16

Classification: Public Hi! My first posting on the mailing list. I've run into a snag trying to get multiline logs concatenated onto one line and written as syslog-format messages. Actually the second part is working fine; I can get the first line identified and incorporated into a syslog message, but all subsequent lines aren't included. Just for testing I've some simple input; 'IMP-386: ORACLE error 386 encountered ORA-01017: invalid username/password; logon deniedUsername: Connected to: Oracle Database 10g Enterprise Edition Release 10.2.0.3.0 - 64bit Production With the Partitioning, OLAP and Data Mining options Export file created by EXPORT:V10.02.01 via conventional path Warning: the objects were exported by FALCON, not by you import done in WE8ISO8859P1 character set and AL16UTF16 NCHAR character set IMP-00034: Warning: FromUser "FALBOS" not found in export file' What I'm aiming to do is log only the Oracle errors, commencing at the string 'IMP' and ending only when the next line is found with 'IMP' (this log only sees errors) So in syslog-ng.conf (version 5.0.6b PE) source s_table { file("/path to Table.log" multi-line-prefix("IMP") multi-line-garbage("set$") flags(no-p arse) flags(no-multi-line) program_override("Table") default-facility(local5) default-priority(info)); }; Just to try to get things working, I've hard-coded the final string present in my sample input - 'set' The output is scheduled to go off-server, but for the moment I want to see the transformed messages in their own file; destination d_table { file("/auditsox/table.log"); }; log { source(s_table); destination(d_table); }; What I get from the above (in d_table) is; Sep 29 14:07:15 p14425dev022 Table: IMP-386: ORACLE error 386 encountered < Ok, first line transformed into syslog messages ORA-01017: invalid username/password; logon deniedUsername: Connected to: Oracle Database 10g Enterprise Edition Release 10.2.0.3.0 - 64bit Production With the Partitioning, OLAP and Data Mining options Export file created by EXPORT:V10.02.01 via conventional path Warning: the objects were exported by FALCON, not by you import done in WE8ISO8859P1 character set and AL16UTF16 NCHAR character Sep 29 14:07:26 p14425dev022 Table: IMP-00034: Warning: FromUser "FALBOS" not found in export file _____________________ The snag is, the lines after the first line in the input aren't concatenated, but are rather individual lines, with an LF. So I'm doing something wrong. With this source; source s_table { file("/path to Table.log " flags(no-multi-line) flags(no-parse) program_override(" Table") default-facility(local5) default-priority(info)); }; I can get every line of input to transform into a syslog message. Not much use though! Can anyone provide a pointer as to what I'm doing wonky? Thanks Brendan Lloyds Banking Group plc. Registered Office: The Mound, Edinburgh EH1 1YZ. Registered in Scotland no. SC95000. Telephone: 0131 225 4555. Lloyds Bank plc. Registered Office: 25 Gresham Street, London EC2V 7HN. Registered in England and Wales no. 2065. Telephone 0207626 1500. Bank of Scotland plc. Registered Office: The Mound, Edinburgh EH1 1YZ. Registered in Scotland no. SC327000. Telephone: 03457 801 801. Cheltenham & Gloucester plc. Registered Office: Barnett Way, Gloucester GL4 3RL. Registered in England and Wales 2299428. Telephone: 0345 603 1637 Lloyds Bank plc, Bank of Scotland plc are authorised by the Prudential Regulation Authority and regulated by the Financial Conduct Authority and Prudential Regulation Authority. Cheltenham & Gloucester plc is authorised and regulated by the Financial Conduct Authority. Halifax is a division of Bank of Scotland plc. Cheltenham & Gloucester Savings is a division of Lloyds Bank plc. HBOS plc. Registered Office: The Mound, Edinburgh EH1 1YZ. Registered in Scotland no. SC218813. This e-mail (including any attachments) is private and confidential and may contain privileged material. If you have received this e-mail in error, please notify the sender and delete it (including any attachments) immediately. You must not copy, distribute, disclose or use any of the information in it or any attachments. Telephone calls may be monitored or recorded.

6 7

Converting multiline text input to concatanated single-line syslog format
by Newport, Brendan (Contractor - Security Operations - Development & Support) 30 Sep '16

30 Sep '16

Classification: Public Ha! I think I might have been trying to achieve the impossible - getting syslog-ng to covert multiline log messages into concatenated syslog messages. That doesn't appear to work. But I defined syslogd instance to direct local5.info messatges received from my syslog-ng client into a log file and all works fine! more local5.log Sep 30 14:45:35 <hostname> local5:info <hostname> Table: IMP-386: ORACLE error 386 encountered ORA-01017: invalid username/passwo rd; logon deniedUsername: Connected to: Oracle Database 10g Enterprise Edition Release 10.2.0.3.0 - 64bit Production Sep 30 14:45:35 <hostname> local5:info <hostname> Table: IMP-00034: Warning: FromUser "FALBOS" not found in export file With the P artitioning, OLAP and Data Mining options Export file created by EXPORT:V10.02.01 via conventional path Sep 30 14:45:35 <hostname> local5:info <hostname> Table: IMP-386: ORACLE error 386 encountered Warning: the objects were exported by FALCON, not by you import done in WE8ISO8859P1 character set and AL16UTF16 NCHAR character set Using this config is syslog-ng.conf; source s_table { file("<path to input multi-line log file>" multi-line-prefix("IMP") flags(no-parse) flags(no-multi-line) program_override("Table") default-facility(local5) default-priority(info)); }; filter f_local5_info { facility(local5) and level(info); }; and with the destination defined as a udp IP address to port 514; log { source(s_table); filter(f_local5_info); destination(d_syslog_udp); }; As the desire is to get multiline non-syslog logs off-host to Qradar as syslog messages, that's the POC passed fine! Lloyds Banking Group plc. Registered Office: The Mound, Edinburgh EH1 1YZ. Registered in Scotland no. SC95000. Telephone: 0131 225 4555. Lloyds Bank plc. Registered Office: 25 Gresham Street, London EC2V 7HN. Registered in England and Wales no. 2065. Telephone 0207626 1500. Bank of Scotland plc. Registered Office: The Mound, Edinburgh EH1 1YZ. Registered in Scotland no. SC327000. Telephone: 03457 801 801. Cheltenham & Gloucester plc. Registered Office: Barnett Way, Gloucester GL4 3RL. Registered in England and Wales 2299428. Telephone: 0345 603 1637 Lloyds Bank plc, Bank of Scotland plc are authorised by the Prudential Regulation Authority and regulated by the Financial Conduct Authority and Prudential Regulation Authority. Cheltenham & Gloucester plc is authorised and regulated by the Financial Conduct Authority. Halifax is a division of Bank of Scotland plc. Cheltenham & Gloucester Savings is a division of Lloyds Bank plc. HBOS plc. Registered Office: The Mound, Edinburgh EH1 1YZ. Registered in Scotland no. SC218813. This e-mail (including any attachments) is private and confidential and may contain privileged material. If you have received this e-mail in error, please notify the sender and delete it (including any attachments) immediately. You must not copy, distribute, disclose or use any of the information in it or any attachments. Telephone calls may be monitored or recorded.

2 1

syslog-ng forwarding and processing issue
by Denis Dolinský 30 Sep '16

30 Sep '16

Hi guys, I have following setup in place: remote server - 192.168.1.10 pseudomizer - syslog-ng PE in client mode - 192.168.2.10 SIEM - 192.168.3.10 So I am sending syslog logs from remote server to pseudomizer: source src { internal()}; destination dst { udp ("192.168.2.10) port (514);}; log { source(src); destination (dst); this is very old config from syslog v4 Then on pseudomizer - syslog-ng LTS 6.0.1 PE, I am collecting the logs, processing them - removing private data, putting pseudonyms instead and forwarding them to SIEM. source s_net_udp514 { syslog( ip(192.168.2.10) ip-protocol(4) transport("udp") so_rcvbuf(2097152) ); }; source src { internal(); unix-dgram("/dev/log"); system (); }; destination d_net_udp514 { syslog ( "192.168.3.10" port(514) transport(udp) spoof_source(yes) mark_mode(periodical));}; rewrite r_rewrite { subst("admin", "pseudonym000001", value("MESSAGE"), flags("global")); log { source(s_net_udp514); source (src); rewrite(r_rewrite); # do the pseudomizing destination(d_net_udp514); }; On SIEM device, I can see only pseudomizer internal logs (src), not processed logs from remote server. Any advice ? Many thanks. Denis

5 10

Converting multiline text input to concatanated single-line syslog format
by Newport, Brendan (Contractor - Security Operations - Development & Support) 30 Sep '16

30 Sep '16

Classification: Public Hi! Thanks for the responses. The multi-line-garbage() line was only in there to aid testing. I will look at the LF/CR elements in the input. We have a support contract with BalaBit with a local Var defined and so I'll raise a support ticket. Version 5.0.6LTS is the version rolled-out across our enterprise, but an upgrade to 6LTS will likely occur before October 2017. So testing was performed with the version in use. Lloyds Banking Group plc. Registered Office: The Mound, Edinburgh EH1 1YZ. Registered in Scotland no. SC95000. Telephone: 0131 225 4555. Lloyds Bank plc. Registered Office: 25 Gresham Street, London EC2V 7HN. Registered in England and Wales no. 2065. Telephone 0207626 1500. Bank of Scotland plc. Registered Office: The Mound, Edinburgh EH1 1YZ. Registered in Scotland no. SC327000. Telephone: 03457 801 801. Cheltenham & Gloucester plc. Registered Office: Barnett Way, Gloucester GL4 3RL. Registered in England and Wales 2299428. Telephone: 0345 603 1637 Lloyds Bank plc, Bank of Scotland plc are authorised by the Prudential Regulation Authority and regulated by the Financial Conduct Authority and Prudential Regulation Authority. Cheltenham & Gloucester plc is authorised and regulated by the Financial Conduct Authority. Halifax is a division of Bank of Scotland plc. Cheltenham & Gloucester Savings is a division of Lloyds Bank plc. HBOS plc. Registered Office: The Mound, Edinburgh EH1 1YZ. Registered in Scotland no. SC218813. This e-mail (including any attachments) is private and confidential and may contain privileged material. If you have received this e-mail in error, please notify the sender and delete it (including any attachments) immediately. You must not copy, distribute, disclose or use any of the information in it or any attachments. Telephone calls may be monitored or recorded.

1 0

attempting to get parsed logs into elasticsearch
by Russell Fulton 30 Sep '16

30 Sep '16

Hi folks I am trying to get some parsed logs into elasticssearch but have ended up with a cryptic error message: Starting syslog-ng /usr/lib/jvm/java-1.7.0-oracle-1.7.0.91.x86_64/jre/lib/amd64/server [2016-09-30T12:43:43.649899] Error initializing message pipeline; which almost certainly relates to the ES set up but I have no idea what is actually wrong. The ES logs do not show anything. Config file: @version: 3.8 @module mod-java @include "scl.conf" options { use_dns (no); use_fqdn (no); keep_hostname (yes); }; source s_loghost { tcp(flags(no-multi-line) port(1514) keep-alive(yes)); }; destination d_syslog { file("/var/log/syslog.log"); }; destination d_elastic { elasticsearch( index("auth_${YEAR}.${MONTH}.${DAY}") type("auth") cluster("security") flush-limit("1000") ); }; parser p_patterns { db-parser( file("/etc/syslog-ng/merged.xml")); }; log { source(s_loghost); parser (p_patterns); destination(d_elastic ); }; The same configuration with a json file destination works fine. Any hints on what to look at appreciated. The ES instance running on the host is set to data: no and I expect it to ship the data to one of the other nodes which has storage. Russell (who admits to being an ES novice)

2 3

moving debian directory from master to release branch on GitHub
by Noémi Ványi 29 Sep '16

29 Sep '16

Hello! Debian directory is being moved from master to another branch named release. Thus, the source code is separated from the build. If someone wants to create a Debian package, he/she must checkout that branch first. If someone has any concerns or notes, please share either here or in the PR on GitHub. PR: https://github.com/balabit/syslog-ng/pull/1215 kvch

2 1

experimental syslog-ng incubator rpm-s for 3.8
by Czanik, Péter 29 Sep '16

29 Sep '16

Hi, I just updated my unofficial syslog-ng 3.8 fedora/rhel repo with an experimental incubator package. I used the latest git HEAD snapshot together with this patch: https://github.com/balabit/syslog-ng-incubator/pull/158 Notable changes since my packages for 3.7: - bugfixes & code changes needed to compile on 3.8 - grok support removed - librdkafka updated to 0.9.1 Due to lack of Kafka, ZeroMQ, etc. I did not test the packages, just compiled them. If you need grok, let me know and I'll try to fix the package... As usual, packages are available at https://copr.fedorainfracloud.org/coprs/czanik/syslog-ng38/ Your feedback, be it negative or positive are very welcome! Bye, Peter Czanik (CzP) <peter.czanik(a)balabit.com> Balabit / syslog-ng upstream http://czanik.blogs.balabit.com/ https://twitter.com/PCzanik

1 0

Re: [syslog-ng] disk based buffering
by thejaguar＠tutanota.de 29 Sep '16

29 Sep '16

So with dqtool ( cool) I noticed that actual problem is that syslog-ng is not writing to disk queue immediately when a message is generated. The message goes to usual place like var/log/message etc but not in the queue where it should because it has not been transmitted to remote host. Now I am thinking there has to be a memory limit before it starts flushing/dumping unsent message to disk queue ? I tried mem-buf-size(1) and log-fifo-size(1) but none of them works. Is it a static value or is it configurable ? Thanks 27. Sep 2016 12:41 by thejaguar(a)tutanota.de: > > Thanks.> So if the queue stays intact, syslog-ng will try to send unsent messages as and when it starts ? even after 2-3 days ? it does not reset the queue or tracking ever ? > Thanks again > > 27. Sep 2016 12:11 by > balazs.scheidler(a)balabit.com> : > > >> >> Syslog-ng attempts to address application level failures with reliable disk buffer but kernel level crashes/power failures are not covered, at least you can suffer message loss, but the queue in general should stay intact. >> >> There's a tool for reading disk queue files, iirc the name is dqtool, should be included in your package. >> >> On Sep 27, 2016 8:35 PM, <>> thejaguar(a)tutanota.de>> > wrote: >> >>> >>> Hi,>>> I have been using disk based buffering with reliable turned on yes as suggested here :->>> https://www.balabit.com/documents/syslog-ng-ose-latest-guides/en/syslog-ng-… >>> >>> This has been working great for me on an embedded linux device which does not have internet connection except when the application running on it turns on the modem/pppd when it has to send some data, basically to save battery power. Now syslog-ng is brilliant and sends all the stored/queued logs immediately upon detecting network connection as long as system stays alive. Now the challenge is if the device has a system reset or kernel crash in between network connection availability, will syslog-ng send unsent logs upon next system reboot when it gets the network connection ? Or it resets the queue and tracking upon system reset/boot ?>>> I noticed any logs generated in between power resets and which are not sent are not transmitted on next net connection. Is it expected behaviour ? If not then what wrong I am doing ? also how can I read whats in /var/lib/syslog-ng/syslog-ng-00000.rqf or syslog-ng.persist ? >>> ======================= >>> >>> destination d_net {>>> network (>>> "`myloghost`" port(`mylogport`) transport("tls")>>> tls( ca-dir("/etc/syslog-ng/ca") peer-verify(required-trusted) ssl-options(no-sslv3,no-tlsv1) )>>> disk-buffer( reliable(yes) mem-buf-size(1M) disk-buf-size(5M) qout-size(64) )>>> template("<$PRI> $FACILITY $ISODATE $HOST $PROGRAM $MSG\n")>>> );>>> }; >>> syslog-ng 3.8.1>>> Installer-Version: 3.8.1>>> Revision:>>> Module-Directory: /usr/lib/syslog-ng>>> Module-Path: /usr/lib/syslog-ng>>> Available-Modules: cef,affile,basicfuncs,system-source,cryptofuncs,graphite,pseudofile,afuser,kvformat,add-contextual-data,date,csvparser,linux-kmsg-format,confgen,syslogformat,afprog,disk-buffer,dbparser,afsot>>> Enable-Debug: off>>> Enable-GProf: off>>> Enable-Memtrace: off>>> Enable-IPv6: off>>> Enable-Spoof-Source: off>>> Enable-TCP-Wrapper: off>>> Enable-Linux-Caps: off >>> ======================= >>> >>> Thanks >>> >>> >>> ______________________________________________________________________________ >>> Member info: >>> https://lists.balabit.hu/mailman/listinfo/syslog-ng >>> Documentation: >>> http://www.balabit.com/support/documentation/?product=syslog-ng >>> FAQ: >>> http://www.balabit.com/wiki/syslog-ng-faq >>> >>> >>> >> >>

5 6