syslog doesn't log if system date is older than date on last reboot
Hi, I have one embedded device running Linux which doesn't have RTC. So it loses date and time after every reboot. uname -a returns: Linux f1 4.9.175+g0746276 #1 SMP PREEMPT Tue Jan 14 03:09:19 UTC 2020 armv7l armv7l armv7l GNU/Linux I have observed after every reboot, my board starts with some older time and my application running in user-space corrects the system time from a timestamp saved on a file (which is updated every hour). Application also connects with some server to get latest time. I have noticed that if system date and time is less than date and time which it had at last shutdown, then syslog doesn't get any logs from my applications till it has a time atleast greater than date and time which it had at last shutdown. Please help me in debugging the issue and fixing it.
Hello! We have seen issues like this in case of systemd, but I dont know if your embedded system runs systemd. Can you share your configuration, please? Are you using system() source? Regards, Gábor On Tue, 4 Feb 2020, 08:33 Abhi Arora, <engr.abhiarora@gmail.com> wrote:
Hi, I have one embedded device running Linux which doesn't have RTC. So it loses date and time after every reboot.
uname -a returns:
Linux f1 4.9.175+g0746276 #1 SMP PREEMPT Tue Jan 14 03:09:19 UTC 2020 armv7l armv7l armv7l GNU/Linux
I have observed after every reboot, my board starts with some older time and my application running in user-space corrects the system time from a timestamp saved on a file (which is updated every hour). Application also connects with some server to get latest time. I have noticed that if system date and time is less than date and time which it had at last shutdown, then syslog doesn't get any logs from my applications till it has a time atleast greater than date and time which it had at last shutdown.
Please help me in debugging the issue and fixing it.
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Ye*s, We are using systemd* *Here is the output of cat /lib/systemd/system/syslog-ng\@.service* *[Unit]Description=System Logger Daemon "%i" instanceDocumentation=man:syslog-ng(8)Conflicts=emergency.service emergency.target[Service]Type=notifyEnvironmentFile=-/etc/default/syslog-ng@%iEnvironmentFile=-/etc/sysconfig/syslog-ng@%iExecStart=/usr/sbin/syslog-ng -F $OTHER_OPTIONS --cfgfile $CONFIG_FILE --control $CONTROL_FILE --persist-file $PERSIST_FILE --pidfile $PID_FILEExecReload=/bin/kill -HUP $MAINPIDStandardOutput=journalStandardError=journalRestart=on-failure[Install]WantedBy=multi-user.target* *This output is returned by cat /etc/syslog-ng/scl.conf* *############################################################################## Copyright (c) 2010-2014 Balabit## This program is free software; you can redistribute it and/or modify it# under the terms of the GNU General Public License version 2 as published# by the Free Software Foundation, or (at your option) any later version.## This program is distributed in the hope that it will be useful,# but WITHOUT ANY WARRANTY; without even the implied warranty of# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the# GNU General Public License for more details.## You should have received a copy of the GNU General Public License# along with this program; if not, write to the Free Software# Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA## As an additional exemption you are allowed to compile & link against the# OpenSSL libraries as published by the OpenSSL project. See the file# COPYING for details.################################################################################ This file is placed into /etc/syslog-ng in order to make it trivial to# include in user written syslog-ng.conf files. It sets up 'scl-root' and# `include-path`, then includes all SCL supplied plugins.#@include 'scl/*/*.conf'@define java-module-dir "`module-install-dir`/java-modules"* ******************** This by* *cat /etc/syslog-ng/syslog-ng.conf* *@version: 3.15## Syslog-ng configuration file, compatible with default Debian syslogd# installation. Originally written by anonymous (I can't find his name)# Revised, and rewrited by me (SZALAY Attila <sasa@debian.org <sasa@debian.org>>)# First, set some global options.options { chain_hostnames(off); flush_lines(0); use_dns(no); use_fqdn(no); owner("root"); group("adm"); perm(0640); stats_freq(0); bad_hostname("^gconfd$"); keep_hostname(yes); keep_timestamp(yes);};######################### Sources######################### This is the default behavior of sysklogd package# Logs may come from unix stream, but not from another machine.#source s_src { systemd_journal(); internal(); file("/proc/kmsg" program_override("kernel"));};# If you wish to get logs from remote machine you should uncomment# this and comment the above source line.##source s_net { tcp(ip(127.0.0.1) port(1000) authentication(required) encrypt(allow)); };######################### Destinations######################### First some standard logfile#destination d_auth { file("/var/log/auth.log"); };destination d_cron { file("/var/log/cron.log"); };destination d_daemon { file("/var/log/daemon.log"); };destination d_kern { file("/var/log/kern.log"); };destination d_lpr { file("/var/log/lpr.log"); };destination d_mail { file("/var/log/mail.log"); };destination d_syslog { file("/var/log/syslog"); };destination d_user { file("/var/log/user.log"); };destination d_uucp { file("/var/log/uucp.log"); };# This files are the log come from the mail subsystem.#destination d_mailinfo { file("/var/log/mail/mail.info <http://mail.info>"); };destination d_mailwarn { file("/var/log/mail/mail.warn"); };destination d_mailerr { file("/var/log/mail/mail.err"); };# Logging for INN news system#destination d_newscrit { file("/var/log/news/news.crit"); };destination d_newserr { file("/var/log/news/news.err"); };destination d_newsnotice { file("/var/log/news/news.notice"); };# Some 'catch-all' logfiles.#destination d_debug { file("/var/log/debug"); };destination d_error { file("/var/log/error"); };destination d_messages { file("/var/log/messages"); };# The root's console.#destination d_console { usertty("root"); };# Virtual console.#destination d_console_all { file("/dev/tty10"); };# The named pipe /dev/xconsole is for the nsole' utility. To use it,# you must invoke nsole' with the -file' option:## $ xconsole -file /dev/xconsole [...]#destination d_xconsole { pipe("/dev/xconsole"); };# Send the messages to an other host##destination d_net { tcp("127.0.0.1" port(1000) authentication(on) encrypt(on) log_fifo_size(1000)); };# Debian onlydestination d_ppp { file("/var/log/ppp.log"); };######################### Filters######################### Here's come the filter options. With this rules, we can set which# message go where.filter f_dbg { level(debug); };filter f_info { level(info); };filter f_notice { level(notice); };filter f_warn { level(warn); };filter f_err { level(err); };filter f_crit { level(crit .. emerg); };filter f_alllevel { level(debug, info, notice, warn,err,crit,emerg) and not filter(f_daemon);};filter f_debug { level(debug) and not facility(auth, authpriv, news, mail); };filter f_error { level(err .. emerg) ; };filter f_messages { level(info,notice,warn) and not facility(auth,authpriv,cron,daemon,mail,news); };filter f_auth { facility(auth, authpriv) and not filter(f_debug); };filter f_cron { facility(cron) and not filter(f_debug); };filter f_daemon { facility(daemon) and not filter(f_debug); };filter f_kern { facility(kern) and not filter(f_debug); };filter f_lpr { facility(lpr) and not filter(f_debug); };filter f_local { facility(local0, local1, local3, local4, local5, local6, local7) and not filter(f_debug); };filter f_mail { facility(mail) and not filter(f_debug); };filter f_news { facility(news) and not filter(f_debug); };filter f_syslog3 { not facility(auth, authpriv, mail) and not filter(f_debug); };filter f_user { facility(user) and not filter(f_debug); };filter f_uucp { facility(uucp) and not filter(f_debug); };filter f_cnews { level(notice, err, crit) and facility(news); };filter f_cother { level(debug, info, notice, warn) or facility(daemon, mail); };filter f_ppp { facility(local2) and not filter(f_debug); };filter f_console { level(warn .. emerg); };######################### Log paths########################log { source(s_src); filter(f_auth); destination(d_auth); };log { source(s_src); filter(f_cron); destination(d_cron); };log { source(s_src); filter(f_daemon); destination(d_daemon); };log { source(s_src); filter(f_kern); destination(d_Gateway); };log { source(s_src); filter(f_lpr); destination(d_lpr); };log { source(s_src); filter(f_syslog3); destination(d_syslog); };log { source(s_src); filter(f_user); destination(d_user); };log { source(s_src); filter(f_uucp); destination(d_uucp); };log { source(s_src); filter(f_mail); destination(d_mail); };#log { source(s_src); filter(f_mail); filter(f_info); destination(d_mailinfo); };#log { source(s_src); filter(f_mail); filter(f_warn); destination(d_mailwarn); };#log { source(s_src); filter(f_mail); filter(f_err); destination(d_mailerr); };log { source(s_src); filter(f_news); filter(f_crit); destination(d_newscrit); };log { source(s_src); filter(f_news); filter(f_err); destination(d_newserr); };log { source(s_src); filter(f_news); filter(f_notice); destination(d_newsnotice); };#log { source(s_src); filter(f_cnews); destination(d_console_all); };#log { source(s_src); filter(f_cother); destination(d_console_all); };#log { source(s_src); filter(f_ppp); destination(d_ppp); };log { source(s_src); filter(f_debug); destination(d_debug); };log { source(s_src); filter(f_error); destination(d_error); };log { source(s_src); filter(f_messages); destination(d_messages); };log { source(s_src); filter(f_console); destination(d_console_all); destination(d_xconsole); };log { source(s_src); filter(f_crit); destination(d_console); };# All messages send to a remote site##log { source(s_src); destination(d_net); };# GentApp/GatewayAppdestination d_Gateway { file("/var/log/Gateway.log"template("$FULLDATE $FULLHOST $PROGRAM $PID [$FACILITY.$LEVEL] $MESSAGE\n") ); };filter f_Gateway { match("Gate*") or match("Gent*") or match("Notifier*") or match("Alarm*"); };log { source(s_src); filter(f_Gateway);filter(f_alllevel); destination(d_Gateway);} ;* On Tue, Feb 4, 2020 at 1:24 PM Nagy Gábor <gabor.hl@gmail.com> wrote:
Hello!
We have seen issues like this in case of systemd, but I dont know if your embedded system runs systemd.
Can you share your configuration, please? Are you using system() source?
Regards, Gábor
On Tue, 4 Feb 2020, 08:33 Abhi Arora, <engr.abhiarora@gmail.com> wrote:
Hi, I have one embedded device running Linux which doesn't have RTC. So it loses date and time after every reboot.
uname -a returns:
Linux f1 4.9.175+g0746276 #1 SMP PREEMPT Tue Jan 14 03:09:19 UTC 2020 armv7l armv7l armv7l GNU/Linux
I have observed after every reboot, my board starts with some older time and my application running in user-space corrects the system time from a timestamp saved on a file (which is updated every hour). Application also connects with some server to get latest time. I have noticed that if system date and time is less than date and time which it had at last shutdown, then syslog doesn't get any logs from my applications till it has a time atleast greater than date and time which it had at last shutdown.
Please help me in debugging the issue and fixing it.
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Continuing my previous email....
Can you share your configuration, please? I have shared over my last email
Are you using system() source? I didn't get you. Please elaborate. You mean source code system() library function. If that you mean, then no we don't use it.
On Tue, Feb 4, 2020 at 1:24 PM Nagy Gábor <gabor.hl@gmail.com> wrote:
Hello!
We have seen issues like this in case of systemd, but I dont know if your embedded system runs systemd.
Can you share your configuration, please? Are you using system() source?
Regards, Gábor
On Tue, 4 Feb 2020, 08:33 Abhi Arora, <engr.abhiarora@gmail.com> wrote:
Hi, I have one embedded device running Linux which doesn't have RTC. So it loses date and time after every reboot.
uname -a returns:
Linux f1 4.9.175+g0746276 #1 SMP PREEMPT Tue Jan 14 03:09:19 UTC 2020 armv7l armv7l armv7l GNU/Linux
I have observed after every reboot, my board starts with some older time and my application running in user-space corrects the system time from a timestamp saved on a file (which is updated every hour). Application also connects with some server to get latest time. I have noticed that if system date and time is less than date and time which it had at last shutdown, then syslog doesn't get any logs from my applications till it has a time atleast greater than date and time which it had at last shutdown.
Please help me in debugging the issue and fixing it.
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Hi, Please let me know if I have missed anything. I really have to debug this issue. Please help me with some pointer as I am clueless. On Tue, Feb 4, 2020 at 1:59 PM Abhi Arora <engr.abhiarora@gmail.com> wrote:
Continuing my previous email....
Can you share your configuration, please? I have shared over my last email
Are you using system() source? I didn't get you. Please elaborate. You mean source code system() library function. If that you mean, then no we don't use it.
On Tue, Feb 4, 2020 at 1:24 PM Nagy Gábor <gabor.hl@gmail.com> wrote:
Hello!
We have seen issues like this in case of systemd, but I dont know if your embedded system runs systemd.
Can you share your configuration, please? Are you using system() source?
Regards, Gábor
On Tue, 4 Feb 2020, 08:33 Abhi Arora, <engr.abhiarora@gmail.com> wrote:
Hi, I have one embedded device running Linux which doesn't have RTC. So it loses date and time after every reboot.
uname -a returns:
Linux f1 4.9.175+g0746276 #1 SMP PREEMPT Tue Jan 14 03:09:19 UTC 2020 armv7l armv7l armv7l GNU/Linux
I have observed after every reboot, my board starts with some older time and my application running in user-space corrects the system time from a timestamp saved on a file (which is updated every hour). Application also connects with some server to get latest time. I have noticed that if system date and time is less than date and time which it had at last shutdown, then syslog doesn't get any logs from my applications till it has a time atleast greater than date and time which it had at last shutdown.
Please help me in debugging the issue and fixing it.
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
On 04.02.20 13:59, Abhi Arora wrote:
Continuing my previous email....
Can you share your configuration, please? I have shared over my last email
well, gmail does not have a good interface to mailing list. (html mail with very bad plaintext conversion.
Are you using system() source? I didn't get you. Please elaborate. You mean source code system() library function. If that you mean, then no we don't use it.
However I saw there: source s_src { systemd_journal(); internal(); file("/proc/kmsg" program_override("kernel")); }; No, you don't use system() source, it looks like: source s_src { system(); ... }; -- Matus UHLAR - fantomas, uhlar@fantomas.sk ; http://www.fantomas.sk/ Warning: I wish NOT to receive e-mail advertising to this address. Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu. 2B|!2B, that's a question!
But you are using journal source, so it might be related to that. I am not sure weather you rely on journald or not, but as a workaround you could just use a unix-dgram() source and bypass journald by making sure /dev/log points to syslog-ng. Journald based logging is pretty slow and unless you have a usecase for it, it might be easier to bypass it completely. Makes the local logging path much simpler. On Tue, Feb 4, 2020, 13:23 Matus UHLAR - fantomas <uhlar@fantomas.sk> wrote:
On 04.02.20 13:59, Abhi Arora wrote:
Continuing my previous email....
Can you share your configuration, please? I have shared over my last email
well, gmail does not have a good interface to mailing list. (html mail with very bad plaintext conversion.
Are you using system() source? I didn't get you. Please elaborate. You mean source code system() library function. If that you mean, then no we don't use it.
However I saw there:
source s_src { systemd_journal(); internal(); file("/proc/kmsg" program_override("kernel")); };
No, you don't use system() source, it looks like:
source s_src { system(); ... };
-- Matus UHLAR - fantomas, uhlar@fantomas.sk ; http://www.fantomas.sk/ Warning: I wish NOT to receive e-mail advertising to this address. Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu. 2B|!2B, that's a question!
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Hi, Is there anything I can to debug it? Any pointer to debug it and find the root cause? Should I modify my syslog conf file to source s_src { unix-dgram(); internal(); file("/proc/kmsg" program_override("kernel")); }; ? Can you help me more with "bypass journald by making sure /dev/log points to syslog-ng."? On Wed, Feb 5, 2020 at 1:57 PM Balazs Scheidler <bazsi77@gmail.com> wrote:
But you are using journal source, so it might be related to that.
I am not sure weather you rely on journald or not, but as a workaround you could just use a unix-dgram() source and bypass journald by making sure /dev/log points to syslog-ng.
Journald based logging is pretty slow and unless you have a usecase for it, it might be easier to bypass it completely. Makes the local logging path much simpler.
On Tue, Feb 4, 2020, 13:23 Matus UHLAR - fantomas <uhlar@fantomas.sk> wrote:
On 04.02.20 13:59, Abhi Arora wrote:
Continuing my previous email....
Can you share your configuration, please? I have shared over my last email
well, gmail does not have a good interface to mailing list. (html mail with very bad plaintext conversion.
Are you using system() source? I didn't get you. Please elaborate. You mean source code system() library function. If that you mean, then no we don't use it.
However I saw there:
source s_src { systemd_journal(); internal(); file("/proc/kmsg" program_override("kernel")); };
No, you don't use system() source, it looks like:
source s_src { system(); ... };
-- Matus UHLAR - fantomas, uhlar@fantomas.sk ; http://www.fantomas.sk/ Warning: I wish NOT to receive e-mail advertising to this address. Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu. 2B|!2B, that's a question!
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
I tried checking journctld logs. Running the following command returned: journalctl Feb 04 12:42:57 f1 systemd[1]: Starting System Logger Daemon "scl" instance... Feb 04 12:42:57 f1 syslog-ng[9777]: [2020-02-04T12:42:57.093710] Error opening control socket, bind() failed; socket='/var/run/syslog-ng/syslog-ng.ctl', erro r='No such file or directory (2)' Feb 04 12:42:57 f1 syslog-ng[9777]: [2020-02-04T12:42:57.098856] Error opening configuration file; filename='--control', error='No such file or directory (2) ' Feb 04 12:42:57 f1 systemd[1]: [[0;1;39m[[0;1;31m[[0;1;39msyslog-ng@scl.service: Main process exited, code=exited, status=1/FAILURE[[0m Feb 04 12:42:57 f1 systemd[1]: [[0;1;39m[[0;1;31m[[0;1;39msyslog-ng@scl.service: Failed with result 'exit-code'.[[0m Feb 04 12:42:57 f1 systemd[1]: [[0;1;31m[[0;1;39m[[0;1;31mFailed to start System Logger Daemon "scl" instance.[[0m : On Wed, Feb 5, 2020 at 9:28 PM Abhi Arora <engr.abhiarora@gmail.com> wrote:
Hi, Is there anything I can to debug it? Any pointer to debug it and find the root cause? Should I modify my syslog conf file to
source s_src { unix-dgram(); internal(); file("/proc/kmsg" program_override("kernel")); }; ? Can you help me more with "bypass journald by making sure /dev/log points to syslog-ng."?
On Wed, Feb 5, 2020 at 1:57 PM Balazs Scheidler <bazsi77@gmail.com> wrote:
But you are using journal source, so it might be related to that.
I am not sure weather you rely on journald or not, but as a workaround you could just use a unix-dgram() source and bypass journald by making sure /dev/log points to syslog-ng.
Journald based logging is pretty slow and unless you have a usecase for it, it might be easier to bypass it completely. Makes the local logging path much simpler.
On Tue, Feb 4, 2020, 13:23 Matus UHLAR - fantomas <uhlar@fantomas.sk> wrote:
On 04.02.20 13:59, Abhi Arora wrote:
Continuing my previous email....
Can you share your configuration, please? I have shared over my last email
well, gmail does not have a good interface to mailing list. (html mail with very bad plaintext conversion.
Are you using system() source? I didn't get you. Please elaborate. You mean source code system() library function. If that you mean, then no we don't use it.
However I saw there:
source s_src { systemd_journal(); internal(); file("/proc/kmsg" program_override("kernel")); };
No, you don't use system() source, it looks like:
source s_src { system(); ... };
-- Matus UHLAR - fantomas, uhlar@fantomas.sk ; http://www.fantomas.sk/ Warning: I wish NOT to receive e-mail advertising to this address. Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu. 2B|!2B, that's a question!
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Looks like putting the following was causing some trouble: source s_src { unix-dgram(); internal(); file("/proc/kmsg" program_override("kernel")); }; I tired running "journalctl" and it has all the logs. However, /var/log/syslog doesn't have the logs if date is older than the date on last shutdown. Any help? On Thu, Feb 6, 2020 at 11:46 AM Abhi Arora <engr.abhiarora@gmail.com> wrote:
I tried checking journctld logs.
Running the following command returned: journalctl
Feb 04 12:42:57 f1 systemd[1]: Starting System Logger Daemon "scl" instance... Feb 04 12:42:57 f1 syslog-ng[9777]: [2020-02-04T12:42:57.093710] Error opening control socket, bind() failed; socket='/var/run/syslog-ng/syslog-ng.ctl', erro r='No such file or directory (2)' Feb 04 12:42:57 f1 syslog-ng[9777]: [2020-02-04T12:42:57.098856] Error opening configuration file; filename='--control', error='No such file or directory (2) ' Feb 04 12:42:57 f1 systemd[1]: [[0;1;39m[[0;1;31m[[0;1;39msyslog-ng@scl.service: Main process exited, code=exited, status=1/FAILURE[[0m Feb 04 12:42:57 f1 systemd[1]: [[0;1;39m[[0;1;31m[[0;1;39msyslog-ng@scl.service: Failed with result 'exit-code'.[[0m Feb 04 12:42:57 f1 systemd[1]: [[0;1;31m[[0;1;39m[[0;1;31mFailed to start System Logger Daemon "scl" instance.[[0m :
On Wed, Feb 5, 2020 at 9:28 PM Abhi Arora <engr.abhiarora@gmail.com> wrote:
Hi, Is there anything I can to debug it? Any pointer to debug it and find the root cause? Should I modify my syslog conf file to
source s_src { unix-dgram(); internal(); file("/proc/kmsg" program_override("kernel")); }; ? Can you help me more with "bypass journald by making sure /dev/log points to syslog-ng."?
On Wed, Feb 5, 2020 at 1:57 PM Balazs Scheidler <bazsi77@gmail.com> wrote:
But you are using journal source, so it might be related to that.
I am not sure weather you rely on journald or not, but as a workaround you could just use a unix-dgram() source and bypass journald by making sure /dev/log points to syslog-ng.
Journald based logging is pretty slow and unless you have a usecase for it, it might be easier to bypass it completely. Makes the local logging path much simpler.
On Tue, Feb 4, 2020, 13:23 Matus UHLAR - fantomas <uhlar@fantomas.sk> wrote:
On 04.02.20 13:59, Abhi Arora wrote:
Continuing my previous email....
Can you share your configuration, please? I have shared over my last email
well, gmail does not have a good interface to mailing list. (html mail with very bad plaintext conversion.
Are you using system() source? I didn't get you. Please elaborate. You mean source code system() library function. If that you mean, then no we don't use it.
However I saw there:
source s_src { systemd_journal(); internal(); file("/proc/kmsg" program_override("kernel")); };
No, you don't use system() source, it looks like:
source s_src { system(); ... };
-- Matus UHLAR - fantomas, uhlar@fantomas.sk ; http://www.fantomas.sk/ Warning: I wish NOT to receive e-mail advertising to this address. Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu. 2B|!2B, that's a question!
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
I would to correct myself. The journalctl has old logs. I thought they are latest logs. So journalctl isn't logging the log if date is older than the date on last shutdown. On Thu, Feb 6, 2020 at 11:55 AM Abhi Arora <engr.abhiarora@gmail.com> wrote:
Looks like putting the following was causing some trouble:
source s_src { unix-dgram(); internal(); file("/proc/kmsg" program_override("kernel")); };
I tired running "journalctl" and it has all the logs. However, /var/log/syslog doesn't have the logs if date is older than the date on last shutdown. Any help?
On Thu, Feb 6, 2020 at 11:46 AM Abhi Arora <engr.abhiarora@gmail.com> wrote:
I tried checking journctld logs.
Running the following command returned: journalctl
Feb 04 12:42:57 f1 systemd[1]: Starting System Logger Daemon "scl" instance... Feb 04 12:42:57 f1 syslog-ng[9777]: [2020-02-04T12:42:57.093710] Error opening control socket, bind() failed; socket='/var/run/syslog-ng/syslog-ng.ctl', erro r='No such file or directory (2)' Feb 04 12:42:57 f1 syslog-ng[9777]: [2020-02-04T12:42:57.098856] Error opening configuration file; filename='--control', error='No such file or directory (2) ' Feb 04 12:42:57 f1 systemd[1]: [[0;1;39m[[0;1;31m[[0;1;39msyslog-ng@scl.service: Main process exited, code=exited, status=1/FAILURE[[0m Feb 04 12:42:57 f1 systemd[1]: [[0;1;39m[[0;1;31m[[0;1;39msyslog-ng@scl.service: Failed with result 'exit-code'.[[0m Feb 04 12:42:57 f1 systemd[1]: [[0;1;31m[[0;1;39m[[0;1;31mFailed to start System Logger Daemon "scl" instance.[[0m :
On Wed, Feb 5, 2020 at 9:28 PM Abhi Arora <engr.abhiarora@gmail.com> wrote:
Hi, Is there anything I can to debug it? Any pointer to debug it and find the root cause? Should I modify my syslog conf file to
source s_src { unix-dgram(); internal(); file("/proc/kmsg" program_override("kernel")); }; ? Can you help me more with "bypass journald by making sure /dev/log points to syslog-ng."?
On Wed, Feb 5, 2020 at 1:57 PM Balazs Scheidler <bazsi77@gmail.com> wrote:
But you are using journal source, so it might be related to that.
I am not sure weather you rely on journald or not, but as a workaround you could just use a unix-dgram() source and bypass journald by making sure /dev/log points to syslog-ng.
Journald based logging is pretty slow and unless you have a usecase for it, it might be easier to bypass it completely. Makes the local logging path much simpler.
On Tue, Feb 4, 2020, 13:23 Matus UHLAR - fantomas <uhlar@fantomas.sk> wrote:
On 04.02.20 13:59, Abhi Arora wrote:
Continuing my previous email.... > Can you share your configuration, please? I have shared over my last email
well, gmail does not have a good interface to mailing list. (html mail with very bad plaintext conversion.
> Are you using system() source? I didn't get you. Please elaborate. You mean source code system() library function. If that you mean, then no we don't use it.
However I saw there:
source s_src { systemd_journal(); internal(); file("/proc/kmsg" program_override("kernel")); };
No, you don't use system() source, it looks like:
source s_src { system(); ... };
-- Matus UHLAR - fantomas, uhlar@fantomas.sk ; http://www.fantomas.sk/ Warning: I wish NOT to receive e-mail advertising to this address. Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu. 2B|!2B, that's a question!
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
I think you need to add /dev/log to unix-dgram. source s_src { unix-dgram("/dev/log"); internal(); file("/proc/kmsg" program_override("kernel")); }; Regards, Gábor
Thanks Tired that. I don't see service start fail messages. However, even with the latest date, syslog doesn't show any logs from my applications. However, journalctl is showing the logs after a latest date update. source s_src { unix-dgram("/dev/log"); internal(); file("/proc/kmsg" program_override("kernel")); }; On Thu, Feb 6, 2020 at 12:21 PM Nagy Gábor <gabor.hl@gmail.com> wrote:
I think you need to add /dev/log to unix-dgram.
source s_src { unix-dgram("/dev/log"); internal(); file("/proc/kmsg" program_override("kernel")); };
Regards, Gábor
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
The following command "cat /dev/log" returns:
cat: /dev/log: No such device or address
Do I need to configure something else for syslog? On Thu, Feb 6, 2020 at 12:30 PM Abhi Arora <engr.abhiarora@gmail.com> wrote:
Thanks Tired that. I don't see service start fail messages. However, even with the latest date, syslog doesn't show any logs from my applications. However, journalctl is showing the logs after a latest date update.
source s_src { unix-dgram("/dev/log"); internal(); file("/proc/kmsg" program_override("kernel")); };
On Thu, Feb 6, 2020 at 12:21 PM Nagy Gábor <gabor.hl@gmail.com> wrote:
I think you need to add /dev/log to unix-dgram.
source s_src { unix-dgram("/dev/log"); internal(); file("/proc/kmsg" program_override("kernel")); };
Regards, Gábor
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
On 06.02.20 12:30, Abhi Arora wrote:
I don't see service start fail messages. However, even with the latest date, syslog doesn't show any logs from my applications. However, journalctl is showing the logs after a latest date update.
source s_src { unix-dgram("/dev/log"); internal(); file("/proc/kmsg" program_override("kernel")); };
try "ls -l /dev/log" in this case: lrwxrwxrwx 1 root root 28 apr 14 2018 /dev/log -> /run/systemd/journal/dev-log is the log redirected to journald and in this case: srw-rw-rw- 1 root root 0 Dec 16 06:54 /dev/log you can verify it's used by syslog-ng: # lsof /dev/log COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME syslog-ng 1747 root 14u unix 0x00000000364c47ad 0t0 1544 /dev/log type=DGRAM
On Thu, Feb 6, 2020 at 12:21 PM Nagy Gábor <gabor.hl@gmail.com> wrote:
I think you need to add /dev/log to unix-dgram.
source s_src { unix-dgram("/dev/log"); internal(); file("/proc/kmsg" program_override("kernel")); };
Regards, Gábor
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
-- Matus UHLAR - fantomas, uhlar@fantomas.sk ; http://www.fantomas.sk/ Warning: I wish NOT to receive e-mail advertising to this address. Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu. "Where do you want to go to die?" [Microsoft]
"ls -l /dev/log" returns:
lrwxrwxrwx 1 root root 28 Sep 30 13:28 /dev/log -> /run/systemd/journal/dev-log
"lsof" returns a huge list of open files. I am putting few related to sysnlog
363 /usr/sbin/syslog-ng /dev/null 363 /usr/sbin/syslog-ng socket:[6284] 363 /usr/sbin/syslog-ng socket:[6284] 363 /usr/sbin/syslog-ng anon_inode:[eventpoll] 363 /usr/sbin/syslog-ng anon_inode:[eventfd] 363 /usr/sbin/syslog-ng socket:[6522] 363 /usr/sbin/syslog-ng anon_inode:[eventfd] 363 /usr/sbin/syslog-ng anon_inode:[eventfd] 363 /usr/sbin/syslog-ng anon_inode:[eventfd] 363 /usr/sbin/syslog-ng anon_inode:[eventfd] 363 /usr/sbin/syslog-ng anon_inode:[eventfd] 363 /usr/sbin/syslog-ng /var/lib/syslog-ng/syslog-ng.persist 363 /usr/sbin/syslog-ng socket:[6537] 363 /usr/sbin/syslog-ng /proc/kmsg 363 /usr/sbin/syslog-ng anon_inode:[timerfd]
On Thu, Feb 6, 2020 at 2:39 PM Matus UHLAR - fantomas <uhlar@fantomas.sk> wrote:
On 06.02.20 12:30, Abhi Arora wrote:
I don't see service start fail messages. However, even with the latest date, syslog doesn't show any logs from my applications. However, journalctl is showing the logs after a latest date update.
source s_src { unix-dgram("/dev/log"); internal(); file("/proc/kmsg" program_override("kernel")); };
try "ls -l /dev/log" in this case:
lrwxrwxrwx 1 root root 28 apr 14 2018 /dev/log -> /run/systemd/journal/dev-log
is the log redirected to journald and in this case:
srw-rw-rw- 1 root root 0 Dec 16 06:54 /dev/log
you can verify it's used by syslog-ng:
# lsof /dev/log COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME syslog-ng 1747 root 14u unix 0x00000000364c47ad 0t0 1544 /dev/log type=DGRAM
On Thu, Feb 6, 2020 at 12:21 PM Nagy Gábor <gabor.hl@gmail.com> wrote:
I think you need to add /dev/log to unix-dgram.
source s_src { unix-dgram("/dev/log"); internal(); file("/proc/kmsg" program_override("kernel")); };
Regards, Gábor
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
-- Matus UHLAR - fantomas, uhlar@fantomas.sk ; http://www.fantomas.sk/ Warning: I wish NOT to receive e-mail advertising to this address. Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu. "Where do you want to go to die?" [Microsoft]
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Hello, Any help? If it is available online, please help me with a link as I am unable to find anything useful over internet. On Thu, Feb 6, 2020 at 3:04 PM Abhi Arora <engr.abhiarora@gmail.com> wrote:
"ls -l /dev/log" returns:
lrwxrwxrwx 1 root root 28 Sep 30 13:28 /dev/log -> /run/systemd/journal/dev-log
"lsof" returns a huge list of open files. I am putting few related to sysnlog
363 /usr/sbin/syslog-ng /dev/null 363 /usr/sbin/syslog-ng socket:[6284] 363 /usr/sbin/syslog-ng socket:[6284] 363 /usr/sbin/syslog-ng anon_inode:[eventpoll] 363 /usr/sbin/syslog-ng anon_inode:[eventfd] 363 /usr/sbin/syslog-ng socket:[6522] 363 /usr/sbin/syslog-ng anon_inode:[eventfd] 363 /usr/sbin/syslog-ng anon_inode:[eventfd] 363 /usr/sbin/syslog-ng anon_inode:[eventfd] 363 /usr/sbin/syslog-ng anon_inode:[eventfd] 363 /usr/sbin/syslog-ng anon_inode:[eventfd] 363 /usr/sbin/syslog-ng /var/lib/syslog-ng/syslog-ng.persist 363 /usr/sbin/syslog-ng socket:[6537] 363 /usr/sbin/syslog-ng /proc/kmsg 363 /usr/sbin/syslog-ng anon_inode:[timerfd]
On Thu, Feb 6, 2020 at 2:39 PM Matus UHLAR - fantomas <uhlar@fantomas.sk> wrote:
On 06.02.20 12:30, Abhi Arora wrote:
I don't see service start fail messages. However, even with the latest date, syslog doesn't show any logs from my applications. However, journalctl is showing the logs after a latest date update.
source s_src { unix-dgram("/dev/log"); internal(); file("/proc/kmsg" program_override("kernel")); };
try "ls -l /dev/log" in this case:
lrwxrwxrwx 1 root root 28 apr 14 2018 /dev/log -> /run/systemd/journal/dev-log
is the log redirected to journald and in this case:
srw-rw-rw- 1 root root 0 Dec 16 06:54 /dev/log
you can verify it's used by syslog-ng:
# lsof /dev/log COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME syslog-ng 1747 root 14u unix 0x00000000364c47ad 0t0 1544 /dev/log type=DGRAM
On Thu, Feb 6, 2020 at 12:21 PM Nagy Gábor <gabor.hl@gmail.com> wrote:
I think you need to add /dev/log to unix-dgram.
source s_src { unix-dgram("/dev/log"); internal(); file("/proc/kmsg" program_override("kernel")); };
Regards, Gábor
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
-- Matus UHLAR - fantomas, uhlar@fantomas.sk ; http://www.fantomas.sk/ Warning: I wish NOT to receive e-mail advertising to this address. Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu. "Where do you want to go to die?" [Microsoft]
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
participants (4)
-
Abhi Arora
-
Balazs Scheidler
-
Matus UHLAR - fantomas
-
Nagy Gábor