<div dir="ltr"><div>Ok, I've now read the relevant code to see how the journald polling operates and now have a general understanding.<br></div><div><br></div>Couple of questions and notes.<br><div><br></div><div>1) which version of of systemd do you have? (Debian buster currently has 247.3-6~bpo10+1), is that right?</div><div>2) syslog-ng 3.36.1, right?</div><div>3) you are using flow-control + disk buffer</div><div>4) your system seems to be lightly loaded, you seem to have had ~30 messages/sec in the journal between stalling and recovering via SIGHUP (based on the "processed" counts and the last timestamp)</div><div><br></div><div><div>syslog-ng uses the libsystemd library to poll for changes, which in
turn uses inotify to watch the directories and files that make up the
journal. You should see the inotify fd in the lsof output. I have mine
like this:</div><div><br></div><div>syslog-ng 3233 root 11r a_inode 0,14 0 10439 inotify</div><div><br></div><div>You should also have a couple of "eventpoll" fd's open, in lsof output this should look like this:</div><div><br></div><div>syslog-ng 3233 root 3u a_inode 0,14 0 10439 [eventpoll]<br>syslog-ng 3233 root 13u a_inode 0,14 0 10439 [eventpoll]<br></div><div><br></div><div>As you can see, I have two of these eventpoll fds, #3 is the main one, usually that comes first.</div><div><br></div><div>If you do an strace on the syslog-ng process (command was "strace -s 256 -f -p <syslogng-pid> -o aaa"), you can see the epoll being manipulated:</div><div><br></div><div># we are grepping for all epoll operations with fd #3<br></div></div><div>$ grep 'epoll.*(3' aaa<br>3233 07:12:55.542368 epoll_wait(3, [{EPOLLIN, {u32=1319656488, u64=94426175659048}}], 10, 921) = 1 <0.462260><br>3233 07:12:56.005163 epoll_ctl(3, EPOLL_CTL_DEL, 11, 0x7ffecc73736c) = 0 <0.000007><br>3233 07:12:56.005265 epoll_wait(3, <unfinished ...><br>625546 07:12:56.005555 epoll_ctl(3, EPOLL_CTL_MOD, 4, {EPOLLIN|EPOLLONESHOT, {u32=1319484192, u64=94426175486752}}) = 0 <0.000008><br>3233 07:12:56.005778 epoll_ctl(3, EPOLL_CTL_ADD, 11, {EPOLLIN, {u32=1319656488, u64=94426175659048}} <unfinished ...><br>3233 07:12:56.005864 epoll_wait(3, <unfinished ...><br>625546 07:12:56.006049 epoll_ctl(3, EPOLL_CTL_MOD, 4, {EPOLLIN|EPOLLONESHOT, {u32=1319484192, u64=94426175486752}}) = 0 <0.000006><br>3233 07:12:56.006114 epoll_wait(3, [{EPOLLIN, {u32=1319656488, u64=94426175659048}}], 10, -1) = 1 <0.250790><br>3233 07:12:56.257450 epoll_ctl(3, EPOLL_CTL_DEL, 11, 0x7ffecc73736c) = 0 <0.000009><br>3233 07:12:56.257518 epoll_wait(3, <unfinished ...><br>625546 07:12:56.257828 epoll_ctl(3, EPOLL_CTL_MOD, 4, {EPOLLIN|EPOLLONESHOT, {u32=1319484192, u64=94426175486752}}) = 0 <0.000021><br>3233 07:12:56.258144 epoll_ctl(3, EPOLL_CTL_ADD, 11, {EPOLLIN, {u32=1319656488, u64=94426175659048}} <unfinished ...><br>3233 07:12:56.258250 epoll_wait(3, <unfinished ...><br>625546 07:12:56.258411 epoll_ctl(3, EPOLL_CTL_MOD, 4, {EPOLLIN|EPOLLONESHOT, {u32=1319484192, u64=94426175486752}}) = 0 <0.000013><br>3233 07:12:56.258496 epoll_wait(3, [{EPOLLIN, {u32=1319484552, u64=94426175487112}}], 10, -1) = 1 <0.203908><br>3233 07:12:56.462711 epoll_wait(3, [{EPOLLIN, {u32=1319656488, u64=94426175659048}}], 10, 1000) = 1 <0.123872><br>3233 07:12:56.587406 epoll_ctl(3, EPOLL_CTL_DEL, 11, 0x7ffecc73736c) = 0 <0.000015><br>3233 07:12:56.587542 epoll_wait(3, <unfinished ...></div><div><br></div><div>From those events, you can see that syslog-ng deletes and adds fd #11 (which is my inotify descriptor) and fd #4 (which is an event). If I zoom in to fd 11 in addition to epoll:</div><div></div><div><br></div><div>$ egrep '(\(11)|(epoll.*\(3)' aaa</div><div><br></div><div># first event and its reaction:<br></div><div>3233 07:12:55.542368 epoll_wait(3, [{EPOLLIN, {u32=1319656488, u64=94426175659048}}], 10, 921) = 1 <0.462260><br>3233 07:12:56.004800 read(11, "\3\0\0\0\2\0\0\0\0\0\0\0\20\0\0\0system.journal\0\0", 272) = 32 <0.000019><br>3233 07:12:56.005090 read(11, 0x7ffecc7372a0, 272) = -1 EAGAIN (Resource temporarily unavailable) <0.000006><br>3233 07:12:56.005163 epoll_ctl(3, EPOLL_CTL_DEL, 11, 0x7ffecc73736c) = 0 <0.000007><br>3233 07:12:56.005265 epoll_wait(3, <unfinished ...><br>625546 07:12:56.005555 epoll_ctl(3, EPOLL_CTL_MOD, 4, {EPOLLIN|EPOLLONESHOT, {u32=1319484192, u64=94426175486752}}) = 0 <0.000008><br>3233 07:12:56.005595 fcntl(11, F_GETFD) = 0x1 (flags FD_CLOEXEC) <0.000007><br>3233 07:12:56.005618 fcntl(11, F_GETFL) = 0x800 (flags O_RDONLY|O_NONBLOCK) <0.000006><br>3233 07:12:56.005659 setsockopt(11, SOL_SOCKET, SO_OOBINLINE, [1], 4) = -1 ENOTSOCK (Socket operation on non-socket) <0.000007><br>3233 07:12:56.005778 epoll_ctl(3, EPOLL_CTL_ADD, 11, {EPOLLIN, {u32=1319656488, u64=94426175659048}} <unfinished ...><br>3233 07:12:56.005864 epoll_wait(3, <unfinished ...></div><div><br></div><div><div># second event and its reaction:<br></div><div></div></div><div>625546 07:12:56.006049 epoll_ctl(3, EPOLL_CTL_MOD, 4, {EPOLLIN|EPOLLONESHOT, {u32=1319484192, u64=94426175486752}}) = 0 <0.000006><br>3233 07:12:56.006114 epoll_wait(3, [{EPOLLIN, {u32=1319656488, u64=94426175659048}}], 10, -1) = 1 <0.250790><br>3233 07:12:56.256994 read(11, "\3\0\0\0\2\0\0\0\0\0\0\0\20\0\0\0system.journal\0\0", 272) = 32 <0.000026><br>3233 07:12:56.257416 read(11, 0x7ffecc7372a0, 272) = -1 EAGAIN (Resource temporarily unavailable) <0.000008><br>3233 07:12:56.257450 epoll_ctl(3, EPOLL_CTL_DEL, 11, 0x7ffecc73736c) = 0 <0.000009><br>3233 07:12:56.257518 epoll_wait(3, <unfinished ...><br>625546 07:12:56.257828 epoll_ctl(3, EPOLL_CTL_MOD, 4, {EPOLLIN|EPOLLONESHOT, {u32=1319484192, u64=94426175486752}}) = 0 <0.000021><br>3233 07:12:56.257940 fcntl(11, F_GETFD) = 0x1 (flags FD_CLOEXEC) <0.000011><br>3233 07:12:56.257978 fcntl(11, F_GETFL) = 0x800 (flags O_RDONLY|O_NONBLOCK) <0.000009><br>3233 07:12:56.258017 setsockopt(11, SOL_SOCKET, SO_OOBINLINE, [1], 4) = -1 ENOTSOCK (Socket operation on non-socket) <0.000011><br>3233 07:12:56.258144 epoll_ctl(3, EPOLL_CTL_ADD, 11, {EPOLLIN, {u32=1319656488, u64=94426175659048}} <unfinished ...><br>3233 07:12:56.258250 epoll_wait(3, <unfinished ...><br></div><div><br></div><div>please notice that syslog-ng reads fd #11 right after epoll indicates there's an event available. The read() has a payload which can be decoded according to struct inotify_event.</div><div><br></div><div>In both cases, after the thread "625546" starts executing, we use the journald APIs to fetch the message. It's not v ery visible from the strace as sdjournal seems to access the files via mmap or something, but you should definitely see the disk buffer being written as messages get delivered, since you are using a reliable disk buffer (which you may not need btw, but that's a separate discussion).<br></div><div><br></div><div>The reasons we might stall could be:</div><div><ul><li>the file is not modified at all (this could be ruled out)</li><li>we don't poll the inotify fd (you can confirm this via strace as above)</li><li>we do poll the fd, but we don't read the events from it (again strace helps here)</li><li>we do poll fd fd and read the events but somehow we don't respond to it properly (this is probably a bug)</li><li>our worker wakes up in response to the event but then it doesn't consume messages from the journal (this is probably a bug)</li></ul><div>If we are not polling the inotify fd, that means that the systemd-journal() source might be suspended. This can happen for example because of flow control:</div><div><ul><li>a source is suspended if some of its flow-controlled destinations are unable to consume the messages it produces. in your case, you have a disk buffer and that consumes all messages up to its capacity. I didn't see disk buffers that were full (the queued counters are all zero), you were also saying that those same files are being written, messages delivered normally.<br></li><li>if for some reason we are stuck in the "suspended" state and the wakeup mechanism does not work (this is probably a bug)</li></ul></div></div><div><br></div><div>With all this said, can you please confirm that we are polling the inotify fd from our main thread and that you can see the inotify event being consumed? Or you don't see that all.</div><div><br></div><div>Of course while syslog-ng is working, this should operate normally. The trick is to look at it when it is stalled.</div><div><br></div><div>Thanks and again sorry for this to take so long.<br></div><div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Nov 1, 2022 at 6:36 AM Balazs Scheidler <<a href="mailto:bazsi77@gmail.com">bazsi77@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>sorry, no I didn't. <br></div><div><br></div><div>I just looked at your configs, If I see correctly then in the error situation the local logs are stalled, counters are not incremented. The journald "processed" counter stays at 85558, while immediately after reload it goes to 271377, which seems to indicate that for some reason we are not getting updates on the journals.<br></div><div><br></div><div>I am reading the code to understand how we poll the journal and I hope I'll have some input how to debug this.</div><div><br></div><div>Bazsi<br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Oct 31, 2022 at 5:15 PM Alexandre Santos <<a href="mailto:ASantos@infinera.com" target="_blank">ASantos@infinera.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div>
<div lang="EN-US">
<div>
<p class="MsoNormal"><span style="color:rgb(112,48,160)">Hi,<u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:rgb(112,48,160)">Did you have a chance to analyze this?<u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:rgb(112,48,160)">Thanks & Regards,<u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:rgb(112,48,160)">Alex<u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:rgb(112,48,160)"><u></u> <u></u></span></p>
<div>
<div style="border-color:rgb(225,225,225) currentcolor currentcolor;border-style:solid none none;border-width:1pt medium medium;padding:3pt 0cm 0cm">
<p class="MsoNormal"><b>From:</b> syslog-ng <<a href="mailto:syslog-ng-bounces@lists.balabit.hu" target="_blank">syslog-ng-bounces@lists.balabit.hu</a>> <b>
On Behalf Of </b>Alexandre Santos<br>
<b>Sent:</b> 16 de setembro de 2022 14:38<br>
<b>To:</b> Syslog-ng users' and developers' mailing list <<a href="mailto:syslog-ng@lists.balabit.hu" target="_blank">syslog-ng@lists.balabit.hu</a>><br>
<b>Subject:</b> Re: [syslog-ng] Local sources seem not to be working<u></u><u></u></p>
</div>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">Hi Balazs,<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">Do you have any other idea of how to debug this?<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">From the last time I was able to reproduce the problem, I found the following:<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal"><i>New developments, on this.<u></u><u></u></i></p>
<p class="MsoNormal"><i>After another test this time with no remote destinations configured, the issue happened again:<u></u><u></u></i></p>
<p class="MsoNormal"><i>In attachment I am sending:<u></u><u></u></i></p>
<p class="MsoNormal"><i>- The stats in error condition: 42116.inerror.stats.txt, where ‘src.journald;s_src#0;journal;a;stamp;1651142246’<u></u><u></u></i></p>
<p class="MsoNormal"><i>- The stats in error condition after some time: 42116.inerror.15m.stats.txt where ‘src.journald;s_src#0;journal;a;stamp;1651142246’
<u></u><u></u></i></p>
<p class="MsoNormal"><i>- The stats after recovering the system with reload: 42116.after.stats.txt<u></u><u></u></i></p>
<p class="MsoNormal"><i>- The syslog-ng configuration: 42116.no.remote.dest.syslog-ng.conf<u></u><u></u></i></p>
<p class="MsoNormal"><i><u></u> <u></u></i></p>
<p class="MsoNormal"><i>It seems the ‘src.journald;s_src#0;journal;a;stamp;1651142246’did not changed. Does this means that last read timestamp from journal did not changed?<u></u><u></u></i></p>
<p class="MsoNormal"><i>Logs from the UDP source are still being written to the /var/log/… files.<u></u><u></u></i></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">Cheers,<u></u><u></u></p>
<p class="MsoNormal">Alex<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<div style="border-color:rgb(225,225,225) currentcolor currentcolor;border-style:solid none none;border-width:1pt medium medium;padding:3pt 0cm 0cm">
<p class="MsoNormal"><b>From:</b> syslog-ng <<a href="mailto:syslog-ng-bounces@lists.balabit.hu" target="_blank">syslog-ng-bounces@lists.balabit.hu</a>>
<b>On Behalf Of </b>Balazs Scheidler<br>
<b>Sent:</b> 1 de julho de 2022 21:02<br>
<b>To:</b> Syslog-ng users' and developers' mailing list <<a href="mailto:syslog-ng@lists.balabit.hu" target="_blank">syslog-ng@lists.balabit.hu</a>><br>
<b>Subject:</b> Re: [syslog-ng] Local sources seem not to be working<u></u><u></u></p>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<p class="MsoNormal">This means that we are indeed polling the journal ourselves and not rely on journald to forward the logs to us.<u></u><u></u></p>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">If the error occurs, try to look at the syslog-ng statistic counters (syslog-ng-ctl stats) to see if your output queue is full. This could cause back pressure and reading to stop.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Also, enabling debug/trace messages of syslog-ng could be helpful. You can also do that via syslog-ng-ctl, no restart is needed. The trace output should be excluded from your normal log processing pipeline, as it can be quite overwhelming
in volume.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">I don't really have any other ideas at the moment. Still on vacation. :)<u></u><u></u></p>
</div>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<div>
<p class="MsoNormal">On Thu, Jun 30, 2022, 19:43 Alexandre Santos <<a href="mailto:ASantos@infinera.com" target="_blank">ASantos@infinera.com</a>> wrote:<u></u><u></u></p>
</div>
<blockquote style="border-color:currentcolor currentcolor currentcolor rgb(204,204,204);border-style:none none none solid;border-width:medium medium medium 1pt;padding:0cm 0cm 0cm 6pt;margin:5pt 0cm 5pt 4.8pt">
<div>
<div>
<p class="MsoNormal">Hi Balazs,<u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal">Thanks for the feedback!<u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal">It is definitely using journald, as you can see below.<u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal">When you say “Try to remove the syslog-ng persist file and check if reading the journal restarts.”, this is to do when the system in the error condition, right?<u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal">Regards,<u></u><u></u></p>
<p class="MsoNormal">Alex<u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal"><b><span style="font-size:8pt;font-family:Consolas">source s_src {</span></b><u></u><u></u></p>
<p class="MsoNormal"><b><span style="font-size:8pt;font-family:Consolas"> </span></b><u></u><u></u></p>
<p class="MsoNormal"><b><span style="font-size:8pt;font-family:Consolas">#Start Block source generator system</span></b><u></u><u></u></p>
<p class="MsoNormal"><b><span style="font-size:8pt;font-family:Consolas">channel {</span></b><u></u><u></u></p>
<p class="MsoNormal"><b><span style="font-size:8pt;font-family:Consolas"> source {</span></b><u></u><u></u></p>
<p class="MsoNormal"><b><span style="font-size:8pt;font-family:Consolas">systemd-journal();</span></b><u></u><u></u></p>
<p class="MsoNormal"><b><span style="font-size:8pt;font-family:Consolas"> </span></b><u></u><u></u></p>
<p class="MsoNormal"><b><span style="font-size:8pt;font-family:Consolas"> }; # source</span></b><u></u><u></u></p>
<p class="MsoNormal"><b><span style="font-size:8pt;font-family:Consolas">channel {</span></b><u></u><u></u></p>
<p class="MsoNormal"><b><span style="font-size:8pt;font-family:Consolas"> channel {</span></b><u></u><u></u></p>
<p class="MsoNormal"><b><span style="font-size:8pt;font-family:Consolas"> parser {</span></b><u></u><u></u></p>
<p class="MsoNormal"><b><span style="font-size:8pt;font-family:Consolas"> </span></b><u></u><u></u></p>
<p class="MsoNormal"><b><span style="font-size:8pt;font-family:Consolas">#Start Block parser generator app-parser</span></b><u></u><u></u></p>
<p class="MsoNormal"><b><span style="font-size:8pt;font-family:Consolas"> </span></b><u></u><u></u></p>
<p class="MsoNormal"><b><span style="font-size:8pt;font-family:Consolas">channel {</span></b><u></u><u></u></p>
<p class="MsoNormal"><b><span style="font-size:8pt;font-family:Consolas"> junction {</span></b><u></u><u></u></p>
<p class="MsoNormal"><b><span style="font-size:8pt;font-family:Consolas"> </span></b><u></u><u></u></p>
<p class="MsoNormal"><b><span style="font-size:8pt;font-family:Consolas">channel { filter { tags('.app.doesnotexist'); }; flags(final); }; };</span></b><u></u><u></u></p>
<p class="MsoNormal"><b><span style="font-size:8pt;font-family:Consolas">}</span></b><u></u><u></u></p>
<p class="MsoNormal"><b><span style="font-size:8pt;font-family:Consolas">#End Block parser generator app-parser</span></b><u></u><u></u></p>
<p class="MsoNormal"><b><span style="font-size:8pt;font-family:Consolas">;</span></b><u></u><u></u></p>
<p class="MsoNormal"><b><span style="font-size:8pt;font-family:Consolas"> };</span></b><u></u><u></u></p>
<p class="MsoNormal"><b><span style="font-size:8pt;font-family:Consolas"> flags(final);</span></b><u></u><u></u></p>
<p class="MsoNormal"><b><span style="font-size:8pt;font-family:Consolas"> };</span></b><u></u><u></u></p>
<p class="MsoNormal"><b><span style="font-size:8pt;font-family:Consolas"> channel { flags(final); };</span></b><u></u><u></u></p>
<p class="MsoNormal"><b><span style="font-size:8pt;font-family:Consolas">};</span></b><u></u><u></u></p>
<p class="MsoNormal"><b><span style="font-size:8pt;font-family:Consolas">}; # channel</span></b><u></u><u></u></p>
<p class="MsoNormal"><b><span style="font-size:8pt;font-family:Consolas"> </span></b><u></u><u></u></p>
<p class="MsoNormal"><b><span style="font-size:8pt;font-family:Consolas">#End Block source generator system</span></b><u></u><u></u></p>
<p class="MsoNormal"><b><span style="font-size:8pt;font-family:Consolas">;</span></b><u></u><u></u></p>
<p class="MsoNormal"><b><span style="font-size:8pt;font-family:Consolas"> internal();</span></b><u></u><u></u></p>
<p class="MsoNormal"><b><span style="font-size:8pt;font-family:Consolas"> syslog(ip(19.88.4.17) transport("udp") port(514) keep-alive(no));</span></b><u></u><u></u></p>
<p class="MsoNormal"><b><span style="font-size:8pt;font-family:Consolas">};</span></b><u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<div style="border-color:rgb(225,225,225) currentcolor currentcolor;border-style:solid none none;border-width:1pt medium medium;padding:3pt 0cm 0cm">
<p class="MsoNormal"><b>From:</b> syslog-ng <<a href="mailto:syslog-ng-bounces@lists.balabit.hu" target="_blank">syslog-ng-bounces@lists.balabit.hu</a>>
<b>On Behalf Of </b>Balazs Scheidler<br>
<b>Sent:</b> 26 de junho de 2022 06:37<br>
<b>To:</b> Syslog-ng users' and developers' mailing list <<a href="mailto:syslog-ng@lists.balabit.hu" target="_blank">syslog-ng@lists.balabit.hu</a>><br>
<b>Subject:</b> Re: [syslog-ng] Local sources seem not to be working<u></u><u></u></p>
</div>
<p class="MsoNormal"> <u></u><u></u></p>
<div>
<p class="MsoNormal">Hi,<u></u><u></u></p>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">I haven't seen anything like this. We are reading the journal files using libsystemd.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Try to remove the syslog-ng persist file and check if reading the journal restarts.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Also there's can be two ways of local messages getting to syslog-ng,<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">1) /dev/log forwarding<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">2) reading the journal files<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">The first one is actively done by journald. Which one syslog-ng uses is automatically detected by our system() source.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">To see which one syslog-ng is trying to use, try to run it with --preprocess-into=some-file and check how system() source is expanded.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">I am unable to check the source code at the moment, so this is all from the top-of-my-head, but I hope this already helps to troubleshoot the issue.<u></u><u></u></p>
</div>
</div>
<p class="MsoNormal"> <u></u><u></u></p>
<div>
<div>
<p class="MsoNormal">On Fri, Jun 24, 2022, 18:21 Alexandre Santos <<a href="mailto:ASantos@infinera.com" target="_blank">ASantos@infinera.com</a>> wrote:<u></u><u></u></p>
</div>
<blockquote style="border-color:currentcolor currentcolor currentcolor rgb(204,204,204);border-style:none none none solid;border-width:medium medium medium 1pt;padding:0cm 0cm 0cm 6pt;margin:5pt 0cm 5pt 4.8pt">
<div>
<div>
<p class="MsoNormal">Hi<u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal">Any news regarding this issue?<u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal">Making a recap of the findings:<u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<ul type="disc">
<li>
Using a Debian 10 buster with first release with 3.36.1;<u></u><u></u></li></ul>
<ul type="disc">
<li>
After some time “system()” source logs are not getting written to the destinations;<u></u><u></u></li><li>
The log messages from other sources, internal() and syslog(…) continue to work fine, being written to the destinations;<u></u><u></u></li><li>
One the things I noticed is that the socket to the journal seems to vanish during the error situation:<u></u><u></u></li></ul>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal">It seems that somehow syslog-ng in unable to read from linux journal.<u></u><u></u></p>
<p class="MsoNormal"><b>Have you ever experienced this problem?</b><u></u><u></u></p>
<p class="MsoNormal"><b>Do know what can be wrong with the system?</b><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:8pt;font-family:Consolas"> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:8pt;font-family:Consolas"> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:8pt;font-family:Consolas">root@machine:~# lsof /run/log/journal/98101a328524447d88917bea845a8966/system*</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:8pt;font-family:Consolas">COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:8pt;font-family:Consolas">systemd-j 1723 root mem REG 0,19 8388608 31745 /run/log/journal/98101a328524447d88917bea845a8966/system.journal</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:8pt;font-family:Consolas">systemd-j 1723 root mem REG 0,19 8388608 26165
<a href="mailto:/run/log/journal/98101a328524447d88917bea845a8966/system@3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal" target="_blank">
/run/log/journal/98101a328524447d88917bea845a8966/system@3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal</a></span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:8pt;font-family:Consolas">systemd-j 1723 root 16u REG 0,19 8388608 26165
<a href="mailto:/run/log/journal/98101a328524447d88917bea845a8966/system@3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal" target="_blank">
/run/log/journal/98101a328524447d88917bea845a8966/system@3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal</a></span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:8pt;font-family:Consolas">systemd-j 1723 root 24u REG 0,19 8388608 31745 /run/log/journal/98101a328524447d88917bea845a8966/system.journal</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:8pt;font-family:Consolas">syslog-ng 3201 root mem REG 0,19 8388608 26165
<a href="mailto:/run/log/journal/98101a328524447d88917bea845a8966/system@3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal" target="_blank">
/run/log/journal/98101a328524447d88917bea845a8966/system@3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal</a></span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:8pt;font-family:Consolas">syslog-ng 3201 root mem REG 0,19 8388608 31745 /run/log/journal/98101a328524447d88917bea845a8966/system.journal</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:8pt;font-family:Consolas">syslog-ng 3201 root 14r REG 0,19 8388608 31745 /run/log/journal/98101a328524447d88917bea845a8966/system.journal</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:8pt;font-family:Consolas">syslog-ng 3201 root 15r REG 0,19 8388608 26165
<a href="mailto:/run/log/journal/98101a328524447d88917bea845a8966/system@3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal" target="_blank">
/run/log/journal/98101a328524447d88917bea845a8966/system@3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal</a></span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:8pt;font-family:Consolas">journalct 6861 root mem REG 0,19 8388608 26165
<a href="mailto:/run/log/journal/98101a328524447d88917bea845a8966/system@3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal" target="_blank">
/run/log/journal/98101a328524447d88917bea845a8966/system@3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal</a></span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:8pt;font-family:Consolas">journalct 6861 root mem REG 0,19 8388608 31745 /run/log/journal/98101a328524447d88917bea845a8966/system.journal</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:8pt;font-family:Consolas">journalct 6861 root 5r REG 0,19 8388608 31745 /run/log/journal/98101a328524447d88917bea845a8966/system.journal</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:8pt;font-family:Consolas">journalct 6861 root 6r REG 0,19 8388608 26165
<a href="mailto:/run/log/journal/98101a328524447d88917bea845a8966/system@3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal" target="_blank">
/run/log/journal/98101a328524447d88917bea845a8966/system@3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal</a></span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:8pt;font-family:Consolas">root@ machine:~# lsof /run/log/journal/98101a328524447d88917bea845a8966/system*</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:8pt;font-family:Consolas">COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:8pt;font-family:Consolas">systemd-j 1723 root mem REG 0,19 8388608 31745 /run/log/journal/98101a328524447d88917bea845a8966/system.journal</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:8pt;font-family:Consolas">systemd-j 1723 root mem REG 0,19 8388608 26165
<a href="mailto:/run/log/journal/98101a328524447d88917bea845a8966/system@3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal" target="_blank">
/run/log/journal/98101a328524447d88917bea845a8966/system@3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal</a></span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:8pt;font-family:Consolas">systemd-j 1723 root 16u REG 0,19 8388608 26165
<a href="mailto:/run/log/journal/98101a328524447d88917bea845a8966/system@3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal" target="_blank">
/run/log/journal/98101a328524447d88917bea845a8966/system@3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal</a></span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:8pt;font-family:Consolas">systemd-j 1723 root 24u REG 0,19 8388608 31745 /run/log/journal/98101a328524447d88917bea845a8966/system.journal</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:8pt;font-family:Consolas">journalct 6861 root mem REG 0,19 8388608 26165
<a href="mailto:/run/log/journal/98101a328524447d88917bea845a8966/system@3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal" target="_blank">
/run/log/journal/98101a328524447d88917bea845a8966/system@3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal</a></span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:8pt;font-family:Consolas">journalct 6861 root mem REG 0,19 8388608 31745 /run/log/journal/98101a328524447d88917bea845a8966/system.journal</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:8pt;font-family:Consolas">journalct 6861 root 5r REG 0,19 8388608 31745 /run/log/journal/98101a328524447d88917bea845a8966/system.journal</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:8pt;font-family:Consolas">journalct 6861 root 6r REG 0,19 8388608 26165
<a href="mailto:/run/log/journal/98101a328524447d88917bea845a8966/system@3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal" target="_blank">
/run/log/journal/98101a328524447d88917bea845a8966/system@3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal</a></span><u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal">Thanks in advance,<u></u><u></u></p>
<p class="MsoNormal">Alex<u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<div>
<div style="border-color:rgb(225,225,225) currentcolor currentcolor;border-style:solid none none;border-width:1pt medium medium;padding:3pt 0cm 0cm">
<p class="MsoNormal"><b>From:</b> syslog-ng <<a href="mailto:syslog-ng-bounces@lists.balabit.hu" target="_blank">syslog-ng-bounces@lists.balabit.hu</a>>
<b>On Behalf Of </b>Alexandre Santos<br>
<b>Sent:</b> 19 de maio de 2022 09:25<br>
<b>To:</b> Syslog-ng users' and developers' mailing list <<a href="mailto:syslog-ng@lists.balabit.hu" target="_blank">syslog-ng@lists.balabit.hu</a>><br>
<b>Subject:</b> Re: [syslog-ng] Local sources seem not to be working<u></u><u></u></p>
</div>
</div>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal">Hi Szilard,<u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal">There is no filter:<u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:8pt"> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:8pt;font-family:Consolas">source syslog_ng_src {</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:8pt;font-family:Consolas"> internal();</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:8pt;font-family:Consolas">};</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:8pt;font-family:Consolas"> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:8pt;font-family:Consolas">destination d_localfile_syslog_ng {</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:8pt;font-family:Consolas"> program("/opt/machine/local/bin/write_with_rotation.sh /var/log/syslog-ng-internal.log 10 10"</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:8pt;font-family:Consolas"> flags(syslog-protocol)</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:8pt;font-family:Consolas"> suppress(5)</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:8pt;font-family:Consolas"> disk-buffer(</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:8pt;font-family:Consolas"> mem-buf-size(2097152)</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:8pt;font-family:Consolas"> disk-buf-size(4194304)</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:8pt;font-family:Consolas"> reliable(yes)</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:8pt;font-family:Consolas"> dir("/tmp")</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:8pt;font-family:Consolas"> )</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:8pt;font-family:Consolas"> );</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:8pt;font-family:Consolas">};</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:8pt;font-family:Consolas">log {</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:8pt;font-family:Consolas"> source(syslog_ng_src);</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:8pt;font-family:Consolas"> destination(d_localfile_syslog_ng);</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:8pt;font-family:Consolas"> flags(flow-control);</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:8pt;font-family:Consolas">};</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:8pt;font-family:Consolas"> </span><u></u><u></u></p>
<p class="MsoNormal">Thanks and Regards,<u></u><u></u></p>
<p class="MsoNormal">Alex<u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<div>
<div style="border-color:rgb(225,225,225) currentcolor currentcolor;border-style:solid none none;border-width:1pt medium medium;padding:3pt 0cm 0cm">
<p class="MsoNormal"><b>From:</b> syslog-ng <<a href="mailto:syslog-ng-bounces@lists.balabit.hu" target="_blank">syslog-ng-bounces@lists.balabit.hu</a>>
<b>On Behalf Of </b>Szilard Parrag (sparrag)<br>
<b>Sent:</b> 19 de maio de 2022 08:59<br>
<b>To:</b> <a href="mailto:syslog-ng@lists.balabit.hu" target="_blank">syslog-ng@lists.balabit.hu</a><br>
<b>Subject:</b> Re: [syslog-ng] Local sources seem not to be working<u></u><u></u></p>
</div>
</div>
<p class="MsoNormal"> <u></u><u></u></p>
<div style="border:1pt solid rgb(156,101,0);padding:2pt">
<p class="MsoNormal" style="line-height:12pt;background:rgb(255,235,156)">
<b><span style="font-size:10pt;color:rgb(156,101,0)">CAUTION:</span></b><span style="font-size:10pt;color:black"> This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content
is safe.</span><u></u><u></u></p>
</div>
<p class="MsoNormal"> <u></u><u></u></p>
<div>
<div>
<p class="MsoNormal"><span style="font-size:12pt;color:black">Hi Alex,</span><u></u><u></u></p>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:12pt;color:black"> </span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12pt;color:black">We've checked it too and syslog-ng does not release the file descriptor of journald even with flow-control
<span style="background:white">enabled</span>.</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12pt;color:black"> </span><u></u><u></u></p>
</div>
<p class="MsoNormal"><span style="font-size:12pt;color:black">Also, your internal logs seem rather terse, maybe there is a filter which filters out the important parts. Could you please check it?</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12pt;color:black"> </span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12pt;color:black">Szilard</span><u></u><u></u></p>
</div>
</div>
</div>
</div>
<p class="MsoNormal" style="margin-bottom:12pt">______________________________________________________________________________<br>
Member info: <a href="https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.balabit.hu%2Fmailman%2Flistinfo%2Fsyslog-ng&data=05%7C01%7Casantos%40infinera.com%7C403ff07f627d49e0925708da97e8a5d5%7C285643de5f5b4b03a1530ae2dc8aaf77%7C1%7C1%7C637989322736795804%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000%7C%7C%7C&sdata=gA7r0xsABgH7TZFZp%2B3l9DE2mG%2B825zVlmpHf1VHv%2Bc%3D&reserved=0" target="_blank">
https://lists.balabit.hu/mailman/listinfo/syslog-ng</a><br>
Documentation: <a href="https://nam11.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com%2Fsupport%2Fdocumentation%2F%3Fproduct%3Dsyslog-ng&data=05%7C01%7Casantos%40infinera.com%7C403ff07f627d49e0925708da97e8a5d5%7C285643de5f5b4b03a1530ae2dc8aaf77%7C1%7C1%7C637989322736795804%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000%7C%7C%7C&sdata=a2TZ40wm6cSogr9U8Cohka%2FM%2FJS25mF2oAKN5UqcyZA%3D&reserved=0" target="_blank">
http://www.balabit.com/support/documentation/?product=syslog-ng</a><br>
FAQ: <a href="https://nam11.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com%2Fwiki%2Fsyslog-ng-faq&data=05%7C01%7Casantos%40infinera.com%7C403ff07f627d49e0925708da97e8a5d5%7C285643de5f5b4b03a1530ae2dc8aaf77%7C1%7C1%7C637989322736795804%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000%7C%7C%7C&sdata=SB1BSl3EuYVAtN3Phj9wMWSv5jwjN0zJ%2BAb0ArWNYkg%3D&reserved=0" target="_blank">
http://www.balabit.com/wiki/syslog-ng-faq</a><u></u><u></u></p>
</blockquote>
</div>
</div>
</div>
<p class="MsoNormal" style="margin-bottom:12pt">______________________________________________________________________________<br>
Member info: <a href="https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.balabit.hu%2Fmailman%2Flistinfo%2Fsyslog-ng&data=05%7C01%7Casantos%40infinera.com%7C403ff07f627d49e0925708da97e8a5d5%7C285643de5f5b4b03a1530ae2dc8aaf77%7C1%7C1%7C637989322736795804%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000%7C%7C%7C&sdata=gA7r0xsABgH7TZFZp%2B3l9DE2mG%2B825zVlmpHf1VHv%2Bc%3D&reserved=0" target="_blank">
https://lists.balabit.hu/mailman/listinfo/syslog-ng</a><br>
Documentation: <a href="https://nam11.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com%2Fsupport%2Fdocumentation%2F%3Fproduct%3Dsyslog-ng&data=05%7C01%7Casantos%40infinera.com%7C403ff07f627d49e0925708da97e8a5d5%7C285643de5f5b4b03a1530ae2dc8aaf77%7C1%7C1%7C637989322736952017%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000%7C%7C%7C&sdata=JG%2FfZBGTh6BuFq76QXUhXLpnrkxOVRhod9Cy3sRRhzU%3D&reserved=0" target="_blank">
http://www.balabit.com/support/documentation/?product=syslog-ng</a><br>
FAQ: <a href="https://nam11.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com%2Fwiki%2Fsyslog-ng-faq&data=05%7C01%7Casantos%40infinera.com%7C403ff07f627d49e0925708da97e8a5d5%7C285643de5f5b4b03a1530ae2dc8aaf77%7C1%7C1%7C637989322736952017%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000%7C%7C%7C&sdata=SldWSDsKiv7WXDjZMsYrUYiAMryqgnKX6zMqwVMN9C8%3D&reserved=0" target="_blank">
http://www.balabit.com/wiki/syslog-ng-faq</a><u></u><u></u></p>
</blockquote>
</div>
</div>
</div>
______________________________________________________________________________<br>
Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" rel="noreferrer" target="_blank">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a><br>
Documentation: <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" rel="noreferrer" target="_blank">http://www.balabit.com/support/documentation/?product=syslog-ng</a><br>
FAQ: <a href="http://www.balabit.com/wiki/syslog-ng-faq" rel="noreferrer" target="_blank">http://www.balabit.com/wiki/syslog-ng-faq</a><br>
<br>
</div></blockquote></div><br clear="all"><br>-- <br><div dir="ltr">Bazsi</div>
</blockquote></div><br clear="all"><br>-- <br><div dir="ltr" class="gmail_signature">Bazsi</div>