- syslog-ng - lists.balabit.hu

3.22.1: understanding the location of the python's part
by Sergey 14 Aug '19

14 Aug '19

Hello I see that in 3.22 has changed the location of the python's part (relative 3.20): architecture independent files moved to architecture specific directory (lib64 on x86_64). For example, rhel spec 3.20 (syslog-ng-3.20.1/packaging/rhel/syslog-ng.spec) %files python /usr/lib/python2.7/site-packages/syslogng-1.0-py2.7.egg-info %dir /usr/lib/python2.7/site-packages/syslogng/ %dir /usr/lib/python2.7/site-packages/syslogng/debuggercli/ /usr/lib/python2.7/site-packages/syslogng/* /usr/lib/python2.7/site-packages/syslogng/debuggercli/* %{_libdir}/%{name}/libmod-python.so The same in 3.22 %files python %{_libdir}/%{name}/python/syslogng-1.0-py%{py_ver}.egg-info %{_libdir}/%{name}/python/syslogng/* %{_libdir}/%{name}/libmod-python.so Is the architecture specific file libmod-python.so only? Or am I missing something? -- Regards, Sergey

2 1

Parse message fields for use as columns in MySQL
by Allen Olivas 09 Aug '19

09 Aug '19

Hello, I'm really scratching my head trying to make this work and thought maybe the community has experienced this before. I'm collecting logs from Wazuh and Syslog-NG. Those logs are sent from my Wazuh server with Sylog-NG configured to send to my MySQL server. The Syslog-ng.conf file on the MySQL server is configured with a destination to mysql. In the declared the destination and list out columns and values. # MySQL define destination destination d_mysql { sql( type(mysql) username("syslog") password("xxxxxxx") database("syslog") host("127.0.0.1") table("logs") columns("host", "id", "location", "facility", "rule", "priority", "level", "tag", "datetime", "program", "msg") values("$HOST", "$ID", "$LOCATION","$FACILITY", "$RULE", "$PRIORITY", "$LEVEL", "$TAG","$YEAR-$MONTH-$DAY $HOUR:$MIN:$SEC","$PROGRAM", "$MSG") indexes("datetime", "host", "id", "location", "rule") ); }; So here's the problem. The Message data contains information like Rule and Location that really equate to the Wazuh Rule and Location = the Wazuh Agent that's reporting it. I had hoped "location" column would populate with the Location date from the message. Same with Rule. SO my question to the community is how on earth do I parse the data in the message field to populate columns (existing or new)? Any thoughts, guidance, recommendations are greatly appreciated. Thanks, Allen Olivas InfoDefense Office: (972) 848-7910 Email: allen.olivas(a)infodefense.com Toll Free: (877) INFODEFENSE www.infodefense.com<https://l.shatrk.com/r/e/DblvLSPvKY2IxMPE?r=https://app.salesforceiq.com/r?…>

2 5

Syslog-ng stops writing until restarted
by Faine, Mark R. (MSFC-IS40)[NICS] 07 Aug '19

07 Aug '19

I have a strange problem where logs are definitely coming in over tcp and udp however they are not written, well not until I restart the server. Whenever I restart the logs are written but shortly after starting they stop until the server is restarted again. Any ideas on where I can look? Thanks, -Mark

2 2

syslog-ng configuration from CLI
by SZIGETVÁRI János 06 Aug '19

06 Aug '19

Hello Team, A few years back I heard about a project that would have added functionality to syslog-ng that would have enabled configurability from the command line. This was the respective PR: https://github.com/balabit/syslog-ng/pull/1326 Since then the PR was closed and part of the functionality provided by the PR has been added to syslog-ng (in https://github.com/balabit/syslog-ng/pull/1605) but not the CLI configurability part. I personally think that being able to configure syslog-ng through its command line arguments would bear a great potential. Of course running it with a complex configuration passed in through CLI argument would be infeasible, but with simple, single-purpose logging setups it would prove to be a very powerful tool. For example when trying to empty a bunch of orphaned disk queue files, it would be very easy to spin up several syslog-ng instances from a script with only slightly differing configuration. What do you guys think? Do you agree as well that this feature would bear a lot of added value to syslog-ng? Thanks for sharing your opinions! Best Regards, János Szigetvári -- Janos SZIGETVARI RHCE, License no. 150-053-692 <https://www.redhat.com/rhtapps/verify/?certId=150-053-692> LinkedIn: linkedin.com/in/janosszigetvari __@__˚V˚ Make the switch to open (source) applications, protocols, formats now: - windows -> Linux, iexplore -> Firefox, msoffice -> LibreOffice - msn -> jabber protocol (Pidgin, Google Talk) - mp3 -> ogg, wmv -> ogg, jpg -> png, doc/xls/ppt -> odt/ods/odp

3 3

pattern documentation for patterndb
by Russell Fulton 06 Aug '19

06 Aug '19

I am sure I have asked this question before but I can’t find the answer now. The Admin Guide has a very sort section on patterns with a couple of examples. I can’t find the detailed docs for the actual pattern elements (parser types). I seem to remember that these we documented in a separate pdf but I can’t find it. I add new rules about once a year and struggle with it every time! Russell

2 1

Quoting in filters
by Faine, Mark R. (MSFC-IS40)[NICS] 05 Aug '19

05 Aug '19

What is the policy on quoting values in filters? In the docs it seems that sometimes they are quoted and other times they are not. For example, are these equivalent: netmask (127.0.0.1/32) and netmask("127.0.0.1/32") I think that maybe some filters, the ones that take strings essentially, are quoted, and not the ones that take numeric data. Is that correct? Thanks, -Mark

2 1

new install of OSE 3.22 fails on ubuntu 16,04
by Russell Fulton 05 Aug '19

05 Aug '19

Running on ubuntu 16.04 I have just compiled and installed syslog-ng version 3.22. When started it prints: iv_tls_user_register: called after iv_init and then exits. Google tells me there was a bug that caused this back in 2013 but there are not recent posts. Any idea what is going on? BTW I can’t use the binary packages as there are conflicts between the rsyslog and syslog-ng packages and I am stuck with having the rsyslog package by corporate policy. Russell

2 1

Fwd: Multithreaded inputs with same output file : is it safe ?
by Jean-Baptiste Fuzier 02 Aug '19

02 Aug '19

Hello, I have a few questions related to multithreading. >From the doc, my understanding is that inputs are multithreaded this way : 1/ Tcp connections are sent to different thread when multithreading is enabled, even if there is a single source enabled in config with a single tcp port listening 2/ UDP datagrams are NOT sent to different thread if there is a single source even if there are multiple listening ports in the source config 3/ UDP datagrams are sent to different threads if there are different sources in config (1 source -> 1 thread), so basically to benefit from multithreading with UDP one needs to configure several sources. Is my understanding correct ? If we have several UDP source with some round robin balancing of the input to different source on the same daemon process, that means that several host with the same name can send their syslog data to different source. For the output, I am using files with the name of the sending host in the name. Is there an issue if several source (= from different threads) ends up being outputed in the same file. Another way to turn the question is: are all the data from the source thread then send to some common thread so that output to a file is always handled by the same thread even if log is received from different source threads ? I tried to ask the same question with a schematic : Case1 : +-------------------------+ +-------------------+ +-----------------------------+ +-----------+ +----------+ | | | | | | | host foo +------->+ |X----->+ thread1 some foo log +----->+ +--->+ thread handling foo.log +------>foo.log +-----------+ | iptables |X XX| some bar log | | black box magic | | | | round | XXXX +-------------------------+ | | +-----------------------------+ | robin | XXXXX | | | |XX XX+-------------------------+ | | +-----------------------------+ +-----------+ | X------>+ | | | | | | host bar +------->+ | | thread2 some foo log +----->+ +--->+ thread handling bar.log +------>bar.log +-----------+ +----------+ | some bar log | | | | | +-------------------------+ +-------------------+ +-----------------------------+ Case2 : +-------------------------+ +-----------+ +----------+ | | | host foo +------->+ |X----->+ thread1 some foo log +---------------------> +--------+ +-----------+ | iptables |X XX| some bar log +----------| |foo.log | | round | XXXX +-------------------------+ +------------> +--------+ 2 different thread writting | robin | XXXXX | | to same file | |XX XX+-------------------------+ | +---------->----------+ likely bad ! +-----------+ | X------>+ +--------+ |bar.log | | host bar +------->+ | | thread2 some foo log +--------------------->----------+ +-----------+ +----------+ | some bar log | +-------------------------+ One example config : # Notice that host name is used in output file name # There is no afinity between 1 host <> 1 udp port => logs from host foo are send to every source/udp port in a round robin fashion source s_syslog__udp5141{ network(ip(192.168.1.1) transport("udp") port(5141) flags("threaded")); }; source s_syslog__udp5142{ network(ip(192.168.1.1) transport("udp") port(5142) flags("threaded")); }; source s_syslog__udp5143{ network(ip(192.168.1.1) transport("udp") port(5143) flags("threaded")); }; source s_syslog__udp5144{ network(ip(192.168.1.1) transport("udp") port(5144) flags("threaded")); }; destination d_network_fw_misc { file("`syslog_log_root_path`/fw/fw/$FULLHOST/`naming_scheme`" group("logs_fw") dir-group("logs_fw") log-fifo-size(500000)); }; destination d_network_wifi { file("`syslog_log_root_path`/wifi/$FULLHOST/`naming_scheme`" group("logs_wifi") dir-group("logs_wifi") log-fifo-size(500000)); }; destination d_network_debug { file("`syslog_log_root_path`/debug/`naming_scheme`" group("logs_network_debug") dir-group("logs_network_debug") log-fifo-size(600000)); }; log { source(s_syslog__udp5141); source(s_syslog__udp5142); source(s_syslog__udp5143); source(s_syslog__udp5144); # Every channel of the junction received a copy of the message, unless there is a final flag before # Final flags are applied only within the junction and not globally junction { channel {filter(f_debug); destination(d_network_debug);}; channel {filter(f_network_wifi); destination(d_network_wifi); flags(final);}; channel {destination(d_network_fw_misc); flags(final);}; }; }; #Round robin with iptables : iptables -t nat -A PREROUTING -d 192.168.1.1 -i bond0 -p udp -m udp --dport 514 -m statistic --mode nth --every 4 --packet 0 -j REDIRECT --to-port 5141 iptables -t nat -A PREROUTING -d 192.168.1.1 -i bond0 -p udp -m udp --dport 514 -m statistic --mode nth --every 3 --packet 0 -j REDIRECT --to-port 5142 iptables -t nat -A PREROUTING -d 192.168.1.1 -i bond0 -p udp -m udp --dport 514 -m statistic --mode nth --every 2 --packet 0 -j REDIRECT --to-port 5143 iptables -t nat -A PREROUTING -d 192.168.1.1 -i bond0 -p udp -m udp --dport 514 -m statistic --mode nth --every 1 --packet 0 -j REDIRECT --to-port 5144 Thanks for your help -- Jean-Baptiste Fuzier

2 1

cisco templetes
by ciprian niculescu 31 Jul '19

31 Jul '19

Hello, i'm building a syslog relay to collect and duplicate the flows to multiple destinations. but the relayed messages are strange looking. my source are cisco network devices (catalyst, nexus, asa) and i want to relay to a Solarwinds, Splunk and a linux-syslog for archiving. i search the net for a templete but found none. What i got so far is that the catalyst is sending in syslog bsd format, but with the relay configured to source bsd and destination bsd, the end message is different (the date is doubled, the relay add his IP) any help is appreciated. Regards, Ciprian

2 1

Tips for handling large message load
by Faine, Mark R. (MSFC-IS40)[NICS] 30 Jul '19

30 Jul '19

I have several Splunk log aggregators that gets thousands of messages per second but we are seeing issues with dropping messages from UDP sources. I've read the section in the docs about handling large message load and we've made many of those changes. Do you have any other suggestions to improve performance? We are using flow control. We have made the following sysctl changes: - net.core.rmem_max = 268435456 - net.core.netdev_max_backlog = 2000 We have increased flush-lines to 100 We are also looking to increase the initial window size below and max connections as well as the so_rcvbuf. network(port(514) transport("tcp") max-connections(100) log_iw_size(10000) flags(syslog-protocol)); network(port(514) transport("udp") max-connections(100) log_iw_size(10000) flags(syslog-protocol)); network(ip(0.0.0.0) transport("tls") port(10514) max-connections(100) log_iw_size(10000) flags(syslog-protocol) I'm currently showing about 50 TCP connections and almost 1000 UDP connections, though it's early on Monday morning so it is likely to increase as the day goes on. Most of the log messages are coming from the VPNs (hence so many UDP connections). The servers are quite beefy with 64GB of RAM and 24 Xeon cores @ 2.4GHz (Dell PowerEdge R530). We are using two separate network interfaces one for TCP and the other for UDP. I'd appreciate any suggestions on how to further increase performance, also, any general rules or calculations I can use to determine optimal values for these parameters from available system metrics would be very helpful. Thanks, -Mark

3 2