- syslog-ng - lists.balabit.hu

Relay log message through Application LoadBalancer
by Ronald Fenner 23 Feb '19

23 Feb '19

I'm trying to figure out how to relay a message through an AWS Application Load Balancer. I have the config setup to relay the message with: destination d_kafka_unstructured { network( "<dns-name>" port(9082) );}; Look at the tcpdump log I can see that the message is relayed but the ALB doesn't know how to handle it and responds with 400 sinceing the message isn't a HTTP Request. What would be the best way to wrap the message in a http like request that the ALB will understand. Is there an easy way to just tell syslog-ng to relay the message as HTTP or would I need to write something to add/strip the HTTP wrapper. Ronald Fenner Network Architect Game Circus LLC. rfenner(a)gamecircus.com

3 3

Latest syslog-ng RPM needed for RHEL7
by Harish Shetty 22 Feb '19

22 Feb '19

Hi I am planning to upgrade my syslog servers to RHEL7 , I am able to see syslog-ng-3.19.1.tar.gz. Can i get RPM for the same Regards Harish Shetty

2 1

Regex Irregularities
by N. Max Pierson 13 Feb '19

13 Feb '19

Hi List, I am having some weird issues with rewrite regex that I cannot explain. I am simply trying to filter out the first part of the message which has the date in this format. Feb 8 09:13:32 CST: (there is one space at the end) When I use the following syntax, it doesn't match as expected. ^\w+\s\d+\s\d+:\d+:\d+\s\w+:\s I know this is the correct pattern because it works just fine on www.regexpal.com. I did some further testing and I have narrowed it down to the below ... ^\w+ 8 09:55:54 CST: (this seemed to also remove the space behind the month) ^\w+\s 8 09:59:37 CST: (notice this is the exact same as the above without the beginning space) ^\w+\s\d+ Feb 8 10:07:04 CST: (doesn't match anything as though the space between Feb and 8 isn't there) ^\w+\d+ Feb 8 10:11:54 CST: (again doesn't match anything as though there is a space between Feb and 8) So it seems to be something either with \w word class or the + quantifier and it somehow eats the space behind it possibly?? I am running 3.19.1 on Centos 7. Can anyone test this to confirm it isn't just local to my install for whatever reason? Regards, Max

3 3

MS-SQL Question
by Garridan 07 Feb '19

07 Feb '19

Hello! I'm a new syslog-ng user, so please be gentle with me. :) I'm attempting to log to an MS-SQL database and would like to send to different tables in the same DB based on the source IP - for example device A to its own table, device B to its own table, and so on. I thought I would simply need to create the same destination but define different table names in each, it works, however under load the syslog-ng service starts restarting over and over and eventually MS-SQL errors and alerts that the login packet is structurally invalid. Is it possible to send to different table names in this manner or is there another way to do it? Thanks!

6 8

syslog-ng 3.19 not processing /dev/log on RH#L6
by Evan Rempel 29 Jan '19

29 Jan '19

We just had a network even in our environment that resulted in many hosts being isolated on the network in some very odd ways. The hosts had some connections inbound, but no outbound and the network seemed to come and go very quickly. syslog-ng detected that it lost connection to its down-stream syslog collectors a did the normal connection reaping. We have this set for 5 seconds. At some point the syslog-ng process terminated and the supervisor process launched a new child. This worked well, but in this odd senario this happened quickly and at on point the supervisor process failed to reap one of its defunct children but still launched a new child process. The defunct process seemed to hold onto the /dev/log handle so the new child could not get access to it and the result is that we lost hours of logs that flow through the /dev/log socket. When we discovered the issue, there where two children of the supervisor process. One sas in a "defunct" state while the other was running, but only logging kernel messages (from /dev/kmsg) and from the internal syslog-ng source. I suspect that the child reaping process on SIGCHLD does not use a while loop and misses some children. Evan.

2 3

syslog-ng survey
by Scheidler, Balázs 29 Jan '19

29 Jan '19

Hi, If you have a bit of a time, please help us by filling this survey about syslog-ng: https://www.surveymonkey.com/r/ZHJ9MHP It is really appreciated. Cheers, Bazsi

1 0

Running multiple syslog-ng instances on the same host (RHEL6 CO6 SL6)
by Evan Rempel 20 Jan '19

20 Jan '19

In my environment we want to have isolation between the standard OS based syslog-ng service and the central syslog-ng collector service, which of course is running on a host that already has an OS based syslog-ng process running on it. Running muliple syslog-ng daemons under systemd is not so difficult. Copy the system service file and systemd automatically registers a .d folder for custom configurations, however, there is nothing automatic about the System V init mechanism used in the RHEL6 based systems. Don't worry because they syslog-ng init script from the copr yum repository (https://copr.fedorainfracloud.org/coprs/czanik) starting at 3.19 has already thought of this and made multiple instances as automatic as they can be. Quick steps to create a second syslog-ng instance: 1. Decide on the new name extension (shown here is yourName) 2. Copy the included init script cp /etc/init.d/syslog-ng /etc/init.d/syslog-ng-yourName 3. Change the SYSLOGNG_SUFFIX value in your new init script to the value "-yourName". You need the leading hyphen because your added -yourName to the init file name. 4. Copy and customize the included customization file cp /etc/sysconfig/syslog-ng /etc/sysconfig/syslog-ng-yourName By default all of the configuration and runtime files will default to file names including the -yourName suffix, so very little if any customization should be needed. 5. Create a new syslog-ng configuration file for your new instance vi /etc/syslog-ng/syslog-ng-yourName 6. Enable the auto-start of your new instance chkconfig --add syslog-ng-yourName 7. Start your instance /sbin/service syslog-ng-yourName start Thanks again to all of the people adding value to an already great product!

1 0

Issues with sql driver
by N. Max Pierson 18 Jan '19

18 Jan '19

Hi List, I have 2 questions about the sql driver. First, I am trying to get messages into sql using the sql driver but I get an error when I try and restart syslog-ng when I enable the log statement with the sql destination. The syslog-ng --syntax-only command runs without any issues but systemd throws and error that it cannot restart the service but doesn't give a clear reason. My config is below, doesn't anyone know where in a log I can see why it won't restart?? source s_network { udp(ip(0.0.0.0) port(514)); }; destination d_mysql { sql(type(mysql) host("127.0.0.1") username("syslog-ng") password("password") database("syslog") table("messages_${HOST}") columns("date", "host", "message") values("${R_DATE}", "${HOST}", "${MESSAGE}") indexes("date", "host") ); }; log { source(s_network); destination(d_mysql); }; My second question is can you use a template with the sql destination driver? I need to reformat some Cisco Nexus logs because of how it formats the date (looks to be non RFC compliant) and if so, can someone give me a sample of config with the template in the sql destination driver? I cannot seem to find in the docs if this is even possible much less and example of how to do it. TIA, Max

3 10

Insider 2019-01: central server; survey; SNMP; Slack; Conferences & Webinars
by Czanik, Péter 17 Jan '19

17 Jan '19

Dear syslog-ng users, This is the 72nd issue of syslog-ng Insider, a monthly newsletter that brings you syslog-ng-related news. NEWS Creating a central syslog server -------------------------------- Your home network might already contain some devices or systems like a home server, a WiFi router, a media player, or home automation system. Creating a central syslog server to store system logs or syslogs of these systems is considered a best practice. Learn, how to get started: https://balagetech.hu/en/2019/01/14/creating-a-central-syslog-server/ Looking for your feedback about syslog-ng ----------------------------------------- Help us set the direction for syslog-ng by providing your valuable feedback through this short, 9-question survey. Your input will be of great value in determining which features and capabilities to focus our efforts on, going forward. While this survey can be completed anonymously, we would appreciate if you would share your email address, in case we would need to clarify your response and input. https://www.surveymonkey.com/r/ZHJ9MHP SNMP traps to Splunk -------------------- As we learned at the Splunk .conf18 in October, forwarding SNMP traps to Splunk can be a challenging task. Luckily, using syslog-ng can simplify it for us. All we need to make sure about is that snmptrapd logs traps to a file. The syslog-ng application can read and parse that file and forward the traps to Splunk HEC (HTTP Event Collector). https://www.syslog-ng.com/community/b/blog/posts/using-syslog-ng-to-forward… Send your log messages to Slack ------------------------------- One of the many ways members of a team collaborate is to use Slack. >From syslog-ng version 3.19, you can send log messages to Slack. You can receive critical log messages in real-time in your Slack client on your mobile or desktop. Learn how to set up alerting to Slack in just a few minutes. https://www.syslog-ng.com/community/b/blog/posts/send-your-log-messages-to-… CONFERENCES * FOSDEM talk: Extending syslog-ng in Python ( https://fosdem.org/2019/schedule/track/python/ ) You can also discuss syslog-ng with me at the following events: * https://devconf.info/cz * https://wiki.centos.org/Events/Dojo/Brussels2019 * https://cfgmgmtcamp.eu/ WEBINARS Coming up: * Send logs using the Advanced Log Transport Protocol (ALTP): https://www.syslog-ng.com/event/send-logs-using-the-advanced-log-transport-… You can also watch our past webinars: * Log ingestion to Splunk HEC: https://www.brighttalk.com/webcast/16207/338190 * High performance log streaming to HDFS with syslog-ng: https://www.brighttalk.com/webcast/16207/335943 Your feedback and news, or tips about the next issue are welcome. To read this newsletter online, visit: https://syslog-ng.com/blog/ Peter Czanik (CzP) <peter.czanik(a)balabit.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik

1 0

Rewrite regex not working entirely
by N. Max Pierson 17 Jan '19

17 Jan '19

Hi List, I am using version 3.5 and it seems as though regex (posix or pcre) doesn't work completely. Take the example string below (which is the message part of the syslog). Jan 15 15:50:57 CST: %DAEMON-3-SYSTEM_MSG: NTP Receive dropping message: Received NTP control mode packet. Drop count:147972 - ntpd[15029] I am trying to match the date at the beginning of the message and remove it. When I use \w, \s, \d, etc, they do not match anything. If I match on a character classes it works fine (ex [a-z]+ or [0-9]+). Here is my statement for the rewrite rule. rewrite r_nexus{ subst("^[a-z]+ [0-9]+ [0-9]+:[0-9]+:[0-9]+ [a-z]+: ", "", value("MESSAGE"), type("posix"), flags("ignore-case"), condition(filter(f_nexus))); }; The above seems to get me what I want but are the character matches not supposed to work in syslog-ng version 3.5?? Regards, Max

5 6