- syslog-ng - lists.balabit.hu

Syslog-ng 3.16 unable to write to pipe
by Aguilu, Miguel 09 Apr '19

09 Apr '19

We are in the middle of a project to Migrate from 3.0 to 3.16 (latest in redhat distro) The standard 3.14 config works fine writing to some standard logs in /var/log But we need to write special files to /opt/log/ * In this directory will be a file type pipe to be read by a syslog probe (parse messages into alerts). Using the following configuration NG returns unable to write. source s_dgram { unix-dgram("/dev/log"); }; source s_kernel { file("/proc/kmsg" program_override("kernel: ")); }; source s_udp { udp(); }; destination netcool { pipe("/var/log/nco.pipe"); }; log { source(s_dgram); source(s_kernel); source(s_udp); filter(f_NOfwflow); filter(f_NOdata_cent); filter(f_NOtacacs); filter(f_netcool); destination(netcool); }; * Even if I remove the filter I get the same error. * Also removed all of the sources with the exception of the upd and got the same error. * Changed the file type to file and getting: * OLLERR occurred while idle; fd='66' * Apr 9 13:17:45 wbucrp-isdmz1a-lb err syslog-ng[2475]: Connection broken; time_reopen='60' * Apr 9 13:17:47 wbucrp-isdmz1b-lb err syslog-ng[10934]: POLLERR occurred while idle; fd='66' * Apr 9 13:17:47 wbucrp-isdmz1b-lb err syslog-ng[10934]: Connection broken; time_reopen='60' * Apr 9 13:17:51 wapcrp-isdmz0a-lb err syslog-ng[3713]: I/O error occurred while writing; fd='38', error='Connection refused (111)' * Apr 9 13:17:51 wapcrp-isdmz0a-lb err syslog-ng[3713]: Connection broken; time_reopen='60' * Apr 9 13:17:51 wapcrp-isdmz0a-lb notice syslog-ng[3713]: Suspending write operation because of an I/O error; fd='38', time_reopen='60' * The permissions in /opt/log are 777 and owned by root which is the user syslog-ng is running under. * Also change the path to where the config was updating file (/varr/log) and attemted to create the pipe and no luck Here are my global options: * options { * flush_lines (0); * # time_reopen (10); * log_fifo_size (1000); * chain_hostnames (on); * use_dns (yes); ## changed test * use_fqdn (yes); * keep_hostname (yes); * owner("root"); * group("root"); * create_dirs(yes); #Test * dir_perm(0755); * perm(0644); * }; ANY Ideas? Thanks Miguel [miki1] Miguel A Aguilu Netcool Suite SME Walt Disney Account Atos Consulting Mobile: (321) 261-1385 1120 Celebration Blvd Celebration, FL 34747 Miguel.Aguilu(a)Atos.net<mailto:Craig.Elliott@Atos.net> www.atos.net<http://www.atos.net/> [/Users/aguim114/Library/Containers/com.microsoft.Outlook/Data/Library/Caches/Signatures/signature_1691773396] Upcoming PTO: TBD This e-mail and the documents attached are confidential and intended solely for the addressee; it may also be privileged. If you receive this e-mail in error, please notify the sender immediately and destroy it. As its integrity cannot be secured on the Internet, the Atos group liability cannot be triggered for the message content. Although the sender endeavors to maintain a computer virus-free network, the sender does not warrant that this transmission is virus-free and will not be liable for any damages resulting from any virus transmitted.

2 1

tcpdump shows device sending syslog from UDP
by Lin, Victor 08 Apr '19

08 Apr '19

Hello Fabien, I was running tcpdump on my syslog-ng server, and able to see the package from 1.2.3.4 but my local logfile for UDP 514 didn't show any syslog from 1.2.3.4. Is there another configuration file need to be setup in somewhere? Thank you very much! VL -----Original Message----- From: syslog-ng [mailto:syslog-ng-bounces@lists.balabit.hu] On Behalf Of syslog-ng-request(a)lists.balabit.hu Sent: 2019, April, 08 8:00 AM To: syslog-ng(a)lists.balabit.hu Subject: syslog-ng Digest, Vol 168, Issue 8 Send syslog-ng mailing list submissions to syslog-ng(a)lists.balabit.hu To subscribe or unsubscribe via the World Wide Web, visit https://lists.balabit.hu/mailman/listinfo/syslog-ng or, via email, send a message with subject or body 'help' to syslog-ng-request(a)lists.balabit.hu You can reach the person managing the list at syslog-ng-owner(a)lists.balabit.hu When replying, please edit your Subject line so it is more specific than "Re: Contents of syslog-ng digest..." Today's Topics: 1. Re: WebGUI for Syslog-ng storing logs in HDFS (Lee Keng Ket) 2. Re: tcpdump shows device sending syslog from UDP 514 but not write to local log file (Fabien Wernli) ---------------------------------------------------------------------- Message: 1 Date: Mon, 8 Apr 2019 14:24:09 +0800 From: Lee Keng Ket <kengket(a)gmail.com> To: "Syslog-ng users' and developers' mailing list" <syslog-ng(a)lists.balabit.hu> Subject: Re: [syslog-ng] WebGUI for Syslog-ng storing logs in HDFS Message-ID: <CAHJFZ9edfAq2tKKfjYxNqFJrzjx-Yb77-BxFak4F+vCQDYWoWw(a)mail.gmail.com> Content-Type: text/plain; charset="utf-8" Hi, Peter Thanks for your info, will look into the link you share. As HDFS is decided to be the storage of logs, I'm trying not to store another copy in ES. Hi, Nik What's the difference between this open-distro and the ES from elastic.co? Thank you. Regards, KK On Fri, Apr 5, 2019 at 10:21 PM Nik Ambrosch <nik(a)ambrosch.com> wrote: > I was considering HDFS but ultimately chose ES because of simplicity. > I too spent a while looking for a pre-packaged suite but most things > are commercial. > > I'd like to check this out sometime, just haven't had the opportunity > - https://opendistro.github.io/for-elasticsearch/ > > On Fri, Apr 5, 2019 at 10:00 AM Czanik, Péter > <peter.czanik(a)balabit.com> > wrote: > >> Hi, >> >> Many syslog-ng users store logs to Elasticsearch and search/analyze >> them using Kibana. Personally I never tested, but should take a look >> at Elasticsearch-Hadoop if it solves your problem: >> https://www.elastic.co/products/hadoop >> >> Bye, >> >> Peter Czanik (CzP) <peter.czanik(a)balabit.com> Balabit (a OneIdentity >> company) / syslog-ng upstream https://syslog-ng.com/community/ >> https://twitter.com/PCzanik >> >> >> On Fri, Apr 5, 2019 at 3:50 PM Lee Keng Ket <kengket(a)gmail.com> wrote: >> >>> Hi, >>> >>> How about open source webGUI? Any recommendations? >>> >>> Thank you. >>> >>> Regards, >>> KK >>> >>> >>> On Fri, 5 Apr 2019, 20:56 Jim Hendrick, <james.r.hendrick(a)gmail.com> >>> wrote: >>> >>>> It's not free - but you might look at the Syslog Store Box - >>>> https://www.syslog-ng.com/products/log-management-appliance/ >>>> >>>> They have also done some cool stuff with a native splunk HEC >>>> destination in the professional edition of syslog-ng. Might look at >>>> using that and splunk for the front-end? >>>> >>>> Jim >>>> >>>> >>>> On Fri, Apr 5, 2019 at 2:38 AM Lee Keng Ket <kengket(a)gmail.com> wrote: >>>> >>>>> Hi, >>>>> >>>>> I'm looking for a WebGUI application for users to search for >>>>> syslog, which the logs are stored by syslog-ng into HDFS. >>>>> >>>>> Can you please share what WebGUI you're using? >>>>> >>>>> I realize ElasticSearch with Kibana are the famous solution for >>>>> this, however, I wish to store my syslog in HDFS. >>>>> >>>>> Thank you. >>>>> >>>>> Regards, >>>>> KK >>>>> >>>>> __________________________________________________________________ >>>>> ____________ Member info: >>>>> https://lists.balabit.hu/mailman/listinfo/syslog-ng >>>>> Documentation: >>>>> http://www.balabit.com/support/documentation/?product=syslog-ng >>>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq >>>>> >>>>> >>>> ___________________________________________________________________ >>>> ___________ Member info: >>>> https://lists.balabit.hu/mailman/listinfo/syslog-ng >>>> Documentation: >>>> http://www.balabit.com/support/documentation/?product=syslog-ng >>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq >>>> >>>> >>> ____________________________________________________________________ >>> __________ Member info: >>> https://lists.balabit.hu/mailman/listinfo/syslog-ng >>> Documentation: >>> http://www.balabit.com/support/documentation/?product=syslog-ng >>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq >>> >>> >> _____________________________________________________________________ >> _________ Member info: >> https://lists.balabit.hu/mailman/listinfo/syslog-ng >> Documentation: >> http://www.balabit.com/support/documentation/?product=syslog-ng >> FAQ: http://www.balabit.com/wiki/syslog-ng-faq >> >> > ______________________________________________________________________ > ________ Member info: > https://lists.balabit.hu/mailman/listinfo/syslog-ng > Documentation: > http://www.balabit.com/support/documentation/?product=syslog-ng > FAQ: http://www.balabit.com/wiki/syslog-ng-faq > > -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.balabit.hu/pipermail/syslog-ng/attachments/20190408/feb5e5c1/a…> ------------------------------ Message: 2 Date: Mon, 8 Apr 2019 10:40:05 +0200 From: Fabien Wernli <wernli(a)in2p3.fr> To: "Syslog-ng users' and developers' mailing list" <syslog-ng(a)lists.balabit.hu> Subject: Re: [syslog-ng] tcpdump shows device sending syslog from UDP 514 but not write to local log file Message-ID: <20190408084005.j4v2k5atrmcn6amp(a)ccfawe.in2p3.fr> Content-Type: text/plain; charset=us-ascii On Fri, Apr 05, 2019 at 03:46:37PM +0000, Lin, Victor wrote: > When use: tcpdump src host 1.3.4.5 and port 514 , I can see host 1.3.4.5 is sending the syslog to my syslog-ng > But when I search my local log special for port 514, I didn't see any > syslog from 1.3.4.5 firewall? ------------------------------ Subject: Digest Footer _______________________________________________ syslog-ng maillist - syslog-ng(a)lists.balabit.hu https://lists.balabit.hu/mailman/listinfo/syslog-ng ------------------------------ End of syslog-ng Digest, Vol 168, Issue 8 ***************************************** _______________________________________________________________________ If you received this email in error, please advise the sender (by return email or otherwise) immediately. You have consented to receive the attached electronically at the above-noted email address; please retain a copy of this confirmation for future reference. Si vous recevez ce courriel par erreur, veuillez en aviser l'expéditeur immédiatement, par retour de courriel ou par un autre moyen. Vous avez accepté de recevoir le(s) document(s) ci-joint(s) par voie électronique à l'adresse courriel indiquée ci-dessus; veuillez conserver une copie de cette confirmation pour les fins de reference future.

2 1

tcpdump shows device sending syslog from UDP 514 but not write to local log file
by Lin, Victor 08 Apr '19

08 Apr '19

Dear all, When use: tcpdump src host 1.3.4.5 and port 514 , I can see host 1.3.4.5 is sending the syslog to my syslog-ng But when I search my local log special for port 514, I didn't see any syslog from 1.3.4.5 Is there another location to config the syslog-ng, except /app/syslog-ng/custom/conf/syslog-ng.conf ? Below is from my syslog-ng.conf # Syslog collection for all devices source s_network { network( transport("udp") port(514) flags(syslog_protocol) keep_hostname(yes) keep_timestamp(yes) use_dns(no) use_fqdn(no) ); }; destination d_all_logs { file("/app/syslog-ng/custom/output/all_devices.log"); network("102.45.2.86" port(514) transport(udp) spoof_source(yes)); }; log { source(s_network); source(s_arista_network); destination(d_all_logs); }; Thank you very much for reply in advance!!!!! VL _______________________________________________________________________ If you received this email in error, please advise the sender (by return email or otherwise) immediately. You have consented to receive the attached electronically at the above-noted email address; please retain a copy of this confirmation for future reference. Si vous recevez ce courriel par erreur, veuillez en aviser l'expéditeur immédiatement, par retour de courriel ou par un autre moyen. Vous avez accepté de recevoir le(s) document(s) ci-joint(s) par voie électronique à l'adresse courriel indiquée ci-dessus; veuillez conserver une copie de cette confirmation pour les fins de reference future.

2 1

WebGUI for Syslog-ng storing logs in HDFS
by Lee Keng Ket 08 Apr '19

08 Apr '19

Hi, I'm looking for a WebGUI application for users to search for syslog, which the logs are stored by syslog-ng into HDFS. Can you please share what WebGUI you're using? I realize ElasticSearch with Kibana are the famous solution for this, however, I wish to store my syslog in HDFS. Thank you. Regards, KK

4 5

Dropping specific messages from Syslog-NG
by Bell, Christopher E. 05 Apr '19

05 Apr '19

Disclaimer: I'm not an expert in either Linux or syslog-ng. I'm also not a programmer :). That being said, I've built a syslog-ng --> Splunk deployment and I'm trying to drop unwanted log messages before they make their way to Splunk... so drop them at syslog-ng. There are going to be approx. 700 devices logging to this deployment (routers, switches, firewalls, IPAM, AAA, etc..) so I need to make sure whatever I do can scale easily and my teammates can help me manage it. I'm trying to drop messages that are similar to the below: Apr 5 06:02:17 x.x.x.x 2019 Apr 5 06:02:17 EDT: %USER-3-SYSTEM_MSG: NTP Receive dropping message: Received NTP control mode packet. Drop count:4200 - ntpd As well as: Apr 5 08:03:49 x.x.x.x Apr 5 2019 12:03:48.895 UTC: %SNMP-3-RESPONSE_DELAYED: processing GetNext of entPhysicalEntry.2.1065 (5739 msecs) Two different devices. I've tried literally dozens of methods, my latest is below and I can't figure out why we're still logging these messages: filter f_trash { match(SNMP-3-RESPONSE_DELAYED value(MSGHDR)); and match(NTP Receive dropping message value(MSG))}; log { source(s_sys); filter(f_trash); flags (final);}; Any help / constructive criticism would be greatly appreciated.

2 3

Interest in GSoC 2019
by Kushal K S V S 03 Apr '19

03 Apr '19

Hi, I am Kushal. I am a Final Year Undergrad in Computer Science IIT Guwahati, India. I am very much interested in working on Syslog-ng and with the community on the project "Analyze and improve JSON performance". I have installed from the source code and am currently going through the code. I have already participated in GSoC in 2017 as a part of The FreeType Org. Many of my projects including the previous GSoC project are in C. I am well-versed in C, C++, Bash, makefile, etc. which will be the technologies to be used for this project. Hope I am not too late. I am thinking of analyzing the JSON parsers available to get an idea on how to proceed. Any help regarding this would be highly appreciated. Yours sincerely, Kushal K S V S

1 0

syslog-ng moving to its own organization on github
by Scheidler, Balázs 01 Apr '19

01 Apr '19

Hi, syslog-ng as an open source project has long been supported and sponsored by a company called Balabit (which I personally was the founder of), hence syslog-ng related online resources like the github project was published under github.com/balabit. Since Balabit was acquired last January by One Identity (www.oneidentity.com) going forward, the Balabit branding is not the right one for the syslog-ng project. Since the acquisition, all on-line syslog-ng presence was concentrated under the "syslog-ng" brand. An example for this is the website, which is now at www.syslog-ng.com. In a similar way, developer and open source related resources should follow suit. Our plan is to use "syslog-ng" everywhere where we were using "balabit". This includes github, dockerhub, gitter, slack, mailing list etc. We expect that the transition will take time, and that old links, chat rooms, docker images will work for a time yet, redirects will be there for those of us still using the old links. However it is a good idea to update bookmarks & git remotes that this will work on the long run. Over time, the name "Balabit" will fade a bit, at least within the syslog-ng community, but that's good. That's progress. Cheers, Bazsi founder of syslog-ng founder of Balabit

1 0

Redhat repo
by Faine, Mark R. (MSFC-IS40)[NICS] 01 Apr '19

01 Apr '19

I have a question about the Redhat repo for the unofficial OSE builds. It seems as though they are tied to a specific version. For example, I'm currently using czanik-syslog-ng318epel6, however, I see that 3.19 has a different repo specifically for that version. This seems to be counter to the point of yum repositories and it imposes an additional maintenance burden. Am I maybe not properly understanding how it works, and if not, is there a repo that tracks with the latest stable version? -Mark

6 9

Bind source IP using localip()?
by Scot 29 Mar '19

29 Mar '19

Is there a way to bind to a src IP when sending forwarding an event from syslog-ng ? We have an alias on the system and want to use that as the sending IP address for a forward. Tried the localip option which seems to pass conf check but ignored for destinations. Looked in the network and syslog driver documentation and didn't see anything obvious. syslog-ng 3 (3.14.1) Config version: 3.14 Installer-Version: 3.14.1 Revision: Compile-Date: Mar 14 2018 09:48:15 Module-Directory: //usr/lib64/syslog-ng Module-Path: //usr/lib64/syslog-ng Available-Modules: add-contextual-data,affile,afprog,afsocket,afstomp,afuser,basicfuncs,cef,confgen,cryptofuncs,csvparser,date,dbparser,disk-buffer,graphite,json-plugin,kvformat,linux-kmsg-format,pseudofile,sdjournal,syslogformat,system-source,afsmtp,geoip-plugin,mod-java,map-value-pairs,snmptrapd-parser,stardate,tfgetent,appmodel,tags-parser,xml,geoip2-plugin,afamqp Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-IPv6: on Enable-Spoof-Source: on Enable-TCP-Wrapper: on Enable-Linux-Caps: on Enable-Systemd: on

3 2

Convert if/else to nested log paths
by Faine, Mark R. (MSFC-IS40)[NICS] 27 Mar '19

27 Mar '19

What is the conversion of an if/else to embedded log path statements? I tried to do this today and didn't have any luck so I reverted back to if/else. I have a log statement with a series of if/else blocks: log { source(pan_splunk); if { filter { host("^[a-z]+\.foo.*$") or netmask('192.168.1.100/32') or netmask('192.168.1.101/32'); }; rewrite { set("foo" value("location")); }; } elif { filter { host("^[a-z]+\.bar.*$") or netmask('192.168.1.102/32') or netmask('192.168.1.103/32'); }; } else { rewrite { set("unknown" value("location")); }; } Can this be written with embedded log statements? The if/else blocks are working for me so this is just an academic exercise but I'd really like to understand how to do it with embedded log paths. Thanks, -Mark

2 2