- syslog-ng - lists.balabit.hu

Converting multiline text input to concatanated single-line syslog format
by Newport, Brendan (Contractor - Security Operations - Development & Support) 30 Sep '16

30 Sep '16

Classification: Public Hi! Thanks for the responses. The multi-line-garbage() line was only in there to aid testing. I will look at the LF/CR elements in the input. We have a support contract with BalaBit with a local Var defined and so I'll raise a support ticket. Version 5.0.6LTS is the version rolled-out across our enterprise, but an upgrade to 6LTS will likely occur before October 2017. So testing was performed with the version in use. Lloyds Banking Group plc. Registered Office: The Mound, Edinburgh EH1 1YZ. Registered in Scotland no. SC95000. Telephone: 0131 225 4555. Lloyds Bank plc. Registered Office: 25 Gresham Street, London EC2V 7HN. Registered in England and Wales no. 2065. Telephone 0207626 1500. Bank of Scotland plc. Registered Office: The Mound, Edinburgh EH1 1YZ. Registered in Scotland no. SC327000. Telephone: 03457 801 801. Cheltenham & Gloucester plc. Registered Office: Barnett Way, Gloucester GL4 3RL. Registered in England and Wales 2299428. Telephone: 0345 603 1637 Lloyds Bank plc, Bank of Scotland plc are authorised by the Prudential Regulation Authority and regulated by the Financial Conduct Authority and Prudential Regulation Authority. Cheltenham & Gloucester plc is authorised and regulated by the Financial Conduct Authority. Halifax is a division of Bank of Scotland plc. Cheltenham & Gloucester Savings is a division of Lloyds Bank plc. HBOS plc. Registered Office: The Mound, Edinburgh EH1 1YZ. Registered in Scotland no. SC218813. This e-mail (including any attachments) is private and confidential and may contain privileged material. If you have received this e-mail in error, please notify the sender and delete it (including any attachments) immediately. You must not copy, distribute, disclose or use any of the information in it or any attachments. Telephone calls may be monitored or recorded.

1 0

attempting to get parsed logs into elasticsearch
by Russell Fulton 30 Sep '16

30 Sep '16

Hi folks I am trying to get some parsed logs into elasticssearch but have ended up with a cryptic error message: Starting syslog-ng /usr/lib/jvm/java-1.7.0-oracle-1.7.0.91.x86_64/jre/lib/amd64/server [2016-09-30T12:43:43.649899] Error initializing message pipeline; which almost certainly relates to the ES set up but I have no idea what is actually wrong. The ES logs do not show anything. Config file: @version: 3.8 @module mod-java @include "scl.conf" options { use_dns (no); use_fqdn (no); keep_hostname (yes); }; source s_loghost { tcp(flags(no-multi-line) port(1514) keep-alive(yes)); }; destination d_syslog { file("/var/log/syslog.log"); }; destination d_elastic { elasticsearch( index("auth_${YEAR}.${MONTH}.${DAY}") type("auth") cluster("security") flush-limit("1000") ); }; parser p_patterns { db-parser( file("/etc/syslog-ng/merged.xml")); }; log { source(s_loghost); parser (p_patterns); destination(d_elastic ); }; The same configuration with a json file destination works fine. Any hints on what to look at appreciated. The ES instance running on the host is set to data: no and I expect it to ship the data to one of the other nodes which has storage. Russell (who admits to being an ES novice)

2 3

moving debian directory from master to release branch on GitHub
by Noémi Ványi 29 Sep '16

29 Sep '16

Hello! Debian directory is being moved from master to another branch named release. Thus, the source code is separated from the build. If someone wants to create a Debian package, he/she must checkout that branch first. If someone has any concerns or notes, please share either here or in the PR on GitHub. PR: https://github.com/balabit/syslog-ng/pull/1215 kvch

2 1

experimental syslog-ng incubator rpm-s for 3.8
by Czanik, Péter 29 Sep '16

29 Sep '16

Hi, I just updated my unofficial syslog-ng 3.8 fedora/rhel repo with an experimental incubator package. I used the latest git HEAD snapshot together with this patch: https://github.com/balabit/syslog-ng-incubator/pull/158 Notable changes since my packages for 3.7: - bugfixes & code changes needed to compile on 3.8 - grok support removed - librdkafka updated to 0.9.1 Due to lack of Kafka, ZeroMQ, etc. I did not test the packages, just compiled them. If you need grok, let me know and I'll try to fix the package... As usual, packages are available at https://copr.fedorainfracloud.org/coprs/czanik/syslog-ng38/ Your feedback, be it negative or positive are very welcome! Bye, Peter Czanik (CzP) <peter.czanik(a)balabit.com> Balabit / syslog-ng upstream http://czanik.blogs.balabit.com/ https://twitter.com/PCzanik

1 0

Re: [syslog-ng] disk based buffering
by thejaguar＠tutanota.de 29 Sep '16

29 Sep '16

So with dqtool ( cool) I noticed that actual problem is that syslog-ng is not writing to disk queue immediately when a message is generated. The message goes to usual place like var/log/message etc but not in the queue where it should because it has not been transmitted to remote host. Now I am thinking there has to be a memory limit before it starts flushing/dumping unsent message to disk queue ? I tried mem-buf-size(1) and log-fifo-size(1) but none of them works. Is it a static value or is it configurable ? Thanks 27. Sep 2016 12:41 by thejaguar(a)tutanota.de: > > Thanks.> So if the queue stays intact, syslog-ng will try to send unsent messages as and when it starts ? even after 2-3 days ? it does not reset the queue or tracking ever ? > Thanks again > > 27. Sep 2016 12:11 by > balazs.scheidler(a)balabit.com> : > > >> >> Syslog-ng attempts to address application level failures with reliable disk buffer but kernel level crashes/power failures are not covered, at least you can suffer message loss, but the queue in general should stay intact. >> >> There's a tool for reading disk queue files, iirc the name is dqtool, should be included in your package. >> >> On Sep 27, 2016 8:35 PM, <>> thejaguar(a)tutanota.de>> > wrote: >> >>> >>> Hi,>>> I have been using disk based buffering with reliable turned on yes as suggested here :->>> https://www.balabit.com/documents/syslog-ng-ose-latest-guides/en/syslog-ng-… >>> >>> This has been working great for me on an embedded linux device which does not have internet connection except when the application running on it turns on the modem/pppd when it has to send some data, basically to save battery power. Now syslog-ng is brilliant and sends all the stored/queued logs immediately upon detecting network connection as long as system stays alive. Now the challenge is if the device has a system reset or kernel crash in between network connection availability, will syslog-ng send unsent logs upon next system reboot when it gets the network connection ? Or it resets the queue and tracking upon system reset/boot ?>>> I noticed any logs generated in between power resets and which are not sent are not transmitted on next net connection. Is it expected behaviour ? If not then what wrong I am doing ? also how can I read whats in /var/lib/syslog-ng/syslog-ng-00000.rqf or syslog-ng.persist ? >>> ======================= >>> >>> destination d_net {>>> network (>>> "`myloghost`" port(`mylogport`) transport("tls")>>> tls( ca-dir("/etc/syslog-ng/ca") peer-verify(required-trusted) ssl-options(no-sslv3,no-tlsv1) )>>> disk-buffer( reliable(yes) mem-buf-size(1M) disk-buf-size(5M) qout-size(64) )>>> template("<$PRI> $FACILITY $ISODATE $HOST $PROGRAM $MSG\n")>>> );>>> }; >>> syslog-ng 3.8.1>>> Installer-Version: 3.8.1>>> Revision:>>> Module-Directory: /usr/lib/syslog-ng>>> Module-Path: /usr/lib/syslog-ng>>> Available-Modules: cef,affile,basicfuncs,system-source,cryptofuncs,graphite,pseudofile,afuser,kvformat,add-contextual-data,date,csvparser,linux-kmsg-format,confgen,syslogformat,afprog,disk-buffer,dbparser,afsot>>> Enable-Debug: off>>> Enable-GProf: off>>> Enable-Memtrace: off>>> Enable-IPv6: off>>> Enable-Spoof-Source: off>>> Enable-TCP-Wrapper: off>>> Enable-Linux-Caps: off >>> ======================= >>> >>> Thanks >>> >>> >>> ______________________________________________________________________________ >>> Member info: >>> https://lists.balabit.hu/mailman/listinfo/syslog-ng >>> Documentation: >>> http://www.balabit.com/support/documentation/?product=syslog-ng >>> FAQ: >>> http://www.balabit.com/wiki/syslog-ng-faq >>> >>> >>> >> >>

5 6

Re: [syslog-ng] disk based buffering
by Scheidler, Balázs 28 Sep '16

28 Sep '16

Syslog-ng attempts to address application level failures with reliable disk buffer but kernel level crashes/power failures are not covered, at least you can suffer message loss, but the queue in general should stay intact. There's a tool for reading disk queue files, iirc the name is dqtool, should be included in your package. On Sep 27, 2016 8:35 PM, <thejaguar(a)tutanota.de> wrote: Hi, I have been using disk based buffering with reliable turned on yes as suggested here :- https://www.balabit.com/documents/syslog-ng-ose-latest-guides/en/syslog-ng- ose-guide-admin/html/configuring-diskbuffer-reliable.html This has been working great for me on an embedded linux device which does not have internet connection except when the application running on it turns on the modem/pppd when it has to send some data, basically to save battery power. Now syslog-ng is brilliant and sends all the stored/queued logs immediately upon detecting network connection as long as system stays alive. Now the challenge is if the device has a system reset or kernel crash in between network connection availability, will syslog-ng send unsent logs upon next system reboot when it gets the network connection ? Or it resets the queue and tracking upon system reset/boot ? I noticed any logs generated in between power resets and which are not sent are not transmitted on next net connection. Is it expected behaviour ? If not then what wrong I am doing ? also how can I read whats in /var/lib/syslog-ng/syslog-ng-00000.rqf or syslog-ng.persist ? ======================= destination d_net { network ( "`myloghost`" port(`mylogport`) transport("tls") tls( ca-dir("/etc/syslog-ng/ca") peer-verify(required-trusted) ssl-options(no-sslv3,no-tlsv1) ) disk-buffer( reliable(yes) mem-buf-size(1M) disk-buf-size(5M) qout-size(64) ) template("<$PRI> $FACILITY $ISODATE $HOST $PROGRAM $MSG\n") ); }; syslog-ng 3.8.1 Installer-Version: 3.8.1 Revision: Module-Directory: /usr/lib/syslog-ng Module-Path: /usr/lib/syslog-ng Available-Modules: cef,affile,basicfuncs,system-source,cryptofuncs,graphite, pseudofile,afuser,kvformat,add-contextual-data,date, csvparser,linux-kmsg-format,confgen,syslogformat,afprog, disk-buffer,dbparser,afsot Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-IPv6: off Enable-Spoof-Source: off Enable-TCP-Wrapper: off Enable-Linux-Caps: off ======================= Thanks ____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/? product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq

2 2

Getting 100k EPS into syslog-ng
by Clayton Dukes 28 Sep '16

28 Sep '16

I have searched high and low for actual configs on how to do this, but can't nail down anything solid. Does anyone have a good document or site I can reference which outlines all the details of getting syslog-ng to receive at 100k eps? Including things like what is needed to add/change in sysctl.conf, etc. I've found pages, like a blog entry by Etsy claiming it, but they left out the details on how to actually do it unfortunately. [cid:image001.png@01D218DE.0E88D320] Clayton Dukes / Founder and CTO (818) 869-4552/ cdukes(a)logzilla.net<mailto:cdukes@logzilla.net> LogZilla Corporation http://www.logzilla.net<http://www.logzilla.net/>

2 1

disk based buffering
by thejaguar＠tutanota.de 27 Sep '16

27 Sep '16

Hi,I have been using disk based buffering with reliable turned on yes as suggested here :-https://www.balabit.com/documents/syslog-ng-ose-latest-guides/en/syslog-ng-ose-guide-admin/html/configuring-diskbuffer-reliable.html This has been working great for me on an embedded linux device which does not have internet connection except when the application running on it turns on the modem/pppd when it has to send some data, basically to save battery power. Now syslog-ng is brilliant and sends all the stored/queued logs immediately upon detecting network connection as long as system stays alive. Now the challenge is if the device has a system reset or kernel crash in between network connection availability, will syslog-ng send unsent logs upon next system reboot when it gets the network connection ? Or it resets the queue and tracking upon system reset/boot ?I noticed any logs generated in between power resets and which are not sent are not transmitted on next net connection. Is it expected behaviour ? If not then what wrong I am doing ? also how can I read whats in /var/lib/syslog-ng/syslog-ng-00000.rqf or syslog-ng.persist ? ======================= destination d_net { network ( "`myloghost`" port(`mylogport`) transport("tls") tls( ca-dir("/etc/syslog-ng/ca") peer-verify(required-trusted) ssl-options(no-sslv3,no-tlsv1) ) disk-buffer( reliable(yes) mem-buf-size(1M) disk-buf-size(5M) qout-size(64) ) template("<$PRI> $FACILITY $ISODATE $HOST $PROGRAM $MSG\n") );}; syslog-ng 3.8.1Installer-Version: 3.8.1Revision:Module-Directory: /usr/lib/syslog-ngModule-Path: /usr/lib/syslog-ngAvailable-Modules: cef,affile,basicfuncs,system-source,cryptofuncs,graphite,pseudofile,afuser,kvformat,add-contextual-data,date,csvparser,linux-kmsg-format,confgen,syslogformat,afprog,disk-buffer,dbparser,afsotEnable-Debug: offEnable-GProf: offEnable-Memtrace: offEnable-IPv6: offEnable-Spoof-Source: offEnable-TCP-Wrapper: offEnable-Linux-Caps: off ======================= Thanks

1 0

syslog message without timestamp
by Csuti Istvan 23 Sep '16

23 Sep '16

Hi All! I would like send UDP message to the syslog server without timestamp. I readed RFC3164 and RFC5424 and syslog-ng-ose-guide-admin.pdf. The new "RFC 5424 (also called the IETF-syslog protocol): see Section 2.8.2, IETF-syslog messages (p. 13)" support NIL values "-" in HEADER. I sended this "<110>1 - hostname procname procid - - Message" The syslog-ng server logged this: "Sep 21 19:29:54 h164-19 1: - hostname procname procid - - Message" Question: What is wrong in my message string? The syslog-ng server why do not decode "1" after ">" as version? Can i send old syslog message (RFC3164) without timestamp? Best Regards Istvan Csuti

3 5

Listen backlog issues?
by Claus Albøge 19 Sep '16

19 Sep '16

Hi, I have ~6000 hosts sending syslog-TLS to a syslog-ng server. I’m seeing a lot of reconnects from the clients, and on the syslog-ng servers I get the following messages: TCP: request_sock_TCP: Possible SYN flooding on port 6514. Sending cookies. Check SNMP counters. netstat -s show the following: $ netstat -s | grep -i listen 10480620 times the listen queue of a socket overflowed 11144791 SYNs to LISTEN sockets dropped A few resources suggest this could be due to the applications socket listen backlog. Starting syslog-ng with strace reveals the following: 30067 listen(5, 255) = 0 30067 listen(11, 255) = 0 30067 listen(12, 255) = 0 30067 listen(17, 255) = 0 It seems like the backlog is hardcoded to 255 in the source. A bit more info: Saving the output from: “netstat -n | grep -c SYN_RECV” to a file a few times per second, shows the following: 2016-09-19 09:54:14.767469015: 0 2016-09-19 09:54:14.906638364: 0 2016-09-19 09:54:15.048591252: 0 2016-09-19 09:54:15.184325070: 0 2016-09-19 09:54:15.324150368: 0 2016-09-19 09:54:15.459749187: 0 2016-09-19 09:54:15.587983284: 0 2016-09-19 09:54:15.722558975: 42 2016-09-19 09:54:15.873601766: 256 2016-09-19 09:54:16.020145083: 247 2016-09-19 09:54:16.177231109: 178 2016-09-19 09:54:16.340875439: 178 2016-09-19 09:54:16.488506916: 178 2016-09-19 09:54:16.637815500: 176 2016-09-19 09:54:16.790781389: 149 2016-09-19 09:54:16.950254517: 141 2016-09-19 09:54:17.125316109: 131 2016-09-19 09:54:17.286839687: 131 2016-09-19 09:54:17.445583267: 127 2016-09-19 09:54:17.598144758: 127 2016-09-19 09:54:17.747967473: 124 2016-09-19 09:54:17.905359412: 2 2016-09-19 09:54:18.062665358: 2 2016-09-19 09:54:18.239973822: 0 2016-09-19 09:54:18.398021514: 0 2016-09-19 09:54:18.548074304: 0 This matches the entry in /var/log/messages Sep 19 09:54:15 log03 kernel: [1116811.081666] TCP: request_sock_TCP: Possible SYN flooding on port 6514. Sending cookies. Check SNMP counters. Are there any tunings in the syslog-ng configuration I need to implement, or do I have to compile syslog-ng myself with a larger listen backlog? Perhaps this should be configurable? My source definition for syslog-TLS is like this: source s_syslog_tls { syslog( ip(0.0.0.0) port(6514) log_fetch_limit(100) log_iw_size(11000000) max-connections(10000) transport("tls") tls( key-file("/etc/pki/tls/certs/logserver.key") cert-file("/etc/pki/tls/certs/logserver.crt") peer-verify(optional-untrusted) ) ); }; $ syslog-ng -V syslog-ng 3.8.1 Installer-Version: 3.8.1 Revision: Module-Directory: //usr/lib64/syslog-ng Module-Path: //usr/lib64/syslog-ng Available-Modules: disk-buffer,sdjournal,afstomp,json-plugin,cryptofuncs,graphite,cef,kvformat,add-contextual-data,dbparser,pseudofile,curl,csvparser,syslogformat,confgen,afsocket,afuser,date,linux-kmsg-format,system-source,basicfuncs,afamqp,affile,afprog Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-IPv6: on Enable-Spoof-Source: on Enable-TCP-Wrapper: on Enable-Linux-Caps: off syslog-ng-3.8.1-1.el7.centos.x86_64 from https://copr-be.cloud.fedoraproject.org/results/czanik/syslog-ng38/epel-7-x… Please let me know if more info is needed. /Claus A

3 2

syslog-ng in ubuntu 16.10
by Gil Bahat 15 Sep '16

15 Sep '16

Ubuntu 16.10 is now in feature freeze and there is (apparently) one bug on s390x platform that prevents v3.7.3 from making it into the release in lieu of the older 3.5.6. that's a real shame, I hope someone from this list can help out. here is the failing build log on ubuntu 16.10: https://launchpadlibrarian.net/281969183/buildlog_ubuntu- yakkety-s390x.syslog-ng-incubator_0.5.0-1_BUILDING.txt.gz this does build correctly on debian sid though: https://buildd.debian.org/status/fetch.php?pkg=syslog- ng-incubator&arch=s390x&ver=0.5.0-1&stamp=1468967484 Regards, Gil Bahat

2 1

syslog-ng 3.8.1 rpm packages
by Czanik, Péter 15 Sep '16

15 Sep '16

Hi, This week my packaging focus was to make sure, that 3.8.1 will be available as part of the next openSUSE release. My initial unofficial 3.8.1 RPM packages with Java modules, etc. are available but still need some polishing: - move curl (HTTP) module into a separate sub package - update rust modules and on Fedora / EPEL: - build Java modules without Internet access - re-enable missing features, like riemann (which were turned off to make local test builds easier) Fedora/EPEL7: https://copr.fedorainfracloud.org/coprs/czanik/syslog-ng38/ EPEL6: https://copr.fedorainfracloud.org/coprs/czanik/syslog-ng38epel6/ openSUSE/SLES: https://build.opensuse.org/project/show/home:czanik:syslog-ng38 I hope to have more time working on this by the end of next week. Any feedback, suggestions, patches are welcome Bye, Peter Czanik (CzP) <peter.czanik(a)balabit.com> Balabit / syslog-ng upstream http://czanik.blogs.balabit.com/ https://twitter.com/PCzanik

4 6

Insider 2016-09: 3.8.1 released; GSoC; Turris Omnia; meetups;
by Czanik, Péter 15 Sep '16

15 Sep '16

Dear syslog-ng users, This is the 51st issue of syslog-ng Insider, a monthly newsletter that brings you syslog-ng-related news. NEWS syslog-ng 3.8.1 released ------------------------ I am happy to announce that after almost a year of development, syslog-ng 3.8.1 has been released. This is the first stable release from the syslog-ng 3.8 branch. Version 3.8.1 adds some long-awaited features to syslog-ng and improves the performance of name-value-pair related operations. Announcement: https://www.balabit.com/blog/syslog-ng-hits-major-milestone-with-version-3-… Development, packaging related changes: https://czanik.blogs.balabit.com/2016/08/syslog-ng-3-8-what-changed/ Google Summer of Code 2016 -------------------------- The syslog-ng project has a long tradition of participating in the Google Summer of Code program. Each year some interesting new features originate from GSoC students. Here are some of the blogs, where students explain what they developed during the Summer: Websocket destination: https://litterbear.github.io/2016-08-26-GSOC2016/ Release automation: http://black-perl.in/gsoc16-syslog-ng-autorel-summary/ Kafka source: https://vithulanmv.wordpress.com/2016/08/21/syslog-ng-kafka-source-in-depth… As command line tool: https://kvch.github.io/syslog-ng-as-a-command-line-tool.html If you are at the GSoC mentor summit (Sunnyvale, California, October 28-30, https://sites.google.com/site/2016gsocmentorsummit/home) you can even meet one of our mentors, Viktor Juhász, at the event. As a syslog-ng developer, he can talk to you about syslog-ng, even get you started developing syslog-ng. Coming up: All Things Open -------------------------- My next syslog-ng presentation will be in Raleigh, NC, at the All Things Open conference. I will talk about scaling a logging infrastructure using syslog-ng: https://allthingsopen.org/talk/scaling-your-logging-infrastructure/ syslog-ng in the Turris Omnia Linux router ------------------------------------------ The Turris Omnia is a Linux-based router running on an ARM CPU. As it turned out during a conference discussion, this popular little device is running syslog-ng: https://czanik.blogs.balabit.com/2016/06/turris-omnia-a-linux-router-powere… If you know of anything that could be added to https://syslog-ng.org/powered-by-syslog-ng/, let me know! syslog-ng at meetups -------------------- There were two meetups this summer where a member of Balabit’s crew talked about syslog-ng. Both were quite a positive experience. If you know of a meetup where participants might be interested in syslog-ng, please let me know! I can be reached by e-mail or on Twitter as @PCzanik San Francisco: https://bazsi.blogs.balabit.com/2016/07/syslog-ng-in-san-francisco/ London: https://www.balabit.com/blog/linuxing-in-london/ NEW RELEASES syslog-ng 3.8.1: https://www.balabit.com/blog/syslog-ng-hits-major-milestone-with-version-3-… SSB 4F4: https://istvanszabo.blogs.balabit.com/2016/07/improve-the-efficiency-of-sec… Your feedback and news, or tips about the next issue are welcome at documentation(a)balabit.com. To read this newsletter online, visit: https://syslog-ng.org/ Peter Czanik (CzP) <peter.czanik(a)balabit.com> Balabit / syslog-ng upstream http://czanik.blogs.balabit.com/ https://twitter.com/PCzanik

1 0

pattern_db ES use case.
by Scot Needy 12 Sep '16

12 Sep '16

Hello List, I’m trying to understand the use case of pattern_db when the destination will be ES. My initial understanding was that I could use patterndb as an engine to tag my log message data with attributes, but it doesn’t seem to work that way. I have a json output like this in Kibana. In a loghost deployment, It looks like I would need to manually align a patterndb filter with each host_message type even before patterned comes into play. Q) What is the right solution for enriching message data into ES ? Example JSON from Kibana MESSAGE is not parsed. ======================= { "_index": "syslog-ng_2016.09.12", "_type": "syslog-ng", "_id": "AVcdnzJla9VjMdxDYo8Z", "_score": null, "_source": { "PROGRAM": “###-asa11", "PRIORITY": "warning", "MESSAGE": "%ASA-4-106023: Deny tcp src outside:###.###.31.2/33553 dst public:###.###.7.191/443 by access-group \"outside_access_in\" [0x2c1c6a65, 0x0]", "ISODATE": "2016-09-12T13:57:03-04:00", "HOST": “###.###.###.###", "FACILITY": "local5", "@timestamp": "2016-09-12T13:57:03-04:00" }, "fields": { "ISODATE": [ 1473703023000 ], "@timestamp": [ 1473703023000 ] }, "sort": [ 1473703023000 ] }

2 5

Archive module ?
by Scot Needy 12 Sep '16

12 Sep '16

Hi All, Is there a better way to archive my syslog data once it’s been received and injected into elk,spunk or whatever ? I see how I can log my data to flat files using date macros then compress and archive as normal data but is there a better way to write out the data in the same format it was received to facilitate re-injesting ? For example: My syslog data on file does not include the original syslog header data . Syslog is just one example.

2 6

Re: [syslog-ng] syslog-ng Digest, Vol 137, Issue 11
by Szabó, István 09 Sep '16

09 Sep '16

Hi Scot, syslog-ng in its premium edition offers logstore: https://www.balabit.com/documents/syslog-ng-pe-6.0-guides/en/syslog-ng-pe-g… Disclaimer: I'm employed by Balabit (being the product manager of syslog-ng there) I don't want to push anything here, the only reason I wrote this is because I felt there is an exact match with what you required and by checking the above documentation you can at least see whether this is indeed the functionality what you are after. /Istvan Date: Thu, 8 Sep 2016 21:56:55 -0400 > From: Scot Needy <scotrn(a)gmail.com> > Subject: Re: [syslog-ng] Archive module ? > To: Syslog-ng users' and developers' mailing list > <syslog-ng(a)lists.balabit.hu> > Message-ID: <3E22C1C9-9275-41F4-A583-F140CAF611A6(a)gmail.com> > Content-Type: text/plain; charset="utf-8" > > > That looks like it would work but if the source isn?t syslog data ? > > I pondered this question when looking at the disk buffer settings in 3.8. > If my data is in ES, MySQL or some other parsed presentation platform then > all I really need to write out blocks of raw data to disk and archive for > SOX compliance. > > Let the storage archive, compress, encrypt and de-dup. > > Just > > > > On Sep 8, 2016, at 5:48 PM, Evan Rempel <erempel(a)uvic.ca> wrote: > > > > You would have to use a syslog-ng template defined as > > > > "<$PRI>$ISO_DATE $FULLHOST $MSGHDR$MESSAGE\n" > > > > and that would be the "on the wire" format. The you can netcat this to > the syslog port which would replay the log. > > > > Evan. > > > > On 09/08/2016 02:39 PM, Scot Needy wrote: > >> Thanks Even, > >> > >> Maybe I?m doing something wrong. > >> I see some of the headers but not all. > >> > >> 2016-09-08T17:31:17-04:00 <IPADDR> <HOST> %ASA-3-305006: regular > translation creation failed for imp src ?. > >> > >> I guess what I was looking for is something that I could write blocks > of data to and read from irregardless of the type of data. > >> Something almost like tcpdump or buffer that could rotate every hour or > so and then be used to re-play directly back into syslog-ng. > >> > >> Log > YYYY.MM.DD.HH_compressed_data. > >> > >> > >> > >> > >> > >>> On Sep 8, 2016, at 5:13 PM, Evan Rempel <erempel(a)uvic.ca> <mailto: > erempel(a)uvic.ca> wrote: > >>> > >>> If you write the syslog data to a file with all of the data, then it > can be ingested again. > >>> > >>> We save the data to the files like > >>> > >>> 2016-09-08T00:00:01-07:00 host.name.here mail.info sm-mta[9521]: > u887019I009521: milter=mimedefang, action=rcpt, continue > >>> > >>> and then we have a syslog-ng pattern database that will turn this back > into an "on the wiire" format that can be sent to any syslog port. > >>> > >>> Alternatively, you could save it in the file as the original format, > and then it can just be sent back to syslog when you need to. Human > readablility is not quite as good, but if you have the data in splunk or > whatever, then humans don't need to read the raw files. > >>> > >>> Evan. > >>> > >>> > >>> On 09/08/2016 01:32 PM, Scot Needy wrote: > >>>> Hi All, > >>>> > >>>> Is there a better way to archive my syslog data once it?s been > received and injected into elk,spunk or whatever ? > >>>> > >>>> I see how I can log my data to flat files using date macros then > compress and archive as normal data but is there a better way to write out > the data in the same format it was received to facilitate re-injesting ? > >>>> > >>>> For example: My syslog data on file does not include the original > syslog header data . > >>>> > >>>> Syslog is just one example. > >>>> > >>>> > >>> ____________________________________________________________ > __________________ > >>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng < > https://lists.balabit.hu/mailman/listinfo/syslog-ng> > >>> Documentation: http://www.balabit.com/support/documentation/? > product=syslog-ng <http://www.balabit.com/support/documentation/? > product=syslog-ng> > >>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq < > http://www.balabit.com/wiki/syslog-ng-faq> > >>> > >> ____________________________________________________________ > __________________ > >> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng < > https://lists.balabit.hu/mailman/listinfo/syslog-ng> > >> Documentation: http://www.balabit.com/support/documentation/? > product=syslog-ng <http://www.balabit.com/support/documentation/? > product=syslog-ng> > >> FAQ: http://www.balabit.com/wiki/syslog-ng-faq < > http://www.balabit.com/wiki/syslog-ng-faq> > >> > > > > > > -- > > Evan Rempel erempel(a)uvic.ca > <mailto:erempel@uvic.ca> > > Senior Systems Administrator 250.721.7691 > > Data Centre Services, University Systems, University of Victoria > > ____________________________________________________________ > __________________ > > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng > > Documentation: http://www.balabit.com/support/documentation/? > product=syslog-ng > > FAQ: http://www.balabit.com/wiki/syslog-ng-faq > > > >

1 0

Help! CentOS 7 ELK Stack from repos no index data.
by Scot Needy 09 Sep '16

09 Sep '16

So, I rebuilt my home stand alone system and only used the supported 3.8 and ES 2.4 current repos. Everything seems to be up and running with no complaints but I am not getting any data in Kibana, syslog-ng_* index and I can’t figure out if it’s something in syslog-ng.conf. Getting a bunch of stuff in /var/log/network.log from my cable modem but nothing is showing in discovery HELP! I’ve looked at this for hours and tried so many variants on destination d_es client-mode, all supported template options… Only thing weird I can see is this in the es logs. data=false [2016-09-08 00:00:01,977][INFO ][cluster.metadata ] [node-1] [syslog-ng_2016.09.08] creating index, cause [auto(bulk api)], templates [], shards [5]/[1], mappings [] [2016-09-08 00:00:02,108][INFO ][cluster.routing.allocation] [node-1] Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[syslog-ng_2016.09.08][0], [syslog-ng_2016.09.08][0]] ...]). [2016-09-08 00:00:02,127][INFO ][cluster.metadata ] [node-1] [syslog-ng_2016.09.08] create_mapping [syslog-ng] [2016-09-08 00:15:28,886][INFO ][cluster.service ] [node-1] removed {{node-1}{kSmxkMoZQu6ZXvWWb70L5g}{127.0.0.1}{127.0.0.1:9301}{client=true, data=false},}, reason: zen-disco-node-left({node-1}{kSmxkMoZQu6ZXvWWb70L5g}{127.0.0.1}{127.0.0.1:9301}{client=true, data=false}), reason(left) [2016-09-08 00:15:39,358][INFO ][cluster.service ] [node-1] added {{node-1}{Df73oV_kQCubR5MeHiJnJA}{127.0.0.1}{127.0.0.1:9301}{client=true, data=false},}, reason: zen-disco-join(join from node[{node-1}{Df73oV_kQCubR5MeHiJnJA}{127.0.0.1}{127.0.0.1:9301}{client=true, data=false}]) [2016-09-08 00:15:44,409][INFO ][cluster.metadata ] [node-1] [syslog-ng_2016.09.08] update_mapping [syslog-ng] [2016-09-08 00:31:14,268][INFO ][cluster.service ] [node-1] removed {{node-1}{Df73oV_kQCubR5MeHiJnJA}{127.0.0.1}{127.0.0.1:9301}{client=true, data=false},}, reason: zen-disco-node-left({node-1}{Df73oV_kQCubR5MeHiJnJA}{127.0.0.1}{127.0.0.1:9301}{client=true, data=false}), reason(left) [2016-09-08 00:31:19,823][INFO ][cluster.service ] [node-1] added {{node-1}{4h_70582RoWKL-jsAPzF4g}{127.0.0.1}{127.0.0.1:9301}{client=true, data=false},}, reason: zen-disco-join(join from node[{node-1}{4h_70582RoWKL-jsAPzF4g}{127.0.0.1}{127.0.0.1:9301}{client=true, data=false}]) [2016-09-08 00:38:29,163][INFO ][cluster.service ] [node-1] removed {{node-1}{4h_70582RoWKL-jsAPzF4g}{127.0.0.1}{127.0.0.1:9301}{client=true, data=false},}, reason: zen-disco-node-left({node-1}{4h_70582RoWKL-jsAPzF4g}{127.0.0.1}{127.0.0.1:9301}{client=true, data=false}), reason(left) https://www.elastic.co/guide/en/elasticsearch/reference/current/setup-repos… <https://www.elastic.co/guide/en/elasticsearch/reference/current/setup-repos…> https://www.elastic.co/guide/en/kibana/4.6/setup-repositories.html <https://www.elastic.co/guide/en/kibana/4.6/setup-repositories.html> https://copr.fedorainfracloud.org/coprs/czanik/syslog-ng38/repo/epel-7/czan… <https://copr.fedorainfracloud.org/coprs/czanik/syslog-ng38/repo/epel-7/czan…> [root@meo syslog-ng]# cat /etc/syslog-ng/syslog-ng.conf @version:3.8 @include "scl.conf" @module mod-java options { flush_lines (0); time_reopen (10); log_fifo_size (1000); chain_hostnames (off); use_dns (no); use_fqdn (no); create_dirs (yes); keep_hostname (yes); ts_format(iso); }; source s_netsyslog { udp(); tcp(); syslog(); }; source s_sys { system(); internal(); }; destination d_es { elasticsearch2( index("syslog-ng_${YEAR}.${MONTH}.${DAY}") type("syslog-ng") # Description: The type of the index. For example, type("test") port("9300") server("127.0.0.1") concurrent-requests("5") flush_limit("1") client-mode("transport") skip-cluster-health-check("yes") cluster("meo") custom_id("syslog-ng") resource("/etc/elasticsearch/elasticsearch.yml") client_lib_dir("/usr/share/elasticsearch/lib") concurrent_requests("100") ); }; destination d_netsyslog { file("/var/log/network.log" owner("root") group("root") perm(0644)); }; log { source(s_netsyslog); destination(d_es); }; log { source(s_sys); source(s_netsyslog); destination(d_netsyslog); }; # Source additional configuration files (.conf extension only) @include "/etc/syslog-ng/conf.d/*.conf"

3 8

Tiered storage for syslog archives.
by Scot Needy 07 Sep '16

07 Sep '16

Hi, Has anyone here looked or would like to collaborate, using tiered storage solutions with their log solution? Specifically, I am looking to use elastic search bucket retention. I know this is a syslog-ng forum but the solution is for syslog archives with Kibana as the presentation layer. Welcome any open feedback or PM. Thanks

1 0

Re: [syslog-ng] Web Visualize logs (Ivan Adji - Krstev)
by Szabó, István 07 Sep '16

07 Sep '16

Hi, For using syslog-ng together with elasticsearch and Kibana this is a pretty good description, also giving you a good understanding of what it enables you to do: https://czanik.blogs.balabit.com/2015/10/how-to-parse-data-with-syslog-ng-s… https://czanik.blogs.balabit.com/2015/12/elasticsearch-and-syslog-ng-fast-a… /Istvan

9 24

3.8.1 CentOS 7 Error initializing message pipeline again!
by Scot Needy 07 Sep '16

07 Sep '16

Fighting with this all weekend and I need some fresh eyes. Source git snapshot 9/5 with --enable-spoof-source --enable-goip --enable-json=yes --enable-java --enable-systemd --enable-java-modules I get no missing libraries or modules on startup and have used every variation I could find on the conf file. My goal is to get a syslog-ng relay to also log to es2 but I’m beginning to think this error is more related to syslog-ng not being able to log to the CentOS 7 Journal ? /opt/syslog-ng/sbin/syslog-ng -V syslog-ng 3.8.1 Installer-Version: 3.8.1 Revision: Module-Directory: /opt/syslog-ng/lib/syslog-ng Module-Path: /opt/syslog-ng/lib/syslog-ng Available-Modules: dbparser,cryptofuncs,afsocket,date,afstomp,afmongodb,linux-kmsg-format,afsmtp,graphite,geoip-plugin,csvparser,kvformat,disk-buffer,cef,sdjournal,basicfuncs,pseudofile,afuser,afamqp,afprog,confgen,mod-java,curl,system-source,syslogformat,add-contextual-data,affile,json-plugin Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-IPv6: on Enable-Spoof-Source: on Enable-TCP-Wrapper: off Enable-Linux-Caps: off ##################### syslog-ng.conf simple ############### @version: 3.8 @module mod-java @include "scl.conf" source s_netsyslog { udp(ip(0.0.0.0) port(514) flags(no-hostname)); tcp(ip(0.0.0.0) port(514) flags(no-hostname)); }; ###### # Elasticsearch destination destination d_es { elasticsearch2( index("syslog-ng_${YEAR}.${MONTH}.${DAY}") type("test") ); }; log { source(s_netsyslog); destination(d_es); flags(flow-control); }; [root@syslog-ng]# /opt/syslog-ng/sbin/syslog-ng -Fevd -f /opt/syslog-ng/etc/syslog-ng.conf [2016-09-06T11:59:52.402514] Systemd is detected as the running init system; [2016-09-06T11:59:52.406577] Module loaded and initialized successfully; module='mod-java' [2016-09-06T11:59:52.406634] Starting to read include file; filename='/opt/syslog-ng/etc/scl.conf', depth='1' [2016-09-06T11:59:52.406954] Adding include file; filename='/opt/syslog-ng/share/syslog-ng/include/scl/apache/apache.conf', depth='2' [2016-09-06T11:59:52.406964] Adding include file; filename='/opt/syslog-ng/share/syslog-ng/include/scl/cim/template.conf', depth='2' [2016-09-06T11:59:52.406970] Adding include file; filename='/opt/syslog-ng/share/syslog-ng/include/scl/elasticsearch/plugin.conf', depth='2' [2016-09-06T11:59:52.406975] Adding include file; filename='/opt/syslog-ng/share/syslog-ng/include/scl/graphite/plugin.conf', depth='2' [2016-09-06T11:59:52.406980] Adding include file; filename='/opt/syslog-ng/share/syslog-ng/include/scl/hdfs/plugin.conf', depth='2' [2016-09-06T11:59:52.406985] Adding include file; filename='/opt/syslog-ng/share/syslog-ng/include/scl/kafka/plugin.conf', depth='2' [2016-09-06T11:59:52.406990] Adding include file; filename='/opt/syslog-ng/share/syslog-ng/include/scl/loggly/loggly.conf', depth='2' [2016-09-06T11:59:52.406995] Adding include file; filename='/opt/syslog-ng/share/syslog-ng/include/scl/logmatic/logmatic.conf', depth='2' [2016-09-06T11:59:52.407000] Adding include file; filename='/opt/syslog-ng/share/syslog-ng/include/scl/mbox/mbox.conf', depth='2' [2016-09-06T11:59:52.407004] Adding include file; filename='/opt/syslog-ng/share/syslog-ng/include/scl/nodejs/plugin.conf', depth='2' [2016-09-06T11:59:52.407009] Adding include file; filename='/opt/syslog-ng/share/syslog-ng/include/scl/pacct/plugin.conf', depth='2' [2016-09-06T11:59:52.407014] Adding include file; filename='/opt/syslog-ng/share/syslog-ng/include/scl/rewrite/cc-mask.conf', depth='2' [2016-09-06T11:59:52.407019] Adding include file; filename='/opt/syslog-ng/share/syslog-ng/include/scl/solaris/plugin.conf', depth='2' [2016-09-06T11:59:52.407024] Adding include file; filename='/opt/syslog-ng/share/syslog-ng/include/scl/syslogconf/plugin.conf', depth='2' [2016-09-06T11:59:52.407029] Adding include file; filename='/opt/syslog-ng/share/syslog-ng/include/scl/system/plugin.conf', depth='2' [2016-09-06T11:59:52.407041] Starting to read include file; filename='/opt/syslog-ng/share/syslog-ng/include/scl/apache/apache.conf', depth='2' [2016-09-06T11:59:52.407108] Reading path for candidate modules; path='/opt/syslog-ng/lib/syslog-ng' [2016-09-06T11:59:52.407160] Reading shared object for a candidate module; path='/opt/syslog-ng/lib/syslog-ng', fname='dbparser.so', module='dbparser' [2016-09-06T11:59:52.407443] Registering candidate plugin; module='dbparser', context='parser', name='db-parser', preference='0' [2016-09-06T11:59:52.407458] Registering candidate plugin; module='dbparser', context='parser', name='grouping-by', preference='0' [2016-09-06T11:59:52.407493] Reading shared object for a candidate module; path='/opt/syslog-ng/lib/syslog-ng', fname='cryptofuncs.so', module='cryptofuncs' [2016-09-06T11:59:52.407662] Registering candidate plugin; module='cryptofuncs', context='template-func', name='uuid', preference='0' [2016-09-06T11:59:52.407672] Registering candidate plugin; module='cryptofuncs', context='template-func', name='hash', preference='0' [2016-09-06T11:59:52.407678] Registering candidate plugin; module='cryptofuncs', context='template-func', name='sha1', preference='0' [2016-09-06T11:59:52.407684] Registering candidate plugin; module='cryptofuncs', context='template-func', name='sha256', preference='0' [2016-09-06T11:59:52.407690] Registering candidate plugin; module='cryptofuncs', context='template-func', name='sha512', preference='0' [2016-09-06T11:59:52.407696] Registering candidate plugin; module='cryptofuncs', context='template-func', name='md4', preference='0' [2016-09-06T11:59:52.407701] Registering candidate plugin; module='cryptofuncs', context='template-func', name='md5', preference='0' [2016-09-06T11:59:52.407725] Reading shared object for a candidate module; path='/opt/syslog-ng/lib/syslog-ng', fname='afsocket.so', module='afsocket' [2016-09-06T11:59:52.408068] Registering candidate plugin; module='afsocket', context='source', name='unix-stream', preference='100' [2016-09-06T11:59:52.408080] Registering candidate plugin; module='afsocket', context='destination', name='unix-stream', preference='100' [2016-09-06T11:59:52.408112] Registering candidate plugin; module='afsocket', context='source', name='unix-dgram', preference='100' [2016-09-06T11:59:52.408119] Registering candidate plugin; module='afsocket', context='destination', name='unix-dgram', preference='100' [2016-09-06T11:59:52.408130] Registering candidate plugin; module='afsocket', context='source', name='tcp', preference='100' [2016-09-06T11:59:52.408136] Registering candidate plugin; module='afsocket', context='destination', name='tcp', preference='100' [2016-09-06T11:59:52.408149] Registering candidate plugin; module='afsocket', context='source', name='tcp6', preference='100' [2016-09-06T11:59:52.408155] Registering candidate plugin; module='afsocket', context='destination', name='tcp6', preference='100' [2016-09-06T11:59:52.408161] Registering candidate plugin; module='afsocket', context='source', name='udp', preference='100' [2016-09-06T11:59:52.408166] Registering candidate plugin; module='afsocket', context='destination', name='udp', preference='100' [2016-09-06T11:59:52.408171] Registering candidate plugin; module='afsocket', context='source', name='udp6', preference='100' [2016-09-06T11:59:52.408193] Registering candidate plugin; module='afsocket', context='destination', name='udp6', preference='100' [2016-09-06T11:59:52.408200] Registering candidate plugin; module='afsocket', context='source', name='syslog', preference='100' [2016-09-06T11:59:52.408206] Registering candidate plugin; module='afsocket', context='destination', name='syslog', preference='100' [2016-09-06T11:59:52.408216] Registering candidate plugin; module='afsocket', context='source', name='network', preference='100' [2016-09-06T11:59:52.408222] Registering candidate plugin; module='afsocket', context='destination', name='network', preference='100' [2016-09-06T11:59:52.408228] Registering candidate plugin; module='afsocket', context='source', name='systemd-syslog', preference='100' [2016-09-06T11:59:52.408268] Reading shared object for a candidate module; path='/opt/syslog-ng/lib/syslog-ng', fname='date.so', module='date' [2016-09-06T11:59:52.408436] Registering candidate plugin; module='date', context='parser', name='date-parser', preference='0' [2016-09-06T11:59:52.408467] Reading shared object for a candidate module; path='/opt/syslog-ng/lib/syslog-ng', fname='afstomp.so', module='afstomp' [2016-09-06T11:59:52.408650] Registering candidate plugin; module='afstomp', context='destination', name='stomp', preference='0' [2016-09-06T11:59:52.408681] Reading shared object for a candidate module; path='/opt/syslog-ng/lib/syslog-ng', fname='afmongodb.so', module='afmongodb' [2016-09-06T11:59:52.409439] Registering candidate plugin; module='afmongodb', context='destination', name='mongodb', preference='0' [2016-09-06T11:59:52.409526] Reading shared object for a candidate module; path='/opt/syslog-ng/lib/syslog-ng', fname='linux-kmsg-format.so', module='linux-kmsg-format' [2016-09-06T11:59:52.409706] Registering candidate plugin; module='linux-kmsg-format', context='format', name='linux-kmsg', preference='0' [2016-09-06T11:59:52.409736] Reading shared object for a candidate module; path='/opt/syslog-ng/lib/syslog-ng', fname='afsmtp.so', module='afsmtp' [2016-09-06T11:59:52.410048] Registering candidate plugin; module='afsmtp', context='destination', name='smtp', preference='0' [2016-09-06T11:59:52.410099] Reading shared object for a candidate module; path='/opt/syslog-ng/lib/syslog-ng', fname='graphite.so', module='graphite' [2016-09-06T11:59:52.410320] Registering candidate plugin; module='graphite', context='template-func', name='graphite_output', preference='0' [2016-09-06T11:59:52.410385] Reading shared object for a candidate module; path='/opt/syslog-ng/lib/syslog-ng', fname='geoip-plugin.so', module='geoip-plugin' [2016-09-06T11:59:52.410718] Registering candidate plugin; module='geoip-plugin', context='parser', name='geoip', preference='0' [2016-09-06T11:59:52.410730] Registering candidate plugin; module='geoip-plugin', context='template-func', name='geoip', preference='0' [2016-09-06T11:59:52.410801] Reading shared object for a candidate module; path='/opt/syslog-ng/lib/syslog-ng', fname='csvparser.so', module='csvparser' [2016-09-06T11:59:52.411002] Registering candidate plugin; module='csvparser', context='parser', name='csv-parser', preference='0' [2016-09-06T11:59:52.411037] Reading shared object for a candidate module; path='/opt/syslog-ng/lib/syslog-ng', fname='kvformat.so', module='kvformat' [2016-09-06T11:59:52.411229] Registering candidate plugin; module='kvformat', context='parser', name='kv-parser', preference='0' [2016-09-06T11:59:52.411239] Registering candidate plugin; module='kvformat', context='parser', name='linux-audit-parser', preference='0' [2016-09-06T11:59:52.411246] Registering candidate plugin; module='kvformat', context='template-func', name='format-welf', preference='0' [2016-09-06T11:59:52.411274] Reading shared object for a candidate module; path='/opt/syslog-ng/lib/syslog-ng', fname='disk-buffer.so', module='disk-buffer' [2016-09-06T11:59:52.411433] Registering candidate plugin; module='disk-buffer', context='inner-dest', name='disk_buffer', preference='0' [2016-09-06T11:59:52.411470] Reading shared object for a candidate module; path='/opt/syslog-ng/lib/syslog-ng', fname='cef.so', module='cef' [2016-09-06T11:59:52.411623] Registering candidate plugin; module='cef', context='template-func', name='format-cef-extension', preference='0' [2016-09-06T11:59:52.411649] Reading shared object for a candidate module; path='/opt/syslog-ng/lib/syslog-ng', fname='sdjournal.so', module='sdjournal' [2016-09-06T11:59:52.411812] Registering candidate plugin; module='sdjournal', context='source', name='systemd-journal', preference='0' [2016-09-06T11:59:52.411846] Reading shared object for a candidate module; path='/opt/syslog-ng/lib/syslog-ng', fname='basicfuncs.so', module='basicfuncs' [2016-09-06T11:59:52.412045] Registering candidate plugin; module='basicfuncs', context='template-func', name='grep', preference='0' [2016-09-06T11:59:52.412056] Registering candidate plugin; module='basicfuncs', context='template-func', name='if', preference='0' [2016-09-06T11:59:52.412061] Registering candidate plugin; module='basicfuncs', context='template-func', name='or', preference='0' [2016-09-06T11:59:52.412067] Registering candidate plugin; module='basicfuncs', context='template-func', name='echo', preference='0' [2016-09-06T11:59:52.412073] Registering candidate plugin; module='basicfuncs', context='template-func', name='length', preference='0' [2016-09-06T11:59:52.412078] Registering candidate plugin; module='basicfuncs', context='template-func', name='substr', preference='0' [2016-09-06T11:59:52.412084] Registering candidate plugin; module='basicfuncs', context='template-func', name='strip', preference='0' [2016-09-06T11:59:52.412090] Registering candidate plugin; module='basicfuncs', context='template-func', name='sanitize', preference='0' [2016-09-06T11:59:52.412095] Registering candidate plugin; module='basicfuncs', context='template-func', name='lowercase', preference='0' [2016-09-06T11:59:52.412101] Registering candidate plugin; module='basicfuncs', context='template-func', name='uppercase', preference='0' [2016-09-06T11:59:52.412106] Registering candidate plugin; module='basicfuncs', context='template-func', name='replace-delimiter', preference='0' [2016-09-06T11:59:52.412112] Registering candidate plugin; module='basicfuncs', context='template-func', name='padding', preference='0' [2016-09-06T11:59:52.412142] Registering candidate plugin; module='basicfuncs', context='template-func', name='+', preference='0' [2016-09-06T11:59:52.412149] Registering candidate plugin; module='basicfuncs', context='template-func', name='-', preference='0' [2016-09-06T11:59:52.412154] Registering candidate plugin; module='basicfuncs', context='template-func', name='*', preference='0' [2016-09-06T11:59:52.412827] Registering candidate plugin; module='basicfuncs', context='template-func', name='/', preference='0' [2016-09-06T11:59:52.412839] Registering candidate plugin; module='basicfuncs', context='template-func', name='%', preference='0' [2016-09-06T11:59:52.412846] Registering candidate plugin; module='basicfuncs', context='template-func', name='sum', preference='0' [2016-09-06T11:59:52.412852] Registering candidate plugin; module='basicfuncs', context='template-func', name='min', preference='0' [2016-09-06T11:59:52.412858] Registering candidate plugin; module='basicfuncs', context='template-func', name='max', preference='0' [2016-09-06T11:59:52.412864] Registering candidate plugin; module='basicfuncs', context='template-func', name='average', preference='0' [2016-09-06T11:59:52.412870] Registering candidate plugin; module='basicfuncs', context='template-func', name='ipv4-to-int', preference='0' [2016-09-06T11:59:52.412876] Registering candidate plugin; module='basicfuncs', context='template-func', name='indent-multi-line', preference='0' [2016-09-06T11:59:52.412881] Registering candidate plugin; module='basicfuncs', context='template-func', name='context-length', preference='0' [2016-09-06T11:59:52.412887] Registering candidate plugin; module='basicfuncs', context='template-func', name='env', preference='0' [2016-09-06T11:59:52.412892] Registering candidate plugin; module='basicfuncs', context='template-func', name='template', preference='0' [2016-09-06T11:59:52.412931] Reading shared object for a candidate module; path='/opt/syslog-ng/lib/syslog-ng', fname='pseudofile.so', module='pseudofile' [2016-09-06T11:59:52.413122] Registering candidate plugin; module='pseudofile', context='destination', name='pseudofile', preference='0' [2016-09-06T11:59:52.413152] Reading shared object for a candidate module; path='/opt/syslog-ng/lib/syslog-ng', fname='afuser.so', module='afuser' [2016-09-06T11:59:52.413343] Registering candidate plugin; module='afuser', context='destination', name='usertty', preference='0' [2016-09-06T11:59:52.413383] Reading shared object for a candidate module; path='/opt/syslog-ng/lib/syslog-ng', fname='afamqp.so', module='afamqp' [2016-09-06T11:59:52.413619] Registering candidate plugin; module='afamqp', context='destination', name='amqp', preference='0' [2016-09-06T11:59:52.413658] Reading shared object for a candidate module; path='/opt/syslog-ng/lib/syslog-ng', fname='afprog.so', module='afprog' [2016-09-06T11:59:52.413834] Registering candidate plugin; module='afprog', context='source', name='program', preference='0' [2016-09-06T11:59:52.413845] Registering candidate plugin; module='afprog', context='destination', name='program', preference='0' [2016-09-06T11:59:52.413871] Reading shared object for a candidate module; path='/opt/syslog-ng/lib/syslog-ng', fname='confgen.so', module='confgen' [2016-09-06T11:59:52.414034] Reading shared object for a candidate module; path='/opt/syslog-ng/lib/syslog-ng', fname='mod-java.so', module='mod-java' [2016-09-06T11:59:52.414055] Registering candidate plugin; module='mod-java', context='destination', name='java', preference='0' [2016-09-06T11:59:52.414062] Reading shared object for a candidate module; path='/opt/syslog-ng/lib/syslog-ng', fname='curl.so', module='curl' [2016-09-06T11:59:52.416694] Registering candidate plugin; module='curl', context='destination', name='http', preference='0' [2016-09-06T11:59:52.416882] Reading shared object for a candidate module; path='/opt/syslog-ng/lib/syslog-ng', fname='system-source.so', module='system-source' [2016-09-06T11:59:52.417069] Reading shared object for a candidate module; path='/opt/syslog-ng/lib/syslog-ng', fname='syslogformat.so', module='syslogformat' [2016-09-06T11:59:52.417263] Registering candidate plugin; module='syslogformat', context='format', name='syslog', preference='0' [2016-09-06T11:59:52.417275] Registering candidate plugin; module='syslogformat', context='parser', name='syslog-parser', preference='0' [2016-09-06T11:59:52.417302] Reading shared object for a candidate module; path='/opt/syslog-ng/lib/syslog-ng', fname='add-contextual-data.so', module='add-contextual-data' [2016-09-06T11:59:52.417471] Registering candidate plugin; module='add-contextual-data', context='parser', name='add_contextual_data', preference='0' [2016-09-06T11:59:52.417502] Reading shared object for a candidate module; path='/opt/syslog-ng/lib/syslog-ng', fname='affile.so', module='affile' [2016-09-06T11:59:52.417664] Registering candidate plugin; module='affile', context='source', name='file', preference='0' [2016-09-06T11:59:52.417676] Registering candidate plugin; module='affile', context='source', name='pipe', preference='0' [2016-09-06T11:59:52.417682] Registering candidate plugin; module='affile', context='destination', name='file', preference='0' [2016-09-06T11:59:52.417689] Registering candidate plugin; module='affile', context='destination', name='pipe', preference='0' [2016-09-06T11:59:52.417714] Reading shared object for a candidate module; path='/opt/syslog-ng/lib/syslog-ng', fname='json-plugin.so', module='json-plugin' [2016-09-06T11:59:52.417957] Registering candidate plugin; module='json-plugin', context='parser', name='json-parser', preference='0' [2016-09-06T11:59:52.417973] Registering candidate plugin; module='json-plugin', context='template-func', name='format_json', preference='0' [2016-09-06T11:59:52.418122] Finishing include; filename='/opt/syslog-ng/share/syslog-ng/include/scl/apache/apache.conf', depth='2' [2016-09-06T11:59:52.418150] Starting to read include file; filename='/opt/syslog-ng/share/syslog-ng/include/scl/cim/template.conf', depth='2' [2016-09-06T11:59:52.418576] Module loaded and initialized successfully; module='json-plugin' [2016-09-06T11:59:52.418755] Finishing include; filename='/opt/syslog-ng/share/syslog-ng/include/scl/cim/template.conf', depth='2' [2016-09-06T11:59:52.418783] Starting to read include file; filename='/opt/syslog-ng/share/syslog-ng/include/scl/elasticsearch/plugin.conf', depth='2' [2016-09-06T11:59:52.419033] Finishing include; filename='/opt/syslog-ng/share/syslog-ng/include/scl/elasticsearch/plugin.conf', depth='2' [2016-09-06T11:59:52.419066] Starting to read include file; filename='/opt/syslog-ng/share/syslog-ng/include/scl/graphite/plugin.conf', depth='2' [2016-09-06T11:59:52.419154] Finishing include; filename='/opt/syslog-ng/share/syslog-ng/include/scl/graphite/plugin.conf', depth='2' [2016-09-06T11:59:52.419197] Starting to read include file; filename='/opt/syslog-ng/share/syslog-ng/include/scl/hdfs/plugin.conf', depth='2' [2016-09-06T11:59:52.419306] Finishing include; filename='/opt/syslog-ng/share/syslog-ng/include/scl/hdfs/plugin.conf', depth='2' [2016-09-06T11:59:52.419327] Starting to read include file; filename='/opt/syslog-ng/share/syslog-ng/include/scl/kafka/plugin.conf', depth='2' [2016-09-06T11:59:52.419429] Finishing include; filename='/opt/syslog-ng/share/syslog-ng/include/scl/kafka/plugin.conf', depth='2' [2016-09-06T11:59:52.419448] Starting to read include file; filename='/opt/syslog-ng/share/syslog-ng/include/scl/loggly/loggly.conf', depth='2' [2016-09-06T11:59:52.419579] Finishing include; filename='/opt/syslog-ng/share/syslog-ng/include/scl/loggly/loggly.conf', depth='2' [2016-09-06T11:59:52.419598] Starting to read include file; filename='/opt/syslog-ng/share/syslog-ng/include/scl/logmatic/logmatic.conf', depth='2' [2016-09-06T11:59:52.419744] Finishing include; filename='/opt/syslog-ng/share/syslog-ng/include/scl/logmatic/logmatic.conf', depth='2' [2016-09-06T11:59:52.419765] Starting to read include file; filename='/opt/syslog-ng/share/syslog-ng/include/scl/mbox/mbox.conf', depth='2' [2016-09-06T11:59:52.419840] Finishing include; filename='/opt/syslog-ng/share/syslog-ng/include/scl/mbox/mbox.conf', depth='2' [2016-09-06T11:59:52.419858] Starting to read include file; filename='/opt/syslog-ng/share/syslog-ng/include/scl/nodejs/plugin.conf', depth='2' [2016-09-06T11:59:52.419940] Finishing include; filename='/opt/syslog-ng/share/syslog-ng/include/scl/nodejs/plugin.conf', depth='2' [2016-09-06T11:59:52.419957] Starting to read include file; filename='/opt/syslog-ng/share/syslog-ng/include/scl/pacct/plugin.conf', depth='2' [2016-09-06T11:59:52.420034] Finishing include; filename='/opt/syslog-ng/share/syslog-ng/include/scl/pacct/plugin.conf', depth='2' [2016-09-06T11:59:52.420053] Starting to read include file; filename='/opt/syslog-ng/share/syslog-ng/include/scl/rewrite/cc-mask.conf', depth='2' [2016-09-06T11:59:52.420136] Global value changed; define='balabit.credit-card-regexp', value='(?P<1>:4[0-9]{12}(?:[0-9]{3})?|5[1-5][0-9]{14}|6(?:011|5[0-9][0-9])[0-9]{12}|3[47][0-9]{13}|3(?:0[0-5]|[68][0-9])[0-9]{11}|(?:2131|1800|35d{3})d{11})' [2016-09-06T11:59:52.420211] Finishing include; filename='/opt/syslog-ng/share/syslog-ng/include/scl/rewrite/cc-mask.conf', depth='2' [2016-09-06T11:59:52.420231] Starting to read include file; filename='/opt/syslog-ng/share/syslog-ng/include/scl/solaris/plugin.conf', depth='2' [2016-09-06T11:59:52.420321] Finishing include; filename='/opt/syslog-ng/share/syslog-ng/include/scl/solaris/plugin.conf', depth='2' [2016-09-06T11:59:52.420341] Starting to read include file; filename='/opt/syslog-ng/share/syslog-ng/include/scl/syslogconf/plugin.conf', depth='2' [2016-09-06T11:59:52.420588] Module loaded and initialized successfully; module='confgen' [2016-09-06T11:59:52.420603] Finishing include; filename='/opt/syslog-ng/share/syslog-ng/include/scl/syslogconf/plugin.conf', depth='2' [2016-09-06T11:59:52.420623] Starting to read include file; filename='/opt/syslog-ng/share/syslog-ng/include/scl/system/plugin.conf', depth='2' [2016-09-06T11:59:52.420858] Module loaded and initialized successfully; module='system-source' [2016-09-06T11:59:52.420873] Finishing include; filename='/opt/syslog-ng/share/syslog-ng/include/scl/system/plugin.conf', depth='2' [2016-09-06T11:59:52.420889] Finishing include; filename='/opt/syslog-ng/etc/scl.conf', depth='1' [2016-09-06T11:59:52.421207] Module loaded and initialized successfully; module='afsocket' [2016-09-06T11:59:52.421558] Finishing include; content='destination block elasticsearch2', depth='1' [2016-09-06T11:59:52.421853] Compiling #unnamed sequence [log] at [/opt/syslog-ng/etc/syslog-ng.conf:35:2] [2016-09-06T11:59:52.421862] Compiling s_netsyslog reference [source] at [/opt/syslog-ng/etc/syslog-ng.conf:35:2] [2016-09-06T11:59:52.421867] Compiling s_netsyslog sequence [source] at [/opt/syslog-ng/etc/syslog-ng.conf:15:1] [2016-09-06T11:59:52.421892] Compiling #unnamed junction [log] at [/opt/syslog-ng/etc/syslog-ng.conf:15:21] [2016-09-06T11:59:52.421897] Compiling #unnamed single [log] at [/opt/syslog-ng/etc/syslog-ng.conf:16:9] [2016-09-06T11:59:52.421903] Compiling #unnamed single [log] at [/opt/syslog-ng/etc/syslog-ng.conf:17:9] [2016-09-06T11:59:52.421910] Compiling d_es reference [destination] at [/opt/syslog-ng/etc/syslog-ng.conf:36:2] [2016-09-06T11:59:52.421915] Compiling d_es sequence [destination] at [/opt/syslog-ng/etc/syslog-ng.conf:27:1] [2016-09-06T11:59:52.421918] Compiling #unnamed junction [log] at [/opt/syslog-ng/etc/syslog-ng.conf:27:19] [2016-09-06T11:59:52.421922] Compiling #unnamed single [log] at [#buffer:2:3] [2016-09-06T11:59:52.422119] Module loaded and initialized successfully; module='syslogformat' [2016-09-06T11:59:52.514371] Add path to classpath: /opt/syslog-ng/lib/syslog-ng/java-modules/syslog-ng-core.jar; [2016-09-06T11:59:52.514816] Add path to classpath: /opt/syslog-ng/lib/syslog-ng/java-modules/elastic.jar; [2016-09-06T11:59:52.514966] Add path to classpath: /opt/syslog-ng/lib/syslog-ng/java-modules/kafka.jar; [2016-09-06T11:59:52.515107] Add path to classpath: /opt/syslog-ng/lib/syslog-ng/java-modules/syslog-ng-core.jar; [2016-09-06T11:59:52.515272] Add path to classpath: /opt/syslog-ng/lib/syslog-ng/java-modules/hdfs.jar; [2016-09-06T11:59:52.515418] Add path to classpath: /opt/syslog-ng/lib/syslog-ng/java-modules/log4j-1.2.16.jar; [2016-09-06T11:59:52.515537] Add path to classpath: /opt/syslog-ng/lib/syslog-ng/java-modules/http.jar; [2016-09-06T11:59:52.515639] Add path to classpath: /opt/syslog-ng/lib/syslog-ng/java-modules/syslog-ng-common.jar; [2016-09-06T11:59:52.515752] Add path to classpath: /opt/syslog-ng/lib/syslog-ng/java-modules/dummy.jar; [2016-09-06T11:59:52.515850] Add path to classpath: /opt/syslog-ng/lib/syslog-ng/java-modules/elastic-v2.jar; [2016-09-06T11:59:52.515944] Add path to classpath: /opt/syslog-ng/lib/syslog-ng/java-modules/elastic-jest-client/commons-lang3-3.4.jar; [2016-09-06T11:59:52.516053] Add path to classpath: /opt/syslog-ng/lib/syslog-ng/java-modules/elastic-jest-client/httpcore-nio-4.4.4.jar; [2016-09-06T11:59:52.516159] Add path to classpath: /opt/syslog-ng/lib/syslog-ng/java-modules/elastic-jest-client/httpasyncclient-4.1.1.jar; [2016-09-06T11:59:52.516286] Add path to classpath: /opt/syslog-ng/lib/syslog-ng/java-modules/elastic-jest-client/httpclient-4.5.2.jar; [2016-09-06T11:59:52.516423] Add path to classpath: /opt/syslog-ng/lib/syslog-ng/java-modules/elastic-jest-client/slf4j-simple-1.7.13.jar; [2016-09-06T11:59:52.516541] Add path to classpath: /opt/syslog-ng/lib/syslog-ng/java-modules/elastic-jest-client/commons-codec-1.9.jar; [2016-09-06T11:59:52.516654] Add path to classpath: /opt/syslog-ng/lib/syslog-ng/java-modules/elastic-jest-client/gson-2.6.2.jar; [2016-09-06T11:59:52.516759] Add path to classpath: /opt/syslog-ng/lib/syslog-ng/java-modules/elastic-jest-client/jest-2.0.2.jar; [2016-09-06T11:59:52.516898] Add path to classpath: /opt/syslog-ng/lib/syslog-ng/java-modules/elastic-jest-client/slf4j-api-1.7.13.jar; [2016-09-06T11:59:52.517034] Add path to classpath: /opt/syslog-ng/lib/syslog-ng/java-modules/elastic-jest-client/httpcore-4.4.4.jar; [2016-09-06T11:59:52.517165] Add path to classpath: /opt/syslog-ng/lib/syslog-ng/java-modules/elastic-jest-client/guava-19.0.jar; [2016-09-06T11:59:52.517317] Add path to classpath: /opt/syslog-ng/lib/syslog-ng/java-modules/elastic-jest-client/jest-common-2.0.2.jar; [2016-09-06T11:59:52.517427] Add path to classpath: /opt/syslog-ng/lib/syslog-ng/java-modules/elastic-jest-client/commons-logging-1.2.jar; [2016-09-06T11:59:52.558918] Add path to classpath: /opt/syslog-ng/lib/syslog-ng/java-modules/syslog-ng-core.jar; [2016-09-06T11:59:52.562470] Error initializing message pipeline; syslog-ng]# ldd /opt/syslog-ng/sbin/syslog-ng linux-vdso.so.1 => (0x00007fff3237a000) libsyslog-ng-3.8.so.0 => /opt/syslog-ng/lib/libsyslog-ng-3.8.so.0 (0x00007f2fd6537000) libsystemd.so.0 => /lib64/libsystemd.so.0 (0x00007f2fd650e000) libssl.so.10 => /lib64/libssl.so.10 (0x00007f2fd62a1000) libcrypto.so.10 => /lib64/libcrypto.so.10 (0x00007f2fd5eb9000) librt.so.1 => /lib64/librt.so.1 (0x00007f2fd5cb0000) libnsl.so.1 => /lib64/libnsl.so.1 (0x00007f2fd5a97000) libgmodule-2.0.so.0 => /lib64/libgmodule-2.0.so.0 (0x00007f2fd5893000) libgthread-2.0.so.0 => /lib64/libgthread-2.0.so.0 (0x00007f2fd5690000) libglib-2.0.so.0 => /lib64/libglib-2.0.so.0 (0x00007f2fd5359000) libevtlog.so.0 => /opt/syslog-ng/lib/libevtlog.so.0 (0x00007f2fd5154000) libpcre.so.1 => /lib64/libpcre.so.1 (0x00007f2fd4ef2000) libdl.so.2 => /lib64/libdl.so.2 (0x00007f2fd4cee000) libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f2fd4ad2000) libc.so.6 => /lib64/libc.so.6 (0x00007f2fd470f000) /lib64/ld-linux-x86-64.so.2 (0x00007f2fd67f2000) libcap.so.2 => /lib64/libcap.so.2 (0x00007f2fd450a000) libm.so.6 => /lib64/libm.so.6 (0x00007f2fd4208000) libselinux.so.1 => /lib64/libselinux.so.1 (0x00007f2fd3fe2000) liblzma.so.5 => /lib64/liblzma.so.5 (0x00007f2fd3dbd000) libgcrypt.so.11 => /lib64/libgcrypt.so.11 (0x00007f2fd3b3c000) libgpg-error.so.0 => /lib64/libgpg-error.so.0 (0x00007f2fd3936000) libresolv.so.2 => /lib64/libresolv.so.2 (0x00007f2fd371c000) libdw.so.1 => /lib64/libdw.so.1 (0x00007f2fd34d5000) libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f2fd32be000) libgssapi_krb5.so.2 => /lib64/libgssapi_krb5.so.2 (0x00007f2fd3072000) libkrb5.so.3 => /lib64/libkrb5.so.3 (0x00007f2fd2d8d000) libcom_err.so.2 => /lib64/libcom_err.so.2 (0x00007f2fd2b88000) libk5crypto.so.3 => /lib64/libk5crypto.so.3 (0x00007f2fd2956000) libz.so.1 => /lib64/libz.so.1 (0x00007f2fd2740000) libattr.so.1 => /lib64/libattr.so.1 (0x00007f2fd253a000) libelf.so.1 => /lib64/libelf.so.1 (0x00007f2fd2324000) libbz2.so.1 => /lib64/libbz2.so.1 (0x00007f2fd2113000) libkrb5support.so.0 => /lib64/libkrb5support.so.0 (0x00007f2fd1f04000) libkeyutils.so.1 => /lib64/libkeyutils.so.1 (0x00007f2fd1d00000

2 1

Is it possible to append a new jSON field?
by Jorge Pereira 05 Sep '16

05 Sep '16

Hi team, Currently, I receive a jSON log from X, but I would like to append a new field. is it possible? e.g: I am trying to do something like: log { source { file("/var/log/app.jsonlog" program_override("ng_app") follow_freq(1) flags(no-parse) template("$(format-json --pair newfield=\"value\")"); ); }; destination(d_remote_collector); };

3 3

Problem with syslog-ng version 2
by Alarie, Maxime 05 Sep '16

05 Sep '16

OS: SLes 11 sp4 syslog-ng: syslog-ng-2.0.9-27.34.39.2 Hi, syslog-ng is configured to read from a application log file and then send it to another file in /var/log (tcpdump is not installed, I cant install it on a blackbox so this is how I test my config) I have the following in my syslog-ng configuration file: source ESRS { file("/opt/esrsve/gateway/xGate.log"); }; destination esrsfile{ file("/var/log/max.log" );}; log { source(ESRS); destination(esrsfile); }; The problem I am seeing is that when syslog writes to the outputfile (esrsfile) it truncatedthe lines. ex: Source File : [ 0, 6, 1, 1007] 08-29-2016 13:56:28.703 IMPORTANT INFO EDDEMC: Data Item::PMStatus Current Value::Offline And the destination file looiks like this: ... ... Aug 29 14:00:02 hostname C Aug 29 14:00:02 hostname u Aug 29 14:00:02 hostname r Aug 29 14:00:02 hostname r Aug 29 14:00:02 hostname e Aug 29 14:00:02 hostname n Aug 29 14:00:02 hostname t Aug 29 14:00:02 hostname Aug 29 14:00:02 hostname V Aug 29 14:00:02 hostname a Aug 29 14:00:02 hostname l Aug 29 14:00:02 hostname u Aug 29 14:00:02 hostname e Aug 29 14:00:02 hostname : Aug 29 14:00:02 hostname : Aug 29 14:00:02 hostname O Aug 29 14:00:02 hostname f Aug 29 14:00:02 hostname f Aug 29 14:00:02 hostname l Aug 29 14:00:02 hostname i Aug 29 14:00:02 hostname n Aug 29 14:00:02 hostname e ... ... What am I doing wrong here? Thanks much!

4 6

Is it possible to append a new jSON field?
by Jorge Pereira 04 Sep '16

04 Sep '16

Hi team, Currently, I receive a jSON log from X, but I would like to append a new field. is it possible? -- Jorge Pereira

1 0

[GSoC 2016] Kafka source in Java
by Vithulan MV 01 Sep '16

01 Sep '16

Hi all, I'd like to take this moment to Thank everyone for all the support and guidance provided to me throughout this period. Specially for my mentor Viktor Juhász and of course to the syslog-ng community! This was a wonderful journey with lot of experiences. Blogpost about Kafka source : https://vithulanmv.wordpress.com/2016/08/21/syslog-ng-kafka-source-in-depth… I am hoping to continue my contribution to syslog-ng project :) Thank you. Best regards, Vithulan. -- Vithulan MV. Undergraduate, Department of Computer Science & Engineering, University of Moratuwa, Sri lanka.

1 0

Problems with syslog-ng 3.7.3 + mod_confgen
by Jorge Pereira 01 Sep '16

01 Sep '16

Hi guys! Following the sample described in https://www.balabit.com/documents/syslog-ng-ose-latest-guides/en/syslog-ng-… 1) I have my 'confgen' script that prints the below *file()* entries. (p.s: these files has content.) # /etc/syslog-ng/scripts/confgen-modsec-skeleton.sh file("/opt/nginx/logs/waf/www.cocada.com" program_override("ng_modsec") flags(no-parse)); file("/opt/nginx/logs/waf/www.caipirinha.com" program_override("ng_modsec") flags(no-parse)); # 2) My config set: # cat /etc/syslog-ng/conf.d/nginx_modsec.conf options { threaded(yes); flush_lines(0); use-dns(no); normalize-hostnames(yes); keep-hostname(yes); }; destination d_collector { tcp("192.168.1.248" port(514) keep-alive(on) ); }; log { @module confgen context(source) name(s_nginx_modsec_log) exec("/etc/syslog-ng/scripts/confgen-modsec-skeleton.sh") destination(d_collector); }; # Conclusion: The syslog-ng doesn't call the script at any time. # strace -fff /usr/sbin/syslog-ng -dvte 2>&1 | grep "confgen-modsec" p.s: I have 'confgen' support. # syslog-ng --version | grep confgen Available-Modules: syslogformat,kvformat,afamqp,sdjournal,system-source,afuser,json-plugin,dbparser,affile,afsocket,linux-kmsg-format,afmongodb,mod-python, *confgen* ,csvparser,pseudofile,afsql,afprog,afstomp,cryptofuncs,graphite,basicfuncs # I appreciate any help. Best, Jorge Pereira

3 4