- syslog-ng - lists.balabit.hu

Re: [syslog-ng] Directory/file create ownership
by Dwijadas Dey 07 Mar '18

07 Mar '18

Add create_dirs (yes) in the options section. On 27-Feb-2018 2:24 PM, "Machiel Richards" <machiel.richards(a)gmail.com> wrote: Good day guys I am having an issue that I seem not be able to resolve thus far. I am fairly new to syslog-ng and have setup a new syslog server. We have managed to get most of our servers sending logs to the server without issues, however I am having trouble to get syslog-ng to create the directories and files with the correct ownership. We have a seperate user and group which it needs to create directories with, however it keeps on creating directories and files as root user. below is my current config : # First, set some global options. options { chain_hostnames(off); flush_lines(0); use_dns(yes); use_fqdn(yes); owner("syslog"); group("syslog"); perm(0640); stats_freq(0); bad_hostname("^gconfd$"); log_fifo_size(10000); }; source inputs { system(); internal(); unix-stream("/dev/log"); tcp(max_connections(50)); udp(); }; destination logpile { file("/var/log/HOSTS/$HOST/$YEAR/$MONTH/$FACILITY.$YEAR$ MONTH$DAY" owner("syslog") group("syslog") perm(0660) create_dirs(yes) dir_perm(0770)); }; log { source(inputs); destination(logpile); }; Can someone please assist me as any help would be appreciated. Regard -- I am a bomb technician...If you see me running, TRY TO KEEP UP! ____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/? product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq

5 8

Re: [syslog-ng] Cisco ASA parsing with patterndb/elasticsearch
by Scheidler, Balázs 05 Mar '18

05 Mar '18

Hi, Could you pls send a couple of messages? I would really like to make sure this works out of the box. Thanks On Mar 1, 2018 16:08, "Scheidler, Balázs" <balazs.scheidler(a)balabit.com> wrote: Can you give me a few inbound logs as received by syslog-ng? I would try to fix up cisco parser that it works for your use-case, as it should. Thanks On Feb 28, 2018 21:48, "Tim Ghetti" <tghetti(a)targetedsupport.com> wrote: I tried using the cisco-parser and not having any luck with that either. When I enable the parser, I actually stop seeing outbound traffic to elasticsearch. My config is below log { source { udp(flags(no-parse)); }; parser { cisco-parser(); }; destination { elasticsearch2( client-mode("http") cluster("ITESCL001") index("logstash-syslogng_${YEAR}.${MONTH}.${DAY}") cluster-url("http://192.168.101.199:9200 http://192.168.101.198:9200") type("syslog") flush-limit("1")); }; } [root@ITLOG001 conf.d]# tcpdump -nnSXi ens192 port 9200 -vv tcpdump: listening on ens192, link-type EN10MB (Ethernet), capture size 262144 bytes ^C 0 packets captured 0 packets received by filter 0 packets dropped by kernel *From:* syslog-ng [mailto:syslog-ng-bounces@lists.balabit.hu] *On Behalf Of *Scheidler, Balázs *Sent:* Tuesday, February 27, 2018 11:06 PM *To:* Syslog-ng users' and developers' mailing list < syslog-ng(a)lists.balabit.hu> *Subject:* Re: [syslog-ng] Cisco ASA parsing with patterndb/elasticsearch One additional feature you might be interested in is our cisco-parser() feature. It basically expects a flags(no-parse) input and then puts things in the right buckets, while recognising the various formats that cisco uses. It would be appreciated if you could check if it works with your setup and I can help fixing it up if it doesn't. The parser got added to recent syslog-ng versions, so you might need to upgrade to a new version. Bazsi On Feb 27, 2018 22:29, "Tim Ghetti" <tghetti(a)targetedsupport.com> wrote: Thanks!! It sounds like this is part of my problem. If you would be so kind as to send an example (or point me in the right direction) of how you are changing the program name and message body for the cisco logs that would be extremely helpful. I also suspect that there may be another issue. I’m almost certain that my parser should be picking up at least some of the logs but I am not seeing any of the fields in ES showing up based on parsed logs. Assuming that patterndb is working, would the below config be correct for getting patterndb parsed variables to show as fields in ES? destination d_elastic { elasticsearch2( client-mode("http") cluster("ITESCL001") index("logstash-syslogng_${YEAR}.${MONTH}.${DAY}") cluster-url("http://X.X.X.X:9200") type("syslog") flush-limit("1") ); }; *From:* syslog-ng [mailto:syslog-ng-bounces@lists.balabit.hu] *On Behalf Of *Evan Rempel *Sent:* Tuesday, February 27, 2018 9:06 PM *To:* syslog-ng(a)lists.balabit.hu *Subject:* Re: [syslog-ng] Cisco ASA parsing with patterndb/elasticsearch The program name will have an implied @ANYSTRING@ on the end, so if the syslog payload could be parsed to detect the program name of %ASA..... then your patterndb.xml would work. The problem is that they syslog payload can not be parsed. What we do is use a hand crafted patterndb to detect all of the different problem formats that Cisco logs (in our environment) and change a) the PROGRAM to be cisco_ASA or generally cisco_XXX where the XXX is the leading characters of the %XXX-#-##### of the syslog body. b) the MESSAGE to be %XXX-#-####... for the rest of the line. All of the poor hosts, sequence numbers, timestamps etc are all thrown away. After that, the internal buffers of syslog-ng contain a usable PROGRAM and MESSAGE such that we can use a patterndb to match the message part of the log line. This does mean that we have two patterndb parsers for every log line but it seems to work well for us. What would work better is if Cisco would fix their logging, but that isn't going to happen in my lifetime :-( I hope that makes sense. Evan. On 02/27/2018 05:37 PM, Tim Ghetti wrote: Ok, that makes sense. I sort of suspected that this was due to the program name not matching, since pdbtool works when specifying the program name. Do you know if there is a way to configure patterndb so that it matches the program name, regardless of the full program name with event code? Is it possible to regex the program or use the patterndb format i.e. <pattern>%ASA@ANYSTRING::@</pattern> or something similar? Tim *From:* syslog-ng [mailto:syslog-ng-bounces@lists.balabit.hu <syslog-ng-bounces(a)lists.balabit.hu>] *On Behalf Of *Evan Rempel *Sent:* Tuesday, February 27, 2018 8:26 PM *To:* syslog-ng(a)lists.balabit.hu *Subject:* Re: [syslog-ng] Cisco ASA parsing with patterndb/elasticsearch Welcome to the horrible world of Cisco logging :-( The issue you are bumping into is that when these log lines are parsed by syslog-ng (or any syslog daemon that does not specifically understand Cisco logs) there is no program name, or the program name is the full %ASA-4-106023 part of the log line. There are so many ways that Cisco can log incorrectly it is not possible to say without seeing your exact log lines. Cisco can add a * to the date/time stamp to indicate that there is no time server configured on the device. That makes the date/time invalid as far as parsing is concerned. Cisco can add a period (.) to the date/time stamp to indicate that there is a time server configured on the device, but the time server can not be reached. Again, this makes the date/time invalid as far as parsing is concerned. Cisco can add a sequence number at the start of the log line rather than starting the line with a date/6time stamp. Invalid parsing again. Cisco can leave out the sequence number but still include the trailing colon from the sequence number. You get the idea. So many ways to get it wrong, and they never get it right :-( Evan. On 02/27/2018 04:46 PM, Tim Ghetti wrote: Hi – having some trouble getting paterndb functional and looking for some help. I would like to use patterndb to parse my cisco ass firewall logs before sending it to elasticsearch. However when the messages get to elasticsearch, I don’t see the messages being parsed. Running pdbtool against the logs seems to work. # pdbtool match -p /etc/syslog-ng/patterndb.d/ciscoasa.pdb -P %ASA -f /var/log/asatest.log |more HOST=X.X.X.X MESSAGE=Built dynamic TCP translation from INSIDE:X.X.X.X/X to OUTSIDE:X.X.X.X/X PROGRAM=%ASA-6-305011 LEGACY_MSGHDR=%ASA-6-305011: .classifier.class=system .classifier.rule_id=e075efdc-c25f-5e49-a208-7661e3b5a29b Protocol=TCP GlobalIP=X.X.X.X GlobalPort=X LocalIP=X.X.X.X LocalPort=X TAGS=.classifier.system ********************** SYSLOG-NG CONF FILE @version: 3.11 source s_network { tcp(); udp(); }; destination d_elastic { elasticsearch2( client-mode("http") cluster("ITESCL001") index("logstash-syslogng_${YEAR}.${MONTH}.${DAY}") cluster-url("http://X.X.X.X:9200") type("syslog") flush-limit("1") ); }; destination d_catchall { file("/var/log/catchall.log"); }; filter f_ciscoasa { host("X.X.X.X"); }; parser p_ciscoasa {db-parser(file("/etc/syslog-n g/patterndb.d/ciscoasa.pdb"));}; log { source(s_network); filter(f_ciscoasa); parser(p_ciscoasa); destination(d_elastic); flags(final, flow-control); }; log { source(s_network); destination(d_catchall); }; ********************** PATTERNDB FILE <?xml version='1.0' encoding='UTF-8'?> <patterndb version='4' pub_date='2018-02-19'> <ruleset name='%ASA' id='a300d776-8bd7-834d-a4a9-23eb81a4b3ba'> <pattern>%ASA</pattern> <description> This ruleset covers the Cisco ASA firewalls </description> <rules> <rule provider="%ASA" id="b3de7699-8213-c744-944e-9413298afe86" class="system">  <patterns> <pattern>Teardown @ESTRING:Protocol: @connection for faddr @IPv4:SrcIP:/@@ESTRING:SrcPort: @gaddr @IPv4:GlobalIP:/@@ESTRING:GlobalPort: @laddr @IPv4:LocalIP:/@@ESTRING:LocalPort:@</pattern> </patterns> <examples> <example> <test_message program='%ASA'>Teardown ICMP connection for faddr X.X.X.X/X gaddr X.X.X.X/X laddr X.X.X.X/X</test_message> </example> </examples> </rule> <rule id='90d0f8c9-7591-d44e-b886-2f7e5cb17ce6' class='system' provider='%ASA'>  <patterns> <pattern>Teardown dynamic @ESTRING:Protocol: @translation from @ESTRING:::@@IPv4:LocalIP:/@@ESTRING:LocalPort: @to @ESTRING:::@ @IPv4:GlobalIP:/@@ESTRING:GlobalPort: @duration@ANYSTRING::@</pattern> </patterns> <examples> <example> <test_message program='%ASA'>Teardown dynamic UDP translation from any:X.X.X.X/X to outside:X.X.X.X/X duration 0:00:00</test_message> </example> </examples> </rule> <rule id='8f0a8d57-80c6-4745-8a8a-5ce018bb0d87' class='system' provider='%ASA'>  <patterns> <pattern>Teardown @ESTRING:Protocol: @connection @ESTRING:: @for @ESTRING:::@@IPv4:DstIP:/@@ESTRING:DstPort: @to @ESTRING:::@ @IPv4:SrcIP:/@@ESTRING:SrcPort: @@ESTRING::@</pattern> </patterns> <examples> <example> <test_message program='%ASA'>Teardown UDP connection 55101037 <(55)%20101%20037> for outside:X.X.X.X/X to inside:X.X.X.X/X duration 0:00:00 bytes 132</test_message> </example> </examples> </rule> <rule id='00c0732d-1e34-7340-a75f-21198bf71137' class='system' provider='%ASA'>  <patterns> <pattern>Built outbound @ESTRING:Protocol: @connection @ESTRING:: @for @ESTRING:::@@IPv4:DstIP:/@@ESTRING:DstPort: @(@ESTRING::)@ to @ESTRING:::@@IPv4:SrcIP:/@@ESTRING:SrcPort: @(@ESTRING::)@</pattern> </patterns> <examples> <example> <test_message program='%ASA'>Built outbound UDP connection 55101037 <(55)%20101%20037> for outside:X.X.X.X/X (X.X.X.X/X) to inside:X.X.X.X/X (X.X.X.X/X)</test_message> </example> </examples> </rule> <rule id='4a586711-ebe2-dc4d-bf6e-e512666d8c5d' class='system' provider='%ASA'>  <patterns> <pattern>Built inbound @ESTRING:Protocol: @connection for faddr @IPv4:SrcIP:/@@ESTRING:SrcPort: @gaddr @IPv4:GlobalIP:/@@ESTRING:GlobalPort: @laddr @IPv4:LocalIP:/@@ESTRING:LocalPort:@</pattern> </patterns> <examples> <example> <test_message program='%ASA'>Built inbound ICMP connection for faddr X.X.X.X/X gaddr X.X.X.X/X laddr X.X.X.X/X</test_message> </example> </examples> </rule> <rule id='8be7928d-66e7-7042-abd5-869d6b49c56e' class='system' provider='%ASA'>  <patterns> <pattern>Built inbound @ESTRING:Protocol: @connection @ESTRING:: @for @ESTRING::@@IPv4:SrcIP:/@@ESTRING:SrcPort: @(@ESTRING::)@ to identity:@IPv4:DstIP:/@@ESTRING:DstPort: @(@ESTRING::)@</pattern> </patterns> <examples> <example> <test_message program='%ASA'>Built inbound UDP connection 55101078 <(55)%20101%20078> for inside:X.X.X.X/X (X.X.X.X/X) to identity:X.X.X.X/X (X.X.X.X/X)</test_message> </example> </examples> </rule> <rule id='20aee256-b4f0-8b4d-93cb-263d5338fd21' class='system' provider='%ASA'>  <patterns> <pattern>Teardown @ESTRING:Protocol: @connection @ESTRING:: @for @ESTRING:::@@IPv4:SrcIP:/@@ESTRING:SrcPort: @to identity:@IPv4 :DstIP:/@@ESTRING:DstPort: @duration@ANYSTRING::@</pattern> </patterns> <examples> <example> <test_message program='%ASA'>Teardown UDP connection 55101084 <(55)%20101%20084> for inside:X.X.X.X/X to identity:X.X.X.X/X duration 0:02:01 bytes 88</test_message> </example> </examples> </rule> <rule id='e075efdc-c25f-5e49-a208-7661e3b5a29b' class='system' provider='%ASA'>  <patterns> <pattern>Built dynamic @ESTRING:Protocol: @translation from @ESTRING:::@@IPv4:LocalIP:/@@ESTRING:LocalPort: @to @ESTRING:::@ @IPv4:GlobalIP:/@@ESTRING:GlobalPort:@</pattern> </patterns> <examples> <example> <test_message program='%ASA'>Built dynamic TCP translation from any:X.X.X.X/X to outside:X.X.X.X/X</test_message> </example> </examples> </rule> <rule provider='%ASA' class='system' id='39'> <patterns> <pattern>Cleared @ESTRING:: @urgent flag from @ESTRING:::@ @ESTRING::/@@NUMBER::@ to @ESTRING: ::@@ESTRING::/@@NUMBER::@</pattern> <pattern>regular translation creation failed for @ESTRING:: @src @ESTRING:::@@ESTRING:: @dst @ESTRING: ::@@ESTRING:: @(type @NUMBER::@, code @NUMBER::@</pattern> </patterns> </rule> </rules> </ruleset> </patterndb> N�n�r��)em�h�yhiם�w^�� ____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product= syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq ____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product= syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq

2 1

syslog-ng 3.14 unofficial fedora / RHEL packages
by Czanik, Péter 05 Mar '18

05 Mar '18

Hi, My unofficial syslog-ng packages for Fedora & RHEL/CentOS 7 are now available. syslog-ng now compiles and works fine on Fedora 27, 28, Rawhide and RHEL 7. Note that as the bundled amqp and mongodb libraries do not compile with openssl 1.1, these features are disabled on all Fedora releases. TCP wrapper support was removed from Fedora and thus also from my packages. All of these are still enabled on RHEL. I hope to spare some time for RHEL 6 packages and make those available later this week. Packages are available at https://copr.fedorainfracloud.org/coprs/czanik/syslog-ng314/ Bye, Peter Czanik (CzP) <peter.czanik(a)balabit.com> Balabit / syslog-ng upstream https://syslog-ng.com/blog/author/peterczanik/ https://twitter.com/PCzanik

1 0

syslog-ng 3.14 unofficial openSUSE / SLES packages
by Czanik, Péter 05 Mar '18

05 Mar '18

Hi, My unofficial syslog-ng packages for openSUSE & SLES are now available. I changed tons of things in packaging (got rid of SLES 11 / sysvinit support) and thus simplified it greatly. It works perfectly for me, but that's a single environment, so any feedback is greatly appreciated. syslog-ng now compiles and works fine on openSUSE Leap, Tumbleweed and SLES 12. Note that as the bundled amqp and mongodb libraries do not compile with openssl 1.1, these features are disabled for Tumbleweed (and will be disabled also for SLES/Leap 15). I started to work on SLES 15 support, but due to some dependency problems that's not yet available. Packages are available at https://build.opensuse.org/project/show/home:czanik:syslog-ng314 Bye, Peter Czanik (CzP) <peter.czanik(a)balabit.com> Balabit / syslog-ng upstream https://syslog-ng.com/blog/author/peterczanik/ https://twitter.com/PCzanik

1 0

syslog-ng 3.14 on FreeBSD
by Czanik, Péter 05 Mar '18

05 Mar '18

Hi, As you can see on Freshports ( https://www.freshports.org/sysutils/syslog-ng314/ ) syslog-ng 3.14 is available now in the FreeBSD ports. Please test it and let us know if you run into any problems with it. We plan to update the syslog-ng metaport to version 3.14 later this week if no blocking problems are found. Thanks for your feedback! Bye, Peter Czanik (CzP) <peter.czanik(a)balabit.com> Balabit / syslog-ng upstream https://syslog-ng.com/blog/author/peterczanik/ https://twitter.com/PCzanik

1 0

add-contextual-data() flags?
by Clayton Dukes 02 Mar '18

02 Mar '18

Hi Guys, Is there an option to ignore case when using add-contextual-data? [cid:image001.png@01D306E3.0FEBC990] Clayton Dukes Founder & CEO LogZilla Corporation 2900 N. Quinlan Park Rd, B240-341 Austin, TX, 78732 Tel: 936-4NetOps (463-8677) Web: www.logzilla.net<http://www.logzilla.net/> [cid:image002.png@01D306E3.0FEBC990]<https://twitter.com/logzilla>[cid:image003.png@01D306E3.0FEBC990]<https://youtu.be/drg5wv_mgfA>[cid:image004.png@01D306E3.0FEBC990]<https://www.linkedin.com/in/lzcdukes/> For NetOps, By NetOps!

2 2

Re: [syslog-ng] Cisco ASA parsing with patterndb/elasticsearch
by Scheidler, Balázs 01 Mar '18

01 Mar '18

Can you give me a few inbound logs as received by syslog-ng? I would try to fix up cisco parser that it works for your use-case, as it should. Thanks On Feb 28, 2018 21:48, "Tim Ghetti" <tghetti(a)targetedsupport.com> wrote: I tried using the cisco-parser and not having any luck with that either. When I enable the parser, I actually stop seeing outbound traffic to elasticsearch. My config is below log { source { udp(flags(no-parse)); }; parser { cisco-parser(); }; destination { elasticsearch2( client-mode("http") cluster("ITESCL001") index("logstash-syslogng_${YEAR}.${MONTH}.${DAY}") cluster-url("http://192.168.101.199:9200 http://192.168.101.198:9200") type("syslog") flush-limit("1")); }; } [root@ITLOG001 conf.d]# tcpdump -nnSXi ens192 port 9200 -vv tcpdump: listening on ens192, link-type EN10MB (Ethernet), capture size 262144 bytes ^C 0 packets captured 0 packets received by filter 0 packets dropped by kernel *From:* syslog-ng [mailto:syslog-ng-bounces@lists.balabit.hu] *On Behalf Of *Scheidler, Balázs *Sent:* Tuesday, February 27, 2018 11:06 PM *To:* Syslog-ng users' and developers' mailing list < syslog-ng(a)lists.balabit.hu> *Subject:* Re: [syslog-ng] Cisco ASA parsing with patterndb/elasticsearch One additional feature you might be interested in is our cisco-parser() feature. It basically expects a flags(no-parse) input and then puts things in the right buckets, while recognising the various formats that cisco uses. It would be appreciated if you could check if it works with your setup and I can help fixing it up if it doesn't. The parser got added to recent syslog-ng versions, so you might need to upgrade to a new version. Bazsi On Feb 27, 2018 22:29, "Tim Ghetti" <tghetti(a)targetedsupport.com> wrote: Thanks!! It sounds like this is part of my problem. If you would be so kind as to send an example (or point me in the right direction) of how you are changing the program name and message body for the cisco logs that would be extremely helpful. I also suspect that there may be another issue. I’m almost certain that my parser should be picking up at least some of the logs but I am not seeing any of the fields in ES showing up based on parsed logs. Assuming that patterndb is working, would the below config be correct for getting patterndb parsed variables to show as fields in ES? destination d_elastic { elasticsearch2( client-mode("http") cluster("ITESCL001") index("logstash-syslogng_${YEAR}.${MONTH}.${DAY}") cluster-url("http://X.X.X.X:9200") type("syslog") flush-limit("1") ); }; *From:* syslog-ng [mailto:syslog-ng-bounces@lists.balabit.hu] *On Behalf Of *Evan Rempel *Sent:* Tuesday, February 27, 2018 9:06 PM *To:* syslog-ng(a)lists.balabit.hu *Subject:* Re: [syslog-ng] Cisco ASA parsing with patterndb/elasticsearch The program name will have an implied @ANYSTRING@ on the end, so if the syslog payload could be parsed to detect the program name of %ASA..... then your patterndb.xml would work. The problem is that they syslog payload can not be parsed. What we do is use a hand crafted patterndb to detect all of the different problem formats that Cisco logs (in our environment) and change a) the PROGRAM to be cisco_ASA or generally cisco_XXX where the XXX is the leading characters of the %XXX-#-##### of the syslog body. b) the MESSAGE to be %XXX-#-####... for the rest of the line. All of the poor hosts, sequence numbers, timestamps etc are all thrown away. After that, the internal buffers of syslog-ng contain a usable PROGRAM and MESSAGE such that we can use a patterndb to match the message part of the log line. This does mean that we have two patterndb parsers for every log line but it seems to work well for us. What would work better is if Cisco would fix their logging, but that isn't going to happen in my lifetime :-( I hope that makes sense. Evan. On 02/27/2018 05:37 PM, Tim Ghetti wrote: Ok, that makes sense. I sort of suspected that this was due to the program name not matching, since pdbtool works when specifying the program name. Do you know if there is a way to configure patterndb so that it matches the program name, regardless of the full program name with event code? Is it possible to regex the program or use the patterndb format i.e. <pattern>%ASA@ANYSTRING::@</pattern> or something similar? Tim *From:* syslog-ng [mailto:syslog-ng-bounces@lists.balabit.hu <syslog-ng-bounces(a)lists.balabit.hu>] *On Behalf Of *Evan Rempel *Sent:* Tuesday, February 27, 2018 8:26 PM *To:* syslog-ng(a)lists.balabit.hu *Subject:* Re: [syslog-ng] Cisco ASA parsing with patterndb/elasticsearch Welcome to the horrible world of Cisco logging :-( The issue you are bumping into is that when these log lines are parsed by syslog-ng (or any syslog daemon that does not specifically understand Cisco logs) there is no program name, or the program name is the full %ASA-4-106023 part of the log line. There are so many ways that Cisco can log incorrectly it is not possible to say without seeing your exact log lines. Cisco can add a * to the date/time stamp to indicate that there is no time server configured on the device. That makes the date/time invalid as far as parsing is concerned. Cisco can add a period (.) to the date/time stamp to indicate that there is a time server configured on the device, but the time server can not be reached. Again, this makes the date/time invalid as far as parsing is concerned. Cisco can add a sequence number at the start of the log line rather than starting the line with a date/6time stamp. Invalid parsing again. Cisco can leave out the sequence number but still include the trailing colon from the sequence number. You get the idea. So many ways to get it wrong, and they never get it right :-( Evan. On 02/27/2018 04:46 PM, Tim Ghetti wrote: Hi – having some trouble getting paterndb functional and looking for some help. I would like to use patterndb to parse my cisco ass firewall logs before sending it to elasticsearch. However when the messages get to elasticsearch, I don’t see the messages being parsed. Running pdbtool against the logs seems to work. # pdbtool match -p /etc/syslog-ng/patterndb.d/ciscoasa.pdb -P %ASA -f /var/log/asatest.log |more HOST=X.X.X.X MESSAGE=Built dynamic TCP translation from INSIDE:X.X.X.X/X to OUTSIDE:X.X.X.X/X PROGRAM=%ASA-6-305011 LEGACY_MSGHDR=%ASA-6-305011: .classifier.class=system .classifier.rule_id=e075efdc-c25f-5e49-a208-7661e3b5a29b Protocol=TCP GlobalIP=X.X.X.X GlobalPort=X LocalIP=X.X.X.X LocalPort=X TAGS=.classifier.system ********************** SYSLOG-NG CONF FILE @version: 3.11 source s_network { tcp(); udp(); }; destination d_elastic { elasticsearch2( client-mode("http") cluster("ITESCL001") index("logstash-syslogng_${YEAR}.${MONTH}.${DAY}") cluster-url("http://X.X.X.X:9200") type("syslog") flush-limit("1") ); }; destination d_catchall { file("/var/log/catchall.log"); }; filter f_ciscoasa { host("X.X.X.X"); }; parser p_ciscoasa {db-parser(file("/etc/syslog- ng/patterndb.d/ciscoasa.pdb"));}; log { source(s_network); filter(f_ciscoasa); parser(p_ciscoasa); destination(d_elastic); flags(final, flow-control); }; log { source(s_network); destination(d_catchall); }; ********************** PATTERNDB FILE <?xml version='1.0' encoding='UTF-8'?> <patterndb version='4' pub_date='2018-02-19'> <ruleset name='%ASA' id='a300d776-8bd7-834d-a4a9-23eb81a4b3ba'> <pattern>%ASA</pattern> <description> This ruleset covers the Cisco ASA firewalls </description> <rules> <rule provider="%ASA" id="b3de7699-8213-c744-944e-9413298afe86" class="system">  <patterns> <pattern>Teardown @ESTRING:Protocol: @connection for faddr @IPv4:SrcIP:/@@ESTRING:SrcPort: @gaddr @IPv4:GlobalIP:/@@ESTRING:GlobalPort: @laddr @IPv4:LocalIP:/@@ESTRING:LocalPort:@</pattern> </patterns> <examples> <example> <test_message program='%ASA'>Teardown ICMP connection for faddr X.X.X.X/X gaddr X.X.X.X/X laddr X.X.X.X/X</test_message> </example> </examples> </rule> <rule id='90d0f8c9-7591-d44e-b886-2f7e5cb17ce6' class='system' provider='%ASA'>  <patterns> <pattern>Teardown dynamic @ESTRING:Protocol: @translation from @ESTRING:::@@IPv4:LocalIP:/@@ESTRING:LocalPort: @to @ESTRING:::@ @IPv4:GlobalIP:/@@ESTRING:GlobalPort: @duration@ANYSTRING::@</pattern> </patterns> <examples> <example> <test_message program='%ASA'>Teardown dynamic UDP translation from any:X.X.X.X/X to outside:X.X.X.X/X duration 0:00:00</test_message> </example> </examples> </rule> <rule id='8f0a8d57-80c6-4745-8a8a-5ce018bb0d87' class='system' provider='%ASA'>  <patterns> <pattern>Teardown @ESTRING:Protocol: @connection @ESTRING:: @for @ESTRING:::@@IPv4:DstIP:/@@ESTRING:DstPort: @to @ESTRING:::@@IPv4:SrcIP:/@@ESTRING:SrcPort: @@ESTRING::@</pattern> </patterns> <examples> <example> <test_message program='%ASA'>Teardown UDP connection 55101037 <(55)%20101%20037> for outside:X.X.X.X/X to inside:X.X.X.X/X duration 0:00:00 bytes 132</test_message> </example> </examples> </rule> <rule id='00c0732d-1e34-7340-a75f-21198bf71137' class='system' provider='%ASA'>  <patterns> <pattern>Built outbound @ESTRING:Protocol: @connection @ESTRING:: @for @ESTRING:::@@IPv4:DstIP:/@@ESTRING:DstPort: @(@ESTRING::)@ to @ESTRING:::@@IPv4:SrcIP:/@@ESTRING:SrcPort: @(@ESTRING::)@</pattern> </patterns> <examples> <example> <test_message program='%ASA'>Built outbound UDP connection 55101037 <(55)%20101%20037> for outside:X.X.X.X/X (X.X.X.X/X) to inside:X.X.X.X/X (X.X.X.X/X)</test_message> </example> </examples> </rule> <rule id='4a586711-ebe2-dc4d-bf6e-e512666d8c5d' class='system' provider='%ASA'>  <patterns> <pattern>Built inbound @ESTRING:Protocol: @connection for faddr @IPv4:SrcIP:/@@ESTRING:SrcPort: @gaddr @IPv4:GlobalIP:/@@ESTRING:GlobalPort: @laddr @IPv4:LocalIP:/@@ESTRING:LocalPort:@</pattern> </patterns> <examples> <example> <test_message program='%ASA'>Built inbound ICMP connection for faddr X.X.X.X/X gaddr X.X.X.X/X laddr X.X.X.X/X</test_message> </example> </examples> </rule> <rule id='8be7928d-66e7-7042-abd5-869d6b49c56e' class='system' provider='%ASA'>  <patterns> <pattern>Built inbound @ESTRING:Protocol: @connection @ESTRING:: @for @ESTRING::@@IPv4:SrcIP:/@@ESTRING:SrcPort: @(@ESTRING::)@ to identity:@IPv4:DstIP:/@@ESTRING:DstPort: @(@ESTRING::)@</pattern> </patterns> <examples> <example> <test_message program='%ASA'>Built inbound UDP connection 55101078 <(55)%20101%20078> for inside:X.X.X.X/X (X.X.X.X/X) to identity:X.X.X.X/X (X.X.X.X/X)</test_message> </example> </examples> </rule> <rule id='20aee256-b4f0-8b4d-93cb-263d5338fd21' class='system' provider='%ASA'>  <patterns> <pattern>Teardown @ESTRING:Protocol: @connection @ESTRING:: @for @ESTRING:::@@IPv4:SrcIP:/@@ESTRING:SrcPort: @to identity:@IPv4:DstIP:/@@ESTRING:DstPort: @duration@ANYSTRING::@</pattern> </patterns> <examples> <example> <test_message program='%ASA'>Teardown UDP connection 55101084 <(55)%20101%20084> for inside:X.X.X.X/X to identity:X.X.X.X/X duration 0:02:01 bytes 88</test_message> </example> </examples> </rule> <rule id='e075efdc-c25f-5e49-a208-7661e3b5a29b' class='system' provider='%ASA'>  <patterns> <pattern>Built dynamic @ESTRING:Protocol: @translation from @ESTRING:::@@IPv4:LocalIP:/@@ESTRING:LocalPort: @to @ESTRING:::@ @IPv4:GlobalIP:/@@ESTRING:GlobalPort:@</pattern> </patterns> <examples> <example> <test_message program='%ASA'>Built dynamic TCP translation from any:X.X.X.X/X to outside:X.X.X.X/X</test_message> </example> </examples> </rule> <rule provider='%ASA' class='system' id='39'> <patterns> <pattern>Cleared @ESTRING:: @urgent flag from @ESTRING:::@ @ESTRING::/@@NUMBER::@ to @ESTRING: ::@@ESTRING::/@@NUMBER::@</pattern> <pattern>regular translation creation failed for @ESTRING:: @src @ESTRING:::@@ESTRING:: @dst @ESTRING: ::@@ESTRING:: @(type @NUMBER::@, code @NUMBER::@</pattern> </patterns> </rule> </rules> </ruleset> </patterndb> N�n�r��)em�h�yhiם�w^�� ____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/? product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq ____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/? product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq

1 0

Cisco ASA parsing with patterndb/elasticsearch
by Tim Ghetti 01 Mar '18

01 Mar '18

Hi - having some trouble getting paterndb functional and looking for some help. I would like to use patterndb to parse my cisco ass firewall logs before sending it to elasticsearch. However when the messages get to elasticsearch, I don't see the messages being parsed. Running pdbtool against the logs seems to work. # pdbtool match -p /etc/syslog-ng/patterndb.d/ciscoasa.pdb -P %ASA -f /var/log/asatest.log |more HOST=X.X.X.X MESSAGE=Built dynamic TCP translation from INSIDE:X.X.X.X/X to OUTSIDE:X.X.X.X/X PROGRAM=%ASA-6-305011 LEGACY_MSGHDR=%ASA-6-305011: .classifier.class=system .classifier.rule_id=e075efdc-c25f-5e49-a208-7661e3b5a29b Protocol=TCP GlobalIP=X.X.X.X GlobalPort=X LocalIP=X.X.X.X LocalPort=X TAGS=.classifier.system ********************** SYSLOG-NG CONF FILE @version: 3.11 source s_network { tcp(); udp(); }; destination d_elastic { elasticsearch2( client-mode("http") cluster("ITESCL001") index("logstash-syslogng_${YEAR}.${MONTH}.${DAY}") cluster-url("http://X.X.X.X:9200") type("syslog") flush-limit("1") ); }; destination d_catchall { file("/var/log/catchall.log"); }; filter f_ciscoasa { host("X.X.X.X"); }; parser p_ciscoasa {db-parser(file("/etc/syslog-ng/patterndb.d/ciscoasa.pdb"));}; log { source(s_network); filter(f_ciscoasa); parser(p_ciscoasa); destination(d_elastic); flags(final, flow-control); }; log { source(s_network); destination(d_catchall); }; ********************** PATTERNDB FILE <?xml version='1.0' encoding='UTF-8'?> <patterndb version='4' pub_date='2018-02-19'> <ruleset name='%ASA' id='a300d776-8bd7-834d-a4a9-23eb81a4b3ba'> <pattern>%ASA</pattern> <description> This ruleset covers the Cisco ASA firewalls </description> <rules> <rule provider="%ASA" id="b3de7699-8213-c744-944e-9413298afe86" class="system">  <patterns> <pattern>Teardown @ESTRING:Protocol: @connection for faddr @IPv4:SrcIP:/@@ESTRING:SrcPort: @gaddr @IPv4:GlobalIP:/@@ESTRING:GlobalPort: @laddr @IPv4:LocalIP:/@@ESTRING:LocalPort:@</pattern> </patterns> <examples> <example> <test_message program='%ASA'>Teardown ICMP connection for faddr X.X.X.X/X gaddr X.X.X.X/X laddr X.X.X.X/X</test_message> </example> </examples> </rule> <rule id='90d0f8c9-7591-d44e-b886-2f7e5cb17ce6' class='system' provider='%ASA'>  <patterns> <pattern>Teardown dynamic @ESTRING:Protocol: @translation from @ESTRING:::@@IPv4:LocalIP:/@@ESTRING:LocalPort: @to @ESTRING:::@@IPv4:GlobalIP:/@@ESTRING:GlobalPort: @duration@ANYSTRING::@</pattern> </patterns> <examples> <example> <test_message program='%ASA'>Teardown dynamic UDP translation from any:X.X.X.X/X to outside:X.X.X.X/X duration 0:00:00</test_message> </example> </examples> </rule> <rule id='8f0a8d57-80c6-4745-8a8a-5ce018bb0d87' class='system' provider='%ASA'>  <patterns> <pattern>Teardown @ESTRING:Protocol: @connection @ESTRING:: @for @ESTRING:::@@IPv4:DstIP:/@@ESTRING:DstPort: @to @ESTRING:::@@IPv4:SrcIP:/@@ESTRING:SrcPort: @@ESTRING::@</pattern> </patterns> <examples> <example> <test_message program='%ASA'>Teardown UDP connection 55101037 for outside:X.X.X.X/X to inside:X.X.X.X/X duration 0:00:00 bytes 132</test_message> </example> </examples> </rule> <rule id='00c0732d-1e34-7340-a75f-21198bf71137' class='system' provider='%ASA'>  <patterns> <pattern>Built outbound @ESTRING:Protocol: @connection @ESTRING:: @for @ESTRING:::@@IPv4:DstIP:/@@ESTRING:DstPort: @(@ESTRING::)@ to @ESTRING:::@@IPv4:SrcIP:/@@ESTRING:SrcPort: @(@ESTRING::)@</pattern> </patterns> <examples> <example> <test_message program='%ASA'>Built outbound UDP connection 55101037 for outside:X.X.X.X/X (X.X.X.X/X) to inside:X.X.X.X/X (X.X.X.X/X)</test_message> </example> </examples> </rule> <rule id='4a586711-ebe2-dc4d-bf6e-e512666d8c5d' class='system' provider='%ASA'>  <patterns> <pattern>Built inbound @ESTRING:Protocol: @connection for faddr @IPv4:SrcIP:/@@ESTRING:SrcPort: @gaddr @IPv4:GlobalIP:/@@ESTRING:GlobalPort: @laddr @IPv4:LocalIP:/@@ESTRING:LocalPort:@</pattern> </patterns> <examples> <example> <test_message program='%ASA'>Built inbound ICMP connection for faddr X.X.X.X/X gaddr X.X.X.X/X laddr X.X.X.X/X</test_message> </example> </examples> </rule> <rule id='8be7928d-66e7-7042-abd5-869d6b49c56e' class='system' provider='%ASA'>  <patterns> <pattern>Built inbound @ESTRING:Protocol: @connection @ESTRING:: @for @ESTRING::@@IPv4:SrcIP:/@@ESTRING:SrcPort: @(@ESTRING::)@ to identity:@IPv4:DstIP:/@@ESTRING:DstPort: @(@ESTRING::)@</pattern> </patterns> <examples> <example> <test_message program='%ASA'>Built inbound UDP connection 55101078 for inside:X.X.X.X/X (X.X.X.X/X) to identity:X.X.X.X/X (X.X.X.X/X)</test_message> </example> </examples> </rule> <rule id='20aee256-b4f0-8b4d-93cb-263d5338fd21' class='system' provider='%ASA'>  <patterns> <pattern>Teardown @ESTRING:Protocol: @connection @ESTRING:: @for @ESTRING:::@@IPv4:SrcIP:/@@ESTRING:SrcPort: @to identity:@IPv4:DstIP:/@@ESTRING:DstPort: @duration@ANYSTRING::@</pattern> </patterns> <examples> <example> <test_message program='%ASA'>Teardown UDP connection 55101084 for inside:X.X.X.X/X to identity:X.X.X.X/X duration 0:02:01 bytes 88</test_message> </example> </examples> </rule> <rule id='e075efdc-c25f-5e49-a208-7661e3b5a29b' class='system' provider='%ASA'>  <patterns> <pattern>Built dynamic @ESTRING:Protocol: @translation from @ESTRING:::@@IPv4:LocalIP:/@@ESTRING:LocalPort: @to @ESTRING:::@@IPv4:GlobalIP:/@@ESTRING:GlobalPort:@</pattern> </patterns> <examples> <example> <test_message program='%ASA'>Built dynamic TCP translation from any:X.X.X.X/X to outside:X.X.X.X/X</test_message> </example> </examples> </rule> <rule provider='%ASA' class='system' id='39'> <patterns> <pattern>Cleared @ESTRING:: @urgent flag from @ESTRING:::@@ESTRING::/@@NUMBER::@ to @ESTRING: ::@@ESTRING::/@@NUMBER::@</pattern> <pattern>regular translation creation failed for @ESTRING:: @src @ESTRING:::@@ESTRING:: @dst @ESTRING: ::@@ESTRING:: @(type @NUMBER::@, code @NUMBER::@</pattern> </patterns> </rule> </rules> </ruleset> </patterndb>

3 6

Problem building syslog-ng 3.11.1 and elastic search
by Ronald Fenner 27 Feb '18

27 Feb '18

I have script that builds syslog-ng and it was working fine but apparently a plugin for elastic search seems to have gone missing and I'm getting this error: 1519328565,,ui,message, amazon-ebs: Could not resolve all files for configuration ':elastic-v2:compileClasspath'. 1519328565,,ui,message, amazon-ebs: > Could not find org.elasticsearch.plugin:shield:2.4.0. 1519328565,,ui,message, amazon-ebs: Searched in the following locations: 1519328565,,ui,message, amazon-ebs: file:/usr/lib/syslog-ng-java-compile-deps-mvn-repo/org/elasticsearch/plugin/shield$ 1519328565,,ui,message, amazon-ebs: file:/usr/lib/syslog-ng-java-compile-deps-mvn-repo/org/elasticsearch/plugin/shield$ 1519328565,,ui,message, amazon-ebs: file:/home/ec2-user/eventlog-0.2.12/syslog-ng-3.11.1/modules/java/syslog-ng-core/l$ 1519328565,,ui,message, amazon-ebs: file:/home/ec2-user/eventlog-0.2.12/syslog-ng-3.11.1/modules/java/syslog-ng-core/l$ 1519328565,,ui,message, amazon-ebs: file:/home/ec2-user/eventlog-0.2.12/syslog-ng-3.11.1/modules/java-modules/common/g$ 1519328565,,ui,message, amazon-ebs: file:/home/ec2-user/eventlog-0.2.12/syslog-ng-3.11.1/modules/java-modules/common/g$ 1519328565,,ui,message, amazon-ebs: https://repo1.maven.org/maven2/org/elasticsearch/plugin/shield/2.4.0/shield… 1519328565,,ui,message, amazon-ebs: https://repo1.maven.org/maven2/org/elasticsearch/plugin/shield/2.4.0/shield… 1519328565,,ui,message, amazon-ebs: http://maven.elasticsearch.org/releases/org/elasticsearch/plugin/shield/2.4… 1519328565,,ui,message, amazon-ebs: 9 actionable tasks: 9 executed 1519328565,,ui,message, amazon-ebs: http://maven.elasticsearch.org/releases/org/elasticsearch/plugin/shield/2.4… 1519328565,,ui,message, amazon-ebs: Required by: 1519328565,,ui,message, amazon-ebs: project :elastic-v2 1519328565,,ui,message, amazon-ebs: 1519328565,,ui,message, amazon-ebs: * Try: 1519328565,,ui,message, amazon-ebs: Run with --stacktrace option to get the stack trace. Run with --info or --debug option t$ 1519328565,,ui,message, amazon-ebs: Is there a way to disable elasticsearch module specifically? Ronald Fenner Network Architect Game Circus LLC. rfenner(a)gamecircus.com

5 9

Directory/file create ownership
by Machiel Richards 27 Feb '18

27 Feb '18

Good day guys I am having an issue that I seem not be able to resolve thus far. I am fairly new to syslog-ng and have setup a new syslog server. We have managed to get most of our servers sending logs to the server without issues, however I am having trouble to get syslog-ng to create the directories and files with the correct ownership. We have a seperate user and group which it needs to create directories with, however it keeps on creating directories and files as root user. below is my current config : # First, set some global options. options { chain_hostnames(off); flush_lines(0); use_dns(yes); use_fqdn(yes); owner("syslog"); group("syslog"); perm(0640); stats_freq(0); bad_hostname("^gconfd$"); log_fifo_size(10000); }; source inputs { system(); internal(); unix-stream("/dev/log"); tcp(max_connections(50)); udp(); }; destination logpile { file("/var/log/HOSTS/$HOST/$YEAR/$MONTH/$FACILITY.$YEAR$MONTH$DAY" owner("syslog") group("syslog") perm(0660) create_dirs(yes) dir_perm(0770)); }; log { source(inputs); destination(logpile); }; Can someone please assist me as any help would be appreciated. Regard -- I am a bomb technician...If you see me running, TRY TO KEEP UP!

1 0