- syslog-ng - lists.balabit.hu

add-contextual-data() flags?
by Clayton Dukes 02 Mar '18

02 Mar '18

Hi Guys, Is there an option to ignore case when using add-contextual-data? [cid:image001.png@01D306E3.0FEBC990] Clayton Dukes Founder & CEO LogZilla Corporation 2900 N. Quinlan Park Rd, B240-341 Austin, TX, 78732 Tel: 936-4NetOps (463-8677) Web: www.logzilla.net<http://www.logzilla.net/> [cid:image002.png@01D306E3.0FEBC990]<https://twitter.com/logzilla>[cid:image003.png@01D306E3.0FEBC990]<https://youtu.be/drg5wv_mgfA>[cid:image004.png@01D306E3.0FEBC990]<https://www.linkedin.com/in/lzcdukes/> For NetOps, By NetOps!

2 2

Re: [syslog-ng] Cisco ASA parsing with patterndb/elasticsearch
by Scheidler, Balázs 01 Mar '18

01 Mar '18

Can you give me a few inbound logs as received by syslog-ng? I would try to fix up cisco parser that it works for your use-case, as it should. Thanks On Feb 28, 2018 21:48, "Tim Ghetti" <tghetti(a)targetedsupport.com> wrote: I tried using the cisco-parser and not having any luck with that either. When I enable the parser, I actually stop seeing outbound traffic to elasticsearch. My config is below log { source { udp(flags(no-parse)); }; parser { cisco-parser(); }; destination { elasticsearch2( client-mode("http") cluster("ITESCL001") index("logstash-syslogng_${YEAR}.${MONTH}.${DAY}") cluster-url("http://192.168.101.199:9200 http://192.168.101.198:9200") type("syslog") flush-limit("1")); }; } [root@ITLOG001 conf.d]# tcpdump -nnSXi ens192 port 9200 -vv tcpdump: listening on ens192, link-type EN10MB (Ethernet), capture size 262144 bytes ^C 0 packets captured 0 packets received by filter 0 packets dropped by kernel *From:* syslog-ng [mailto:syslog-ng-bounces@lists.balabit.hu] *On Behalf Of *Scheidler, Balázs *Sent:* Tuesday, February 27, 2018 11:06 PM *To:* Syslog-ng users' and developers' mailing list < syslog-ng(a)lists.balabit.hu> *Subject:* Re: [syslog-ng] Cisco ASA parsing with patterndb/elasticsearch One additional feature you might be interested in is our cisco-parser() feature. It basically expects a flags(no-parse) input and then puts things in the right buckets, while recognising the various formats that cisco uses. It would be appreciated if you could check if it works with your setup and I can help fixing it up if it doesn't. The parser got added to recent syslog-ng versions, so you might need to upgrade to a new version. Bazsi On Feb 27, 2018 22:29, "Tim Ghetti" <tghetti(a)targetedsupport.com> wrote: Thanks!! It sounds like this is part of my problem. If you would be so kind as to send an example (or point me in the right direction) of how you are changing the program name and message body for the cisco logs that would be extremely helpful. I also suspect that there may be another issue. I’m almost certain that my parser should be picking up at least some of the logs but I am not seeing any of the fields in ES showing up based on parsed logs. Assuming that patterndb is working, would the below config be correct for getting patterndb parsed variables to show as fields in ES? destination d_elastic { elasticsearch2( client-mode("http") cluster("ITESCL001") index("logstash-syslogng_${YEAR}.${MONTH}.${DAY}") cluster-url("http://X.X.X.X:9200") type("syslog") flush-limit("1") ); }; *From:* syslog-ng [mailto:syslog-ng-bounces@lists.balabit.hu] *On Behalf Of *Evan Rempel *Sent:* Tuesday, February 27, 2018 9:06 PM *To:* syslog-ng(a)lists.balabit.hu *Subject:* Re: [syslog-ng] Cisco ASA parsing with patterndb/elasticsearch The program name will have an implied @ANYSTRING@ on the end, so if the syslog payload could be parsed to detect the program name of %ASA..... then your patterndb.xml would work. The problem is that they syslog payload can not be parsed. What we do is use a hand crafted patterndb to detect all of the different problem formats that Cisco logs (in our environment) and change a) the PROGRAM to be cisco_ASA or generally cisco_XXX where the XXX is the leading characters of the %XXX-#-##### of the syslog body. b) the MESSAGE to be %XXX-#-####... for the rest of the line. All of the poor hosts, sequence numbers, timestamps etc are all thrown away. After that, the internal buffers of syslog-ng contain a usable PROGRAM and MESSAGE such that we can use a patterndb to match the message part of the log line. This does mean that we have two patterndb parsers for every log line but it seems to work well for us. What would work better is if Cisco would fix their logging, but that isn't going to happen in my lifetime :-( I hope that makes sense. Evan. On 02/27/2018 05:37 PM, Tim Ghetti wrote: Ok, that makes sense. I sort of suspected that this was due to the program name not matching, since pdbtool works when specifying the program name. Do you know if there is a way to configure patterndb so that it matches the program name, regardless of the full program name with event code? Is it possible to regex the program or use the patterndb format i.e. <pattern>%ASA@ANYSTRING::@</pattern> or something similar? Tim *From:* syslog-ng [mailto:syslog-ng-bounces@lists.balabit.hu <syslog-ng-bounces(a)lists.balabit.hu>] *On Behalf Of *Evan Rempel *Sent:* Tuesday, February 27, 2018 8:26 PM *To:* syslog-ng(a)lists.balabit.hu *Subject:* Re: [syslog-ng] Cisco ASA parsing with patterndb/elasticsearch Welcome to the horrible world of Cisco logging :-( The issue you are bumping into is that when these log lines are parsed by syslog-ng (or any syslog daemon that does not specifically understand Cisco logs) there is no program name, or the program name is the full %ASA-4-106023 part of the log line. There are so many ways that Cisco can log incorrectly it is not possible to say without seeing your exact log lines. Cisco can add a * to the date/time stamp to indicate that there is no time server configured on the device. That makes the date/time invalid as far as parsing is concerned. Cisco can add a period (.) to the date/time stamp to indicate that there is a time server configured on the device, but the time server can not be reached. Again, this makes the date/time invalid as far as parsing is concerned. Cisco can add a sequence number at the start of the log line rather than starting the line with a date/6time stamp. Invalid parsing again. Cisco can leave out the sequence number but still include the trailing colon from the sequence number. You get the idea. So many ways to get it wrong, and they never get it right :-( Evan. On 02/27/2018 04:46 PM, Tim Ghetti wrote: Hi – having some trouble getting paterndb functional and looking for some help. I would like to use patterndb to parse my cisco ass firewall logs before sending it to elasticsearch. However when the messages get to elasticsearch, I don’t see the messages being parsed. Running pdbtool against the logs seems to work. # pdbtool match -p /etc/syslog-ng/patterndb.d/ciscoasa.pdb -P %ASA -f /var/log/asatest.log |more HOST=X.X.X.X MESSAGE=Built dynamic TCP translation from INSIDE:X.X.X.X/X to OUTSIDE:X.X.X.X/X PROGRAM=%ASA-6-305011 LEGACY_MSGHDR=%ASA-6-305011: .classifier.class=system .classifier.rule_id=e075efdc-c25f-5e49-a208-7661e3b5a29b Protocol=TCP GlobalIP=X.X.X.X GlobalPort=X LocalIP=X.X.X.X LocalPort=X TAGS=.classifier.system ********************** SYSLOG-NG CONF FILE @version: 3.11 source s_network { tcp(); udp(); }; destination d_elastic { elasticsearch2( client-mode("http") cluster("ITESCL001") index("logstash-syslogng_${YEAR}.${MONTH}.${DAY}") cluster-url("http://X.X.X.X:9200") type("syslog") flush-limit("1") ); }; destination d_catchall { file("/var/log/catchall.log"); }; filter f_ciscoasa { host("X.X.X.X"); }; parser p_ciscoasa {db-parser(file("/etc/syslog- ng/patterndb.d/ciscoasa.pdb"));}; log { source(s_network); filter(f_ciscoasa); parser(p_ciscoasa); destination(d_elastic); flags(final, flow-control); }; log { source(s_network); destination(d_catchall); }; ********************** PATTERNDB FILE <?xml version='1.0' encoding='UTF-8'?> <patterndb version='4' pub_date='2018-02-19'> <ruleset name='%ASA' id='a300d776-8bd7-834d-a4a9-23eb81a4b3ba'> <pattern>%ASA</pattern> <description> This ruleset covers the Cisco ASA firewalls </description> <rules> <rule provider="%ASA" id="b3de7699-8213-c744-944e-9413298afe86" class="system">  <patterns> <pattern>Teardown @ESTRING:Protocol: @connection for faddr @IPv4:SrcIP:/@@ESTRING:SrcPort: @gaddr @IPv4:GlobalIP:/@@ESTRING:GlobalPort: @laddr @IPv4:LocalIP:/@@ESTRING:LocalPort:@</pattern> </patterns> <examples> <example> <test_message program='%ASA'>Teardown ICMP connection for faddr X.X.X.X/X gaddr X.X.X.X/X laddr X.X.X.X/X</test_message> </example> </examples> </rule> <rule id='90d0f8c9-7591-d44e-b886-2f7e5cb17ce6' class='system' provider='%ASA'>  <patterns> <pattern>Teardown dynamic @ESTRING:Protocol: @translation from @ESTRING:::@@IPv4:LocalIP:/@@ESTRING:LocalPort: @to @ESTRING:::@ @IPv4:GlobalIP:/@@ESTRING:GlobalPort: @duration@ANYSTRING::@</pattern> </patterns> <examples> <example> <test_message program='%ASA'>Teardown dynamic UDP translation from any:X.X.X.X/X to outside:X.X.X.X/X duration 0:00:00</test_message> </example> </examples> </rule> <rule id='8f0a8d57-80c6-4745-8a8a-5ce018bb0d87' class='system' provider='%ASA'>  <patterns> <pattern>Teardown @ESTRING:Protocol: @connection @ESTRING:: @for @ESTRING:::@@IPv4:DstIP:/@@ESTRING:DstPort: @to @ESTRING:::@@IPv4:SrcIP:/@@ESTRING:SrcPort: @@ESTRING::@</pattern> </patterns> <examples> <example> <test_message program='%ASA'>Teardown UDP connection 55101037 <(55)%20101%20037> for outside:X.X.X.X/X to inside:X.X.X.X/X duration 0:00:00 bytes 132</test_message> </example> </examples> </rule> <rule id='00c0732d-1e34-7340-a75f-21198bf71137' class='system' provider='%ASA'>  <patterns> <pattern>Built outbound @ESTRING:Protocol: @connection @ESTRING:: @for @ESTRING:::@@IPv4:DstIP:/@@ESTRING:DstPort: @(@ESTRING::)@ to @ESTRING:::@@IPv4:SrcIP:/@@ESTRING:SrcPort: @(@ESTRING::)@</pattern> </patterns> <examples> <example> <test_message program='%ASA'>Built outbound UDP connection 55101037 <(55)%20101%20037> for outside:X.X.X.X/X (X.X.X.X/X) to inside:X.X.X.X/X (X.X.X.X/X)</test_message> </example> </examples> </rule> <rule id='4a586711-ebe2-dc4d-bf6e-e512666d8c5d' class='system' provider='%ASA'>  <patterns> <pattern>Built inbound @ESTRING:Protocol: @connection for faddr @IPv4:SrcIP:/@@ESTRING:SrcPort: @gaddr @IPv4:GlobalIP:/@@ESTRING:GlobalPort: @laddr @IPv4:LocalIP:/@@ESTRING:LocalPort:@</pattern> </patterns> <examples> <example> <test_message program='%ASA'>Built inbound ICMP connection for faddr X.X.X.X/X gaddr X.X.X.X/X laddr X.X.X.X/X</test_message> </example> </examples> </rule> <rule id='8be7928d-66e7-7042-abd5-869d6b49c56e' class='system' provider='%ASA'>  <patterns> <pattern>Built inbound @ESTRING:Protocol: @connection @ESTRING:: @for @ESTRING::@@IPv4:SrcIP:/@@ESTRING:SrcPort: @(@ESTRING::)@ to identity:@IPv4:DstIP:/@@ESTRING:DstPort: @(@ESTRING::)@</pattern> </patterns> <examples> <example> <test_message program='%ASA'>Built inbound UDP connection 55101078 <(55)%20101%20078> for inside:X.X.X.X/X (X.X.X.X/X) to identity:X.X.X.X/X (X.X.X.X/X)</test_message> </example> </examples> </rule> <rule id='20aee256-b4f0-8b4d-93cb-263d5338fd21' class='system' provider='%ASA'>  <patterns> <pattern>Teardown @ESTRING:Protocol: @connection @ESTRING:: @for @ESTRING:::@@IPv4:SrcIP:/@@ESTRING:SrcPort: @to identity:@IPv4:DstIP:/@@ESTRING:DstPort: @duration@ANYSTRING::@</pattern> </patterns> <examples> <example> <test_message program='%ASA'>Teardown UDP connection 55101084 <(55)%20101%20084> for inside:X.X.X.X/X to identity:X.X.X.X/X duration 0:02:01 bytes 88</test_message> </example> </examples> </rule> <rule id='e075efdc-c25f-5e49-a208-7661e3b5a29b' class='system' provider='%ASA'>  <patterns> <pattern>Built dynamic @ESTRING:Protocol: @translation from @ESTRING:::@@IPv4:LocalIP:/@@ESTRING:LocalPort: @to @ESTRING:::@ @IPv4:GlobalIP:/@@ESTRING:GlobalPort:@</pattern> </patterns> <examples> <example> <test_message program='%ASA'>Built dynamic TCP translation from any:X.X.X.X/X to outside:X.X.X.X/X</test_message> </example> </examples> </rule> <rule provider='%ASA' class='system' id='39'> <patterns> <pattern>Cleared @ESTRING:: @urgent flag from @ESTRING:::@ @ESTRING::/@@NUMBER::@ to @ESTRING: ::@@ESTRING::/@@NUMBER::@</pattern> <pattern>regular translation creation failed for @ESTRING:: @src @ESTRING:::@@ESTRING:: @dst @ESTRING: ::@@ESTRING:: @(type @NUMBER::@, code @NUMBER::@</pattern> </patterns> </rule> </rules> </ruleset> </patterndb> N�n�r��)em�h�yhiם�w^�� ____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/? product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq ____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/? product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq

1 0

Cisco ASA parsing with patterndb/elasticsearch
by Tim Ghetti 01 Mar '18

01 Mar '18

Hi - having some trouble getting paterndb functional and looking for some help. I would like to use patterndb to parse my cisco ass firewall logs before sending it to elasticsearch. However when the messages get to elasticsearch, I don't see the messages being parsed. Running pdbtool against the logs seems to work. # pdbtool match -p /etc/syslog-ng/patterndb.d/ciscoasa.pdb -P %ASA -f /var/log/asatest.log |more HOST=X.X.X.X MESSAGE=Built dynamic TCP translation from INSIDE:X.X.X.X/X to OUTSIDE:X.X.X.X/X PROGRAM=%ASA-6-305011 LEGACY_MSGHDR=%ASA-6-305011: .classifier.class=system .classifier.rule_id=e075efdc-c25f-5e49-a208-7661e3b5a29b Protocol=TCP GlobalIP=X.X.X.X GlobalPort=X LocalIP=X.X.X.X LocalPort=X TAGS=.classifier.system ********************** SYSLOG-NG CONF FILE @version: 3.11 source s_network { tcp(); udp(); }; destination d_elastic { elasticsearch2( client-mode("http") cluster("ITESCL001") index("logstash-syslogng_${YEAR}.${MONTH}.${DAY}") cluster-url("http://X.X.X.X:9200") type("syslog") flush-limit("1") ); }; destination d_catchall { file("/var/log/catchall.log"); }; filter f_ciscoasa { host("X.X.X.X"); }; parser p_ciscoasa {db-parser(file("/etc/syslog-ng/patterndb.d/ciscoasa.pdb"));}; log { source(s_network); filter(f_ciscoasa); parser(p_ciscoasa); destination(d_elastic); flags(final, flow-control); }; log { source(s_network); destination(d_catchall); }; ********************** PATTERNDB FILE <?xml version='1.0' encoding='UTF-8'?> <patterndb version='4' pub_date='2018-02-19'> <ruleset name='%ASA' id='a300d776-8bd7-834d-a4a9-23eb81a4b3ba'> <pattern>%ASA</pattern> <description> This ruleset covers the Cisco ASA firewalls </description> <rules> <rule provider="%ASA" id="b3de7699-8213-c744-944e-9413298afe86" class="system">  <patterns> <pattern>Teardown @ESTRING:Protocol: @connection for faddr @IPv4:SrcIP:/@@ESTRING:SrcPort: @gaddr @IPv4:GlobalIP:/@@ESTRING:GlobalPort: @laddr @IPv4:LocalIP:/@@ESTRING:LocalPort:@</pattern> </patterns> <examples> <example> <test_message program='%ASA'>Teardown ICMP connection for faddr X.X.X.X/X gaddr X.X.X.X/X laddr X.X.X.X/X</test_message> </example> </examples> </rule> <rule id='90d0f8c9-7591-d44e-b886-2f7e5cb17ce6' class='system' provider='%ASA'>  <patterns> <pattern>Teardown dynamic @ESTRING:Protocol: @translation from @ESTRING:::@@IPv4:LocalIP:/@@ESTRING:LocalPort: @to @ESTRING:::@@IPv4:GlobalIP:/@@ESTRING:GlobalPort: @duration@ANYSTRING::@</pattern> </patterns> <examples> <example> <test_message program='%ASA'>Teardown dynamic UDP translation from any:X.X.X.X/X to outside:X.X.X.X/X duration 0:00:00</test_message> </example> </examples> </rule> <rule id='8f0a8d57-80c6-4745-8a8a-5ce018bb0d87' class='system' provider='%ASA'>  <patterns> <pattern>Teardown @ESTRING:Protocol: @connection @ESTRING:: @for @ESTRING:::@@IPv4:DstIP:/@@ESTRING:DstPort: @to @ESTRING:::@@IPv4:SrcIP:/@@ESTRING:SrcPort: @@ESTRING::@</pattern> </patterns> <examples> <example> <test_message program='%ASA'>Teardown UDP connection 55101037 for outside:X.X.X.X/X to inside:X.X.X.X/X duration 0:00:00 bytes 132</test_message> </example> </examples> </rule> <rule id='00c0732d-1e34-7340-a75f-21198bf71137' class='system' provider='%ASA'>  <patterns> <pattern>Built outbound @ESTRING:Protocol: @connection @ESTRING:: @for @ESTRING:::@@IPv4:DstIP:/@@ESTRING:DstPort: @(@ESTRING::)@ to @ESTRING:::@@IPv4:SrcIP:/@@ESTRING:SrcPort: @(@ESTRING::)@</pattern> </patterns> <examples> <example> <test_message program='%ASA'>Built outbound UDP connection 55101037 for outside:X.X.X.X/X (X.X.X.X/X) to inside:X.X.X.X/X (X.X.X.X/X)</test_message> </example> </examples> </rule> <rule id='4a586711-ebe2-dc4d-bf6e-e512666d8c5d' class='system' provider='%ASA'>  <patterns> <pattern>Built inbound @ESTRING:Protocol: @connection for faddr @IPv4:SrcIP:/@@ESTRING:SrcPort: @gaddr @IPv4:GlobalIP:/@@ESTRING:GlobalPort: @laddr @IPv4:LocalIP:/@@ESTRING:LocalPort:@</pattern> </patterns> <examples> <example> <test_message program='%ASA'>Built inbound ICMP connection for faddr X.X.X.X/X gaddr X.X.X.X/X laddr X.X.X.X/X</test_message> </example> </examples> </rule> <rule id='8be7928d-66e7-7042-abd5-869d6b49c56e' class='system' provider='%ASA'>  <patterns> <pattern>Built inbound @ESTRING:Protocol: @connection @ESTRING:: @for @ESTRING::@@IPv4:SrcIP:/@@ESTRING:SrcPort: @(@ESTRING::)@ to identity:@IPv4:DstIP:/@@ESTRING:DstPort: @(@ESTRING::)@</pattern> </patterns> <examples> <example> <test_message program='%ASA'>Built inbound UDP connection 55101078 for inside:X.X.X.X/X (X.X.X.X/X) to identity:X.X.X.X/X (X.X.X.X/X)</test_message> </example> </examples> </rule> <rule id='20aee256-b4f0-8b4d-93cb-263d5338fd21' class='system' provider='%ASA'>  <patterns> <pattern>Teardown @ESTRING:Protocol: @connection @ESTRING:: @for @ESTRING:::@@IPv4:SrcIP:/@@ESTRING:SrcPort: @to identity:@IPv4:DstIP:/@@ESTRING:DstPort: @duration@ANYSTRING::@</pattern> </patterns> <examples> <example> <test_message program='%ASA'>Teardown UDP connection 55101084 for inside:X.X.X.X/X to identity:X.X.X.X/X duration 0:02:01 bytes 88</test_message> </example> </examples> </rule> <rule id='e075efdc-c25f-5e49-a208-7661e3b5a29b' class='system' provider='%ASA'>  <patterns> <pattern>Built dynamic @ESTRING:Protocol: @translation from @ESTRING:::@@IPv4:LocalIP:/@@ESTRING:LocalPort: @to @ESTRING:::@@IPv4:GlobalIP:/@@ESTRING:GlobalPort:@</pattern> </patterns> <examples> <example> <test_message program='%ASA'>Built dynamic TCP translation from any:X.X.X.X/X to outside:X.X.X.X/X</test_message> </example> </examples> </rule> <rule provider='%ASA' class='system' id='39'> <patterns> <pattern>Cleared @ESTRING:: @urgent flag from @ESTRING:::@@ESTRING::/@@NUMBER::@ to @ESTRING: ::@@ESTRING::/@@NUMBER::@</pattern> <pattern>regular translation creation failed for @ESTRING:: @src @ESTRING:::@@ESTRING:: @dst @ESTRING: ::@@ESTRING:: @(type @NUMBER::@, code @NUMBER::@</pattern> </patterns> </rule> </rules> </ruleset> </patterndb>

3 6

Problem building syslog-ng 3.11.1 and elastic search
by Ronald Fenner 27 Feb '18

27 Feb '18

I have script that builds syslog-ng and it was working fine but apparently a plugin for elastic search seems to have gone missing and I'm getting this error: 1519328565,,ui,message, amazon-ebs: Could not resolve all files for configuration ':elastic-v2:compileClasspath'. 1519328565,,ui,message, amazon-ebs: > Could not find org.elasticsearch.plugin:shield:2.4.0. 1519328565,,ui,message, amazon-ebs: Searched in the following locations: 1519328565,,ui,message, amazon-ebs: file:/usr/lib/syslog-ng-java-compile-deps-mvn-repo/org/elasticsearch/plugin/shield$ 1519328565,,ui,message, amazon-ebs: file:/usr/lib/syslog-ng-java-compile-deps-mvn-repo/org/elasticsearch/plugin/shield$ 1519328565,,ui,message, amazon-ebs: file:/home/ec2-user/eventlog-0.2.12/syslog-ng-3.11.1/modules/java/syslog-ng-core/l$ 1519328565,,ui,message, amazon-ebs: file:/home/ec2-user/eventlog-0.2.12/syslog-ng-3.11.1/modules/java/syslog-ng-core/l$ 1519328565,,ui,message, amazon-ebs: file:/home/ec2-user/eventlog-0.2.12/syslog-ng-3.11.1/modules/java-modules/common/g$ 1519328565,,ui,message, amazon-ebs: file:/home/ec2-user/eventlog-0.2.12/syslog-ng-3.11.1/modules/java-modules/common/g$ 1519328565,,ui,message, amazon-ebs: https://repo1.maven.org/maven2/org/elasticsearch/plugin/shield/2.4.0/shield… 1519328565,,ui,message, amazon-ebs: https://repo1.maven.org/maven2/org/elasticsearch/plugin/shield/2.4.0/shield… 1519328565,,ui,message, amazon-ebs: http://maven.elasticsearch.org/releases/org/elasticsearch/plugin/shield/2.4… 1519328565,,ui,message, amazon-ebs: 9 actionable tasks: 9 executed 1519328565,,ui,message, amazon-ebs: http://maven.elasticsearch.org/releases/org/elasticsearch/plugin/shield/2.4… 1519328565,,ui,message, amazon-ebs: Required by: 1519328565,,ui,message, amazon-ebs: project :elastic-v2 1519328565,,ui,message, amazon-ebs: 1519328565,,ui,message, amazon-ebs: * Try: 1519328565,,ui,message, amazon-ebs: Run with --stacktrace option to get the stack trace. Run with --info or --debug option t$ 1519328565,,ui,message, amazon-ebs: Is there a way to disable elasticsearch module specifically? Ronald Fenner Network Architect Game Circus LLC. rfenner(a)gamecircus.com

5 9

Directory/file create ownership
by Machiel Richards 27 Feb '18

27 Feb '18

Good day guys I am having an issue that I seem not be able to resolve thus far. I am fairly new to syslog-ng and have setup a new syslog server. We have managed to get most of our servers sending logs to the server without issues, however I am having trouble to get syslog-ng to create the directories and files with the correct ownership. We have a seperate user and group which it needs to create directories with, however it keeps on creating directories and files as root user. below is my current config : # First, set some global options. options { chain_hostnames(off); flush_lines(0); use_dns(yes); use_fqdn(yes); owner("syslog"); group("syslog"); perm(0640); stats_freq(0); bad_hostname("^gconfd$"); log_fifo_size(10000); }; source inputs { system(); internal(); unix-stream("/dev/log"); tcp(max_connections(50)); udp(); }; destination logpile { file("/var/log/HOSTS/$HOST/$YEAR/$MONTH/$FACILITY.$YEAR$MONTH$DAY" owner("syslog") group("syslog") perm(0660) create_dirs(yes) dir_perm(0770)); }; log { source(inputs); destination(logpile); }; Can someone please assist me as any help would be appreciated. Regard -- I am a bomb technician...If you see me running, TRY TO KEEP UP!

1 0

html posting
by Benny Pedersen 27 Feb '18

27 Feb '18

please disable html when posting reply and or questions i say this in generic so not blame some for doing it, but it more easy to read if want help

1 0

Ubuntu syslog-ng (version 3.8.1 or better)
by Alicia Smith 26 Feb '18

26 Feb '18

Can anyone tell me where I could download the syslog-ng ubuntu package for Ubuntu 14.04? I installed from source to get the amqp and format json plugins going but was wondering if there was a precompiled package with these modules available for this version of ubuntu? It would make the process a lot smoother Please let me know! Thanks :) -- Alicia Smith @phrozyn Information Security Engineer asmith(a)mozilla.com

2 1

like to volunteer The syslog-ng project.
by Janaka Nanayakkara 24 Feb '18

24 Feb '18

Dear Mentors of The syslog-ng project, Hello, I would like to take part in your project, I’m a Post graduate student who’s following software engineering degree, And I have experience in Java, C,C++,C#, Phython. I have basic knowledge about CEH (Certified Ethical Hacking).As a new Comer to the GSOC. I have less experienced about how to take part in GSOC. Can you tell me how to take part and help Your project? I would like to help for your project, because I like your concept in syslog-ng. I’m so excited to volunteer your Project to become much more effective. Thank You! Janaka Nanayakkara janaka_nanayakkara(a)aol.com +(94)776980292 Janaka Nanayakkara janaka_nanayakkara(a)aol.com

1 0

syslog-ng in Google Summer of Code (GSoC)
by Czanik, Péter 22 Feb '18

22 Feb '18

Dear syslog-ng users, This year the syslog-ng project participates the Google Summer of Code again. We are looking for students interested in working on an open source project – syslog-ng – and participating in the Google Summer of Code (GSoC). If you are new to GSoC, here are a few words about it (from https://summerofcode.withgoogle.com/) "Google Summer of Code is a global program focused on bringing more student developers into open source software development. Students work with an open source organization on a 3 month programming project during their break from school." "Students work for an open source software organization, and earn a stipend for successfully completing the project." In our view, being part of an open source community is a great learning experience with exposure to many aspects of software development. During university and later at the workplace, most people lack this kind of experience, while many of the open source projects are actually the cornerstones of IT. A good example is of course Linux, but there are many more similar projects, such as: Kubernetes, GIMP, Blender, Docker, and the list could go on. We think Google Summer of Code – and syslog-ng in particular – is a great opportunity to start contributing to an open source project, gain invaluable experience, and get that reference that you can put on your CV. :-) If you are interested, go to the syslog-ng page of GSoC, https://summerofcode.withgoogle.com/organizations/4514364854370304/, where you can find contact information, project ideas, and details about how to apply. If you are a teacher or you know students for any other reasons, make sure that your students know about syslog-ng and that they check our project idea list! Please re-tweet this tweet ( https://twitter.com/PCzanik/status/966608289910919168 ) to reach as many students (and their teachers) as possible! Thank you for your help, Peter Peter Czanik (CzP) <peter.czanik(a)balabit.com> Balabit / syslog-ng upstream https://syslog-ng.com/blog/author/peterczanik/ https://twitter.com/PCzanik

1 0

multiple file sources, worked - some have now gone silent
by Declan White 21 Feb '18

21 Feb '18

I'm looking at a syslog-ng client that has multiple file() sources feeding a single destination unix-domain socket with flags(flow-control) and some disk-buffer. Two days ago it was sending all these files and tracking their rotations fine etc. The destination server restarted, so the socket disappeared for a while, and the client ran out of the default buffering. The intention is that it recovers when the destination returns and resumes reading the files where it left off. When the server restarted data started flowing again. But I've noticed it's no longer sending data from ALL of the file sources. Some have gone silent. The log is saying nothing about the frozen sources. The statistics counters simply stopped incrementing for them. This is kind of fatal. Has anyone seen this before or know how to avoid it? The faulty instance is still running, so if there's any way of interrogating it to find out its mental state, that's still possible. syslog-ng 3.12.1 OS: Solaris -- Declan White

4 7

enforcing file permissions
by Nik Ambrosch 20 Feb '18

20 Feb '18

i know it's possible to create a file with permissions using owner() and group() however if the file already exists with different permissions or the permissions are changed later syslog-ng does not change the permissions back. in a future release is it possible to change the behavior of owner() and group() to enforce permissions or possibly add a new option like enforce-permissions(true) with a default of false (to mimic the current behavior)? maybe something exists already and i'm just missing it :) thanks all.

2 1

syslog-ng 3.13 key-value parser crashes and aborts
by Don C 19 Feb '18

19 Feb '18

Hi, I was upgrading syslog-ng from 3.12 to 3.13 using the prebuilt unofficial RPMs. I was testing my config on 3.13 and ran into the following issue where the key-value parsing aborts.This appears to be a regression in 3.13, the abort never happened in 3.12. If I remove the kv_parser from my config, there is no issue. Here's the end of the debug and verbose output I get. Notice the garbage values in the key-value names parsed from the message. [2018-01-25T17:33:08.924895] Filter rule evaluation result; msg='0x7fc0a807b370', result='match', rule='f_compliant_hosts', location='/etc/syslog-ng/syslog-ng.conf:66:27' [2018-01-25T17:33:08.924919] Setting value; msg='0x7fc0a80612f0', name='@\x15\x01pÀ', value='2017-11-21 19:11:24.817041' [2018-01-25T17:33:08.924930] Setting value; msg='0x7fc0a80612f0', name='', value='debug' [2018-01-25T17:33:08.924936] Setting value; msg='0x7fc0a80612f0', name='f_compliant_hosts', value='[robotnats kafka transport prepared]' [2018-01-25T17:33:08.924944] Setting value; msg='0x7fc0a80612f0', name='PK\x06¨À', value='ROBOT_serviceTracking_neo4j-topo-svc-2717019760-1sdn1' [2018-01-25T17:33:08.924952] Setting value; msg='0x7fc0a80612f0', name='', value='nats://nats:4222' [2018-01-25T17:33:08.924957] Setting value; msg='0x7fc0a80612f0', name='', value='[robot-kafka:9092]' [2018-01-25T17:33:08.924963] Setting value; msg='0x7fc0a80612f0', name='kv_kafka', value='robotnats_kafka' [2018-01-25T17:33:08.924969] Setting value; msg='0x7fc0a80612f0', name=' »\x08¬À', value='ROBOT_serviceTracking_neo4j-topo-svc-2717019760-1sdn1' [2018-01-25T17:33:08.924975] Message parsing complete; result='1', rule='p_kv', location='/etc/syslog-ng/syslog-ng.conf:75:5' [2018-01-25T17:33:08.924998] Incoming log entry; line='time="2017-11-21 19:11:24.817095" level=info msg="[skeleton core configuration parse stage complete]" config=/etc/robot/servicetracking.conf debug=true logfile= maxthreads=16 tag=ROBOT_serviceTracking_neo4j-topo-svc-2717019760-1sdn1 version=' [2018-01-25T17:33:08.925030] Setting value; msg='0x7fc0a807bd60', name='MESSAGE', value='time="2017-11-21 19:11:24.817095" level=info msg="[skeleton core configuration parse stage complete]" config=/etc/robot/servicetracking.conf debug=true logfile= maxthreads=16 tag=ROBOT_serviceTracking_neo4j-topo-svc-2717019760-1sdn1 version=' [2018-01-25T17:33:08.925040] Setting value; msg='0x7fc0a807bd60', name='HOST_FROM', value='syslog-ng-logging-1040168119-qsmrf' [2018-01-25T17:33:08.925046] Setting value; msg='0x7fc0a807bd60', name='HOST', value='syslog-ng-logging-1040168119-qsmrf/syslog-ng-logging-1040168119-qsmrf' [2018-01-25T17:33:08.925053] Setting value; msg='0x7fc0a807bd60', name='FILE_NAME', value='/mnt/logfs/neo4j-topo-svc/1/servicetracking_stdout.log' [2018-01-25T17:33:08.925057] Setting value; msg='0x7fc0a807bd60', name='SOURCE', value='s_file_stdout' [2018-01-25T17:33:08.925062] Requesting flow control; location='/etc/syslog-ng/syslog-ng.conf:81:5' [2018-01-25T17:33:08.824606] Incoming log entry; line='time="2017-11-21 19:10:56.399341" level=debug msg="[section not required]" section=etcd tag=ROBOT_serviceTracking_robot-topo-svc-2107321943-48rnw' [2018-01-25T17:33:09.022629] Filter rule evaluation begins; msg='0x7fc0a807bd60', rule='f_compliant_hosts', location='/etc/syslog-ng/syslog-ng.conf:66:27' [2018-01-25T17:33:08.824700] Setting value; msg='0x7fc0a409b810', name='HOST_FROM', value='syslog-ng-logging-1040168119-qsmrf' # # If you would like to submit a bug report, please visit: # http://bugreport.java.com/bugreport/crash.jsp # The crash happened outside the Java Virtual Machine in native code. # See problematic frame for where to report the bug. # ... [2018-01-25T17:33:09.022911] Requesting flow control; location='/etc/syslog-ng/syslog-ng.conf:81:5' [2018-01-25T17:33:09.022916] Filter rule evaluation begins; msg='0x7fc08c0f5550', rule='f_compliant_hosts', location='/etc/syslog-ng/syslog-ng.conf:66:27' [2018-01-25T17:33:09.022922] Filter node evaluation result; msg='0x7fc08c0f5550', result='not-match', type='==' [2018-01-25T17:33:09.022927] Filter rule evaluation result; msg='0x7fc08c0f5550', result='not-match', rule='f_compliant_hosts', location='/etc/syslog-ng/syslog-ng.conf:66:27' [2018-01-25T17:33:09.022932] Filter rule evaluation begins; msg='0x7fc08c0f5550', rule='f_noncompliant_hosts', location='/etc/syslog-ng/syslog-ng.conf:70:30' [2018-01-25T17:33:09.022937] Filter node evaluation result; msg='0x7fc08c0f5550', result='not-match', type='==' [2018-01-25T17:33:09.022942] Filter node evaluation result; msg='0x7fc08c0f5550', result='match', type='filter(f_compliant_hosts)' [2018-01-25T17:33:09.022947] Filter rule evaluation result; msg='0x7fc08c0f5550', result='match', rule='f_noncompliant_hosts', location='/etc/syslog-ng/syslog-ng.conf:70:30' Aborted Is this a known issue with the key-value parser in 3.13? Regards, Don

3 2

vCenter 6.5 RFC5425 prefacing messages with a space
by Evan Rempel 18 Feb '18

18 Feb '18

Our VMWare team has been busy upgrading out infrastructure to 6.5 which is great, but the syslogs started showing up as 2018-02-17T07:00:12 esx.host.name auth.info 1 2018-02-17T07:00:12.162028-08:00 esx.host.name sshd 4662 - - Did not receive identification string from 142.104.139.163 port 51088 which I recognise as RFC5425, So I added the flags(syslog-protocol) to my network source definition. Now the log message arrives much nicer as 2018-02-17T07:48:12 esx.host.name auth.info sshd[20835]: Did not receive identification string from 142.104.139.163 port 42060 but if you look very closely you will notice that there are 2 spaces in "sshd[20835]: Did" and in the original logged message there were 2 spaces preceding the word "Did". Has anyone seen this before (or currently with your own VMWare 6.5). I'm trying to figure out if this is a bug in syslog-ng or a bug in vCenter 6.5 or a config error on my systems. Thanks, Evan.

2 1

confgen not usable (by me) 3.13.2 on Redhat 7
by Evan Rempel 18 Feb '18

18 Feb '18

I am using the copr RPM build syslog-ng-3.13.2-3.el7.centos.x86_64 on a redhat 7 system. I am trying to use the confgen to create a value and use it in a rewrite. I realize that I can use $LOGHOST in this context but am using it as an example. @module confgen context(rewrite) name(z_myself) exec("/usr/bin/hostname") rewrite r_local { set( "z_myself()", value("HOST") ); }; log { source(local); rewrite(r_local); destination(d_var_syslog); }; d_var_syslog is any file destination to capture the log message. The above configuration actually sets the hostname in the log line to be "z_myself()". 2018-02-17T21:15:39.220-08:00 z_myself() solaris-cron.err erempel: just testing if I make the rewrite rewrite r_local { set( z_myself(), value("HOST") ); }; I get syntax errors. The top example works on redhat 6 with syslog-ng 3.9 I can't find any documentation on how to use a block (confgen) define in a rewrite statement. I have tried back ticks, $(z_myself()) macros "`z_myself()`" and any other combination but they all result in syslog-ng failing to start. What am I doing wrong? Evan.

2 1

Re: [syslog-ng] QuickQ syslog source drivers
by Balazs Scheidler 17 Feb '18

17 Feb '18

Yes, recent versions would accept normal bsd like messages. The key to watch out is the framing, as that needs to be correct. syslog(transport(tcp)) expects rfc5425 (or rfc5426, cant remember which) style framing network(transport(tcp) flags(syslog-protocol)) accepts both old and new formats without framing syslog(transport(udp)) has no framing, so again it accepts 5424 and 3164 and uses udp. Actually, we may want to imply flags(syslog-protocol) everywhere in the future, there's an easy way to autodetect the format. On Feb 16, 2018 21:31, "Scot" <scotrn(a)gmail.com> wrote: Will the syslog() source driver support both RFC5424 and RFC3164 from different hosts at the same time in flight? source s_syslog_tcp { syslog(port(514) transport("tcp")); }; source s_syslog_udp { syslog(port(514) transport("udp")); }; ____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/? product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq

1 0

QuickQ syslog source drivers
by Scot 16 Feb '18

16 Feb '18

Will the syslog() source driver support both RFC5424 and RFC3164 from different hosts at the same time in flight? source s_syslog_tcp { syslog(port(514) transport("tcp")); }; source s_syslog_udp { syslog(port(514) transport("udp")); };

1 0

syslog-ng destination to multiple mssql database
by Garry De Toffoli 16 Feb '18

16 Feb '18

Hello, I have configured syslog.ng OSE version 3.5.3 to log to a MSSQL destination and that all works well. However when I have specified 2 MSSQL destination, same SQL Server, same table on different database, syslog-ng have problem to insert on database, both on the first and on the second. Some time will insert, sometimes not. On /var/log/syslog I have found some lines reporting error CREATE TABLE on the second database, Restarting the service, sudo service syslog-ng restart, takes a long time. When I comment out the destination of the 2nd database, syslog-ng run very well. Is it a limit of the OSE version? Or may be an error on the configuration file? See attached my configuration file. Thank you for the patience. ----------------------- Garry De Toffoli -----------------------

2 1

tcp wrappers
by Czanik, Péter 15 Feb '18

15 Feb '18

Hi, I'm running a poll about tcp wrappers on twitter: https://twitter.com/PCzanik/status/961985140128911361 but feedback is also very welcome here: Is there anybody still using tcp wrappers support built into syslog-ng on Fedora / RHEL / CentOS? Fedora rawhide removed support for tcp wrappers, and if nobody opposes I'd remove it from the rest of my pacages as well to keep packaging simple. Some background info for those who don't even know what I'm talking about: https://fedoraproject.org/wiki/Changes/Deprecate_TCP_wrappers (In short: it's really archaic stuff I last time used in 1995... :) ) Bye, Peter Czanik (CzP) <peter.czanik(a)balabit.com> Balabit / syslog-ng upstream https://syslog-ng.com/blog/author/peterczanik/ https://twitter.com/PCzanik

3 3

Umask funkiness
by Declan White 10 Feb '18

10 Feb '18

I have a directory owned by the syslog-ng user. Its group however belongs to a group of which the user is not a member. The directory is g+s, so that all files and dirs made within it inherit the group owner (and the g+s in the case of dirs). syslog-ng is running with a umask of 022 (interrogated running process to be sure). The file("/dir/${FOO}/${BAR}") destination driver has : create-dirs(yes) perm() dir-owner() dir-group() dir-perm() i.e. "don't change any perms" The aim of the game is to end up with files and dirs readable, but not writable, by the inherited group owner. I can't get it working. I am always ending up with drwx--S--- dirs and -rw------- files Solaris. syslog-ng-3.12.1 -- Declan White

4 7

Re: [syslog-ng] problem elasticsearch2 creating index per HOST
by Abe Lebo 08 Feb '18

08 Feb '18

ah! that must be it! I was not aware that upper case characters are not allowed in elasticsearch index names! $HOST = B8-27-AB-23-11-26 Thanks a million Fabien. Op 8-2-2018 om 10:00 schreef Fabien Wernli: > On Thu, Feb 08, 2018 at 09:42:38AM +0100, Abe Lebo wrote: >> i have templates, but not one specific for this index patern. >> >> i see no errors in the logs, i only see the indices being created if i do >> not add the $HOST, but only $YEAR.$MONTH.$DAY >> >> i'll see if i can set syslog-ng logging to debug > > yes, definitely try that: syslog-ng -Fdv > > One other thing, what does your $HOST resolve to? There are some > restrictions in ES index names, perhaps your macro contains upper case or > other illegal chars? > >

2 1

Insider 2018-02: New website; One Identity; Docker; Error messages; Elastic 6; DevConf;
by Czanik, Péter 08 Feb '18

08 Feb '18

Dear syslog-ng users, This is the 64th issue of syslog-ng Insider, a monthly newsletter that brings you syslog-ng-related news. NEWS New syslog-ng website --------------------- Our new website – http://syslog-ng.com/ – is now live. It is a central hub of information for all syslog-ng related content, including commercial products built around syslog-ng. All documentation and blogs were migrated here. The old syslog-ng.org site will go away once all open source related content found its place on the new site. Feedback is much appreciated! Balabit acquired by One Identity -------------------------------- You might have already heard that Balabit, the company behind syslog-ng, has been acquired by One Identity (http://www.oneidentity.com) and you may be wondering what would happen to syslog-ng in the future. Learn how Balázs Scheidler, original author of syslog-ng, sees the future: https://lists.balabit.hu/pipermail/syslog-ng/2018-January/024254.html Tl;dr: nothing to worry about! Logging in Docker using syslog-ng --------------------------------- Collecting logs can be especially important in a containerized environment, where you start and stop containers around the clock, and the container might not even exist anymore by the time you investigate an event. Read this white paper to find out how to: Run your central log server in Docker Collect Docker infrastructure logs using syslog-ng Collect logs from containers using Docker volumes https://syslog-ng.com/white-papers Common syslog-ng error messages and their solutions --------------------------------------------------- In this post, we explain a few common syslog-ng error and warning messages, what they mean, and how to solve them. https://syslog-ng.com/blog/common-syslog-ng-error-messages-solutions/ Getting started with Elasticsearch 6 on RHEL/CentOS --------------------------------------------------- Version 6 of the Elastic Stack has now been available for some time, packed with new features and improved performance. Compatibility of syslog-ng was checked already during the alpha phase of development, as syslog-ng is becoming popular among Elasticsearch users: it can greatly simplify logging to Elasticsearch. There are no major changes from a syslog-ng point of view but – to improve your copy & paste experience – I updated my getting started guide from Elastic Stack 5 to 6. https://syslog-ng.com/blog/syslog-ng-and-elasticsearch-6-getting-started-on… DevConf 2018 ------------ DevConf is a yearly conference for developers, administrators, and users of Linux and related technologies. It is organized by Red Hat in Brno, home to one of their major development centers. Since the majority of syslog-ng users are running their logging server on Red Hat Enterprise Linux or CentOS, it is important to know their needs and the directions where RHEL is heading. It is also a good possibility to discuss logging-related questions with Red Hat engineers in person. https://syslog-ng.com/blog/devconf-long-live-containerization/ UPCOMING EVENTS SCALE: Logging Docker using syslog-ng: https://www.socallinuxexpo.org/scale/16x/presentations/logging-docker-using… Pasadena Convention Center March 8 - 11, 2018 Your feedback and news, or tips about the next issue are welcome at documentation(a)balabit.com. To read this newsletter online, visit: https://syslog-ng.org/ Peter Czanik (CzP) <peter.czanik(a)balabit.com> Balabit / syslog-ng upstream https://syslog-ng.com/blog/author/peterczanik/ https://twitter.com/PCzanik

1 0

problem elasticsearch2 creating index per HOST
by Abe Lebo 07 Feb '18

07 Feb '18

LS, i want to create an Elasticsearch index per host i receive logs from (also syslog-ng) , somehow that fails. in have a setup that creates an index per day (see below), that works. I can build a file output per host per day by adding the $HOST to the filename, that works too.(see below) but if add $HOST to the indexname it no longer creates indices. Anybody know what's wrong? Thanks in advance, Abel ...... destination d_testfile { file( "/tmp/testlogabel-$HOST$YEAR$MONTH" flags(syslog-protocol) template("$(format-json @timestamp=$ISODATE --scope nv-pairs)\n") ); }; ################################################################################ # Elasticsearch destination ################################################################################ destination d_es_per_day { elasticsearch2( client-lib-dir("/usr/share/elasticsearch/lib/:/usr/lib/syslog-ng/3.13/java-modules/") index("test-abel_$YEAR.$MONTH.$DAY") type("test") cluster-url("http://node01:9400 http://node02:9400 http://node03:9400") client-mode("http") template("$(format-json @timestamp=$ISODATE --scope nv-pairs)\n") ); }; ###### # sending logs to ES destination log { source(s_net); source(s_src); filter(f_abel); parser(p_abel); destination(d_es_per_day); flags(flow-control); };

2 2

OSE, Shared Storage, & Load Balanced syslog Messages
by Pierce, Stevan - PHX 07 Feb '18

07 Feb '18

I am working on setting up OSE as a relay and would like to have storage set aside in case a destination upstream from the relay goes down. How does load balancing syslog messages work when clustering at the OS level? If I am sharing storage across multiple relays, how would I go about setting it up so that messages from one server does not overwrite another server? swp

2 1

Rsyslog relay or syslog-ng ?
by Scot 06 Feb '18

06 Feb '18

Hi, Hoping someone has seen an easy fix for this. Sorry is it's specifically referenced somewhere I'm not seeing. Dealing with a vendor who is not able to leverage the RFC headers or TCP input. We have rsyslog relays in remote sites sending TCP/514 to syslog-ng and others locally sending directly to syslog-ng TCP/UDP 514. The devices sending directly to syslog-ng are reporting to the IDS correctly. Hosts relaying through rsyslog are showing a source address of the relay. */etc/rsyslog.d/forward.conf* *$ActionQueueFileName fwdRule1 # unique name prefix for spool files* *$ActionQueueMaxDiskSpace 1g # 1gb space limit (use as much as possible)* *$ActionQueueSaveOnShutdown on # save messages to disk on shutdown* *$ActionQueueType LinkedList # run asynchronously* *$ActionResumeRetryCount -1 # infinite retries if host is down* *# remote host is: name/ip:port, e.g. 192.168.0.1:514 <http://192.168.0.1:514>, port optional* **.* @@syslog-ngIP:514* */etc/syslog-ng/conf.d* *source s_net_tcp {tcp(ip(0.0.0.0) port(514) max-connections(300) keep_hostname(yes) so_rcvbuf(262142) log_iw_size(25000)); };* *source s_net_udp {syslog(ip(0.0.0.0) keep_hostname(yes) port(514) transport("udp") flags(no-hostname) so_rcvbuf(262142));};* *destination d_ids {network("IDSHOSTNAME" spoof_source(yes) transport(udp) port(514) flags(syslog-protocol)); };* *log { source(s_net_udp);* * channel {filter(f_ids); destination (d_ids);};* * channel {parser(pattern_db); destination (d_es);};* *};* *log { source(s_net_tcp);* * channel {filter(f_ids); destination (d_ids);};* * channel {parser(pattern_db); destination (d_es);};* *};*

3 4

RFC5424 sending - dest=unix-stream. Framing?
by Declan White 05 Feb '18

05 Feb '18

I have syslog-ng pumping data into a unix-stream destination socketfilething. Data going into this domain socket has lots of adventures, and then emerges as a source unix-stream on another host. Both have flags(syslog-protocol), but the data gets framing errors at the receiver. I think it's become a bytestream (instead of a message stream) enroute, and the recv() is relying on only getting one message at a time, because it can't know the message length any other way, as unix-stream gets messages without framing(?). Is there a way to salvage this? (I noticed TLS might add framing, but since the tunnel is already TLS ...) -- Declan White

2 5