February 2018 - syslog-ng - lists.balabit.hu

One source multiple destinations ?
by Scot 10 Mar '18

10 Mar '18

I'm having a problem where I am trying to take input source(s) and write them out to multiple destinations. Before I go barking up the wrong tree I just wanted to make sure I wasn't missing something. We should be able to take a source and send it to file, elastic-search and SPLUNK and sumologic all at the same time right ? Troubleshooting an odd behavior where only one network destination will work but then I switch the order the other starts working. I know it's vague but has anyone seen this behavior? Thanks Scot

4 10

Re: [syslog-ng] Directory/file create ownership
by Dwijadas Dey 07 Mar '18

07 Mar '18

Add create_dirs (yes) in the options section. On 27-Feb-2018 2:24 PM, "Machiel Richards" <machiel.richards(a)gmail.com> wrote: Good day guys I am having an issue that I seem not be able to resolve thus far. I am fairly new to syslog-ng and have setup a new syslog server. We have managed to get most of our servers sending logs to the server without issues, however I am having trouble to get syslog-ng to create the directories and files with the correct ownership. We have a seperate user and group which it needs to create directories with, however it keeps on creating directories and files as root user. below is my current config : # First, set some global options. options { chain_hostnames(off); flush_lines(0); use_dns(yes); use_fqdn(yes); owner("syslog"); group("syslog"); perm(0640); stats_freq(0); bad_hostname("^gconfd$"); log_fifo_size(10000); }; source inputs { system(); internal(); unix-stream("/dev/log"); tcp(max_connections(50)); udp(); }; destination logpile { file("/var/log/HOSTS/$HOST/$YEAR/$MONTH/$FACILITY.$YEAR$ MONTH$DAY" owner("syslog") group("syslog") perm(0660) create_dirs(yes) dir_perm(0770)); }; log { source(inputs); destination(logpile); }; Can someone please assist me as any help would be appreciated. Regard -- I am a bomb technician...If you see me running, TRY TO KEEP UP! ____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/? product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq

5 8

Cisco ASA parsing with patterndb/elasticsearch
by Tim Ghetti 01 Mar '18

01 Mar '18

Hi - having some trouble getting paterndb functional and looking for some help. I would like to use patterndb to parse my cisco ass firewall logs before sending it to elasticsearch. However when the messages get to elasticsearch, I don't see the messages being parsed. Running pdbtool against the logs seems to work. # pdbtool match -p /etc/syslog-ng/patterndb.d/ciscoasa.pdb -P %ASA -f /var/log/asatest.log |more HOST=X.X.X.X MESSAGE=Built dynamic TCP translation from INSIDE:X.X.X.X/X to OUTSIDE:X.X.X.X/X PROGRAM=%ASA-6-305011 LEGACY_MSGHDR=%ASA-6-305011: .classifier.class=system .classifier.rule_id=e075efdc-c25f-5e49-a208-7661e3b5a29b Protocol=TCP GlobalIP=X.X.X.X GlobalPort=X LocalIP=X.X.X.X LocalPort=X TAGS=.classifier.system ********************** SYSLOG-NG CONF FILE @version: 3.11 source s_network { tcp(); udp(); }; destination d_elastic { elasticsearch2( client-mode("http") cluster("ITESCL001") index("logstash-syslogng_${YEAR}.${MONTH}.${DAY}") cluster-url("http://X.X.X.X:9200") type("syslog") flush-limit("1") ); }; destination d_catchall { file("/var/log/catchall.log"); }; filter f_ciscoasa { host("X.X.X.X"); }; parser p_ciscoasa {db-parser(file("/etc/syslog-ng/patterndb.d/ciscoasa.pdb"));}; log { source(s_network); filter(f_ciscoasa); parser(p_ciscoasa); destination(d_elastic); flags(final, flow-control); }; log { source(s_network); destination(d_catchall); }; ********************** PATTERNDB FILE <?xml version='1.0' encoding='UTF-8'?> <patterndb version='4' pub_date='2018-02-19'> <ruleset name='%ASA' id='a300d776-8bd7-834d-a4a9-23eb81a4b3ba'> <pattern>%ASA</pattern> <description> This ruleset covers the Cisco ASA firewalls </description> <rules> <rule provider="%ASA" id="b3de7699-8213-c744-944e-9413298afe86" class="system">  <patterns> <pattern>Teardown @ESTRING:Protocol: @connection for faddr @IPv4:SrcIP:/@@ESTRING:SrcPort: @gaddr @IPv4:GlobalIP:/@@ESTRING:GlobalPort: @laddr @IPv4:LocalIP:/@@ESTRING:LocalPort:@</pattern> </patterns> <examples> <example> <test_message program='%ASA'>Teardown ICMP connection for faddr X.X.X.X/X gaddr X.X.X.X/X laddr X.X.X.X/X</test_message> </example> </examples> </rule> <rule id='90d0f8c9-7591-d44e-b886-2f7e5cb17ce6' class='system' provider='%ASA'>  <patterns> <pattern>Teardown dynamic @ESTRING:Protocol: @translation from @ESTRING:::@@IPv4:LocalIP:/@@ESTRING:LocalPort: @to @ESTRING:::@@IPv4:GlobalIP:/@@ESTRING:GlobalPort: @duration@ANYSTRING::@</pattern> </patterns> <examples> <example> <test_message program='%ASA'>Teardown dynamic UDP translation from any:X.X.X.X/X to outside:X.X.X.X/X duration 0:00:00</test_message> </example> </examples> </rule> <rule id='8f0a8d57-80c6-4745-8a8a-5ce018bb0d87' class='system' provider='%ASA'>  <patterns> <pattern>Teardown @ESTRING:Protocol: @connection @ESTRING:: @for @ESTRING:::@@IPv4:DstIP:/@@ESTRING:DstPort: @to @ESTRING:::@@IPv4:SrcIP:/@@ESTRING:SrcPort: @@ESTRING::@</pattern> </patterns> <examples> <example> <test_message program='%ASA'>Teardown UDP connection 55101037 for outside:X.X.X.X/X to inside:X.X.X.X/X duration 0:00:00 bytes 132</test_message> </example> </examples> </rule> <rule id='00c0732d-1e34-7340-a75f-21198bf71137' class='system' provider='%ASA'>  <patterns> <pattern>Built outbound @ESTRING:Protocol: @connection @ESTRING:: @for @ESTRING:::@@IPv4:DstIP:/@@ESTRING:DstPort: @(@ESTRING::)@ to @ESTRING:::@@IPv4:SrcIP:/@@ESTRING:SrcPort: @(@ESTRING::)@</pattern> </patterns> <examples> <example> <test_message program='%ASA'>Built outbound UDP connection 55101037 for outside:X.X.X.X/X (X.X.X.X/X) to inside:X.X.X.X/X (X.X.X.X/X)</test_message> </example> </examples> </rule> <rule id='4a586711-ebe2-dc4d-bf6e-e512666d8c5d' class='system' provider='%ASA'>  <patterns> <pattern>Built inbound @ESTRING:Protocol: @connection for faddr @IPv4:SrcIP:/@@ESTRING:SrcPort: @gaddr @IPv4:GlobalIP:/@@ESTRING:GlobalPort: @laddr @IPv4:LocalIP:/@@ESTRING:LocalPort:@</pattern> </patterns> <examples> <example> <test_message program='%ASA'>Built inbound ICMP connection for faddr X.X.X.X/X gaddr X.X.X.X/X laddr X.X.X.X/X</test_message> </example> </examples> </rule> <rule id='8be7928d-66e7-7042-abd5-869d6b49c56e' class='system' provider='%ASA'>  <patterns> <pattern>Built inbound @ESTRING:Protocol: @connection @ESTRING:: @for @ESTRING::@@IPv4:SrcIP:/@@ESTRING:SrcPort: @(@ESTRING::)@ to identity:@IPv4:DstIP:/@@ESTRING:DstPort: @(@ESTRING::)@</pattern> </patterns> <examples> <example> <test_message program='%ASA'>Built inbound UDP connection 55101078 for inside:X.X.X.X/X (X.X.X.X/X) to identity:X.X.X.X/X (X.X.X.X/X)</test_message> </example> </examples> </rule> <rule id='20aee256-b4f0-8b4d-93cb-263d5338fd21' class='system' provider='%ASA'>  <patterns> <pattern>Teardown @ESTRING:Protocol: @connection @ESTRING:: @for @ESTRING:::@@IPv4:SrcIP:/@@ESTRING:SrcPort: @to identity:@IPv4:DstIP:/@@ESTRING:DstPort: @duration@ANYSTRING::@</pattern> </patterns> <examples> <example> <test_message program='%ASA'>Teardown UDP connection 55101084 for inside:X.X.X.X/X to identity:X.X.X.X/X duration 0:02:01 bytes 88</test_message> </example> </examples> </rule> <rule id='e075efdc-c25f-5e49-a208-7661e3b5a29b' class='system' provider='%ASA'>  <patterns> <pattern>Built dynamic @ESTRING:Protocol: @translation from @ESTRING:::@@IPv4:LocalIP:/@@ESTRING:LocalPort: @to @ESTRING:::@@IPv4:GlobalIP:/@@ESTRING:GlobalPort:@</pattern> </patterns> <examples> <example> <test_message program='%ASA'>Built dynamic TCP translation from any:X.X.X.X/X to outside:X.X.X.X/X</test_message> </example> </examples> </rule> <rule provider='%ASA' class='system' id='39'> <patterns> <pattern>Cleared @ESTRING:: @urgent flag from @ESTRING:::@@ESTRING::/@@NUMBER::@ to @ESTRING: ::@@ESTRING::/@@NUMBER::@</pattern> <pattern>regular translation creation failed for @ESTRING:: @src @ESTRING:::@@ESTRING:: @dst @ESTRING: ::@@ESTRING:: @(type @NUMBER::@, code @NUMBER::@</pattern> </patterns> </rule> </rules> </ruleset> </patterndb>

3 6

Problem building syslog-ng 3.11.1 and elastic search
by Ronald Fenner 27 Feb '18

27 Feb '18

I have script that builds syslog-ng and it was working fine but apparently a plugin for elastic search seems to have gone missing and I'm getting this error: 1519328565,,ui,message, amazon-ebs: Could not resolve all files for configuration ':elastic-v2:compileClasspath'. 1519328565,,ui,message, amazon-ebs: > Could not find org.elasticsearch.plugin:shield:2.4.0. 1519328565,,ui,message, amazon-ebs: Searched in the following locations: 1519328565,,ui,message, amazon-ebs: file:/usr/lib/syslog-ng-java-compile-deps-mvn-repo/org/elasticsearch/plugin/shield$ 1519328565,,ui,message, amazon-ebs: file:/usr/lib/syslog-ng-java-compile-deps-mvn-repo/org/elasticsearch/plugin/shield$ 1519328565,,ui,message, amazon-ebs: file:/home/ec2-user/eventlog-0.2.12/syslog-ng-3.11.1/modules/java/syslog-ng-core/l$ 1519328565,,ui,message, amazon-ebs: file:/home/ec2-user/eventlog-0.2.12/syslog-ng-3.11.1/modules/java/syslog-ng-core/l$ 1519328565,,ui,message, amazon-ebs: file:/home/ec2-user/eventlog-0.2.12/syslog-ng-3.11.1/modules/java-modules/common/g$ 1519328565,,ui,message, amazon-ebs: file:/home/ec2-user/eventlog-0.2.12/syslog-ng-3.11.1/modules/java-modules/common/g$ 1519328565,,ui,message, amazon-ebs: https://repo1.maven.org/maven2/org/elasticsearch/plugin/shield/2.4.0/shield… 1519328565,,ui,message, amazon-ebs: https://repo1.maven.org/maven2/org/elasticsearch/plugin/shield/2.4.0/shield… 1519328565,,ui,message, amazon-ebs: http://maven.elasticsearch.org/releases/org/elasticsearch/plugin/shield/2.4… 1519328565,,ui,message, amazon-ebs: 9 actionable tasks: 9 executed 1519328565,,ui,message, amazon-ebs: http://maven.elasticsearch.org/releases/org/elasticsearch/plugin/shield/2.4… 1519328565,,ui,message, amazon-ebs: Required by: 1519328565,,ui,message, amazon-ebs: project :elastic-v2 1519328565,,ui,message, amazon-ebs: 1519328565,,ui,message, amazon-ebs: * Try: 1519328565,,ui,message, amazon-ebs: Run with --stacktrace option to get the stack trace. Run with --info or --debug option t$ 1519328565,,ui,message, amazon-ebs: Is there a way to disable elasticsearch module specifically? Ronald Fenner Network Architect Game Circus LLC. rfenner(a)gamecircus.com

5 9

Directory/file create ownership
by Machiel Richards 27 Feb '18

27 Feb '18

Good day guys I am having an issue that I seem not be able to resolve thus far. I am fairly new to syslog-ng and have setup a new syslog server. We have managed to get most of our servers sending logs to the server without issues, however I am having trouble to get syslog-ng to create the directories and files with the correct ownership. We have a seperate user and group which it needs to create directories with, however it keeps on creating directories and files as root user. below is my current config : # First, set some global options. options { chain_hostnames(off); flush_lines(0); use_dns(yes); use_fqdn(yes); owner("syslog"); group("syslog"); perm(0640); stats_freq(0); bad_hostname("^gconfd$"); log_fifo_size(10000); }; source inputs { system(); internal(); unix-stream("/dev/log"); tcp(max_connections(50)); udp(); }; destination logpile { file("/var/log/HOSTS/$HOST/$YEAR/$MONTH/$FACILITY.$YEAR$MONTH$DAY" owner("syslog") group("syslog") perm(0660) create_dirs(yes) dir_perm(0770)); }; log { source(inputs); destination(logpile); }; Can someone please assist me as any help would be appreciated. Regard -- I am a bomb technician...If you see me running, TRY TO KEEP UP!

1 0

html posting
by Benny Pedersen 27 Feb '18

27 Feb '18

please disable html when posting reply and or questions i say this in generic so not blame some for doing it, but it more easy to read if want help

1 0

Ubuntu syslog-ng (version 3.8.1 or better)
by Alicia Smith 26 Feb '18

26 Feb '18

Can anyone tell me where I could download the syslog-ng ubuntu package for Ubuntu 14.04? I installed from source to get the amqp and format json plugins going but was wondering if there was a precompiled package with these modules available for this version of ubuntu? It would make the process a lot smoother Please let me know! Thanks :) -- Alicia Smith @phrozyn Information Security Engineer asmith(a)mozilla.com

2 1

like to volunteer The syslog-ng project.
by Janaka Nanayakkara 24 Feb '18

24 Feb '18

Dear Mentors of The syslog-ng project, Hello, I would like to take part in your project, I’m a Post graduate student who’s following software engineering degree, And I have experience in Java, C,C++,C#, Phython. I have basic knowledge about CEH (Certified Ethical Hacking).As a new Comer to the GSOC. I have less experienced about how to take part in GSOC. Can you tell me how to take part and help Your project? I would like to help for your project, because I like your concept in syslog-ng. I’m so excited to volunteer your Project to become much more effective. Thank You! Janaka Nanayakkara janaka_nanayakkara(a)aol.com +(94)776980292 Janaka Nanayakkara janaka_nanayakkara(a)aol.com

1 0

syslog-ng in Google Summer of Code (GSoC)
by Czanik, Péter 22 Feb '18

22 Feb '18

Dear syslog-ng users, This year the syslog-ng project participates the Google Summer of Code again. We are looking for students interested in working on an open source project – syslog-ng – and participating in the Google Summer of Code (GSoC). If you are new to GSoC, here are a few words about it (from https://summerofcode.withgoogle.com/) "Google Summer of Code is a global program focused on bringing more student developers into open source software development. Students work with an open source organization on a 3 month programming project during their break from school." "Students work for an open source software organization, and earn a stipend for successfully completing the project." In our view, being part of an open source community is a great learning experience with exposure to many aspects of software development. During university and later at the workplace, most people lack this kind of experience, while many of the open source projects are actually the cornerstones of IT. A good example is of course Linux, but there are many more similar projects, such as: Kubernetes, GIMP, Blender, Docker, and the list could go on. We think Google Summer of Code – and syslog-ng in particular – is a great opportunity to start contributing to an open source project, gain invaluable experience, and get that reference that you can put on your CV. :-) If you are interested, go to the syslog-ng page of GSoC, https://summerofcode.withgoogle.com/organizations/4514364854370304/, where you can find contact information, project ideas, and details about how to apply. If you are a teacher or you know students for any other reasons, make sure that your students know about syslog-ng and that they check our project idea list! Please re-tweet this tweet ( https://twitter.com/PCzanik/status/966608289910919168 ) to reach as many students (and their teachers) as possible! Thank you for your help, Peter Peter Czanik (CzP) <peter.czanik(a)balabit.com> Balabit / syslog-ng upstream https://syslog-ng.com/blog/author/peterczanik/ https://twitter.com/PCzanik

1 0

multiple file sources, worked - some have now gone silent
by Declan White 21 Feb '18

21 Feb '18

I'm looking at a syslog-ng client that has multiple file() sources feeding a single destination unix-domain socket with flags(flow-control) and some disk-buffer. Two days ago it was sending all these files and tracking their rotations fine etc. The destination server restarted, so the socket disappeared for a while, and the client ran out of the default buffering. The intention is that it recovers when the destination returns and resumes reading the files where it left off. When the server restarted data started flowing again. But I've noticed it's no longer sending data from ALL of the file sources. Some have gone silent. The log is saying nothing about the frozen sources. The statistics counters simply stopped incrementing for them. This is kind of fatal. Has anyone seen this before or know how to avoid it? The faulty instance is still running, so if there's any way of interrogating it to find out its mental state, that's still possible. syslog-ng 3.12.1 OS: Solaris -- Declan White

4 7