- syslog-ng - lists.balabit.hu

patterndb and smtp to() issue
by Alexandre DEPREZ 21 Jan '16

21 Jan '16

Hi all, I'm using the pattern-db to extract values from a firewall's log. This far, everything's working great. The log looks something like this: Jan 20 2016 21:48:45: %ASA-7-746012: user-identity: Add IP-User mapping 10.10.99.7 - LOCAL\alex Succeeded - VPN user Using pdbtool and matching the log against the xml pattern file, this is showing me good results : :$pdbtool match -P "%ASA-7-746012" -M "user-identity: Add IP-User mapping 10.10.99.7 - LOCAL\alex Succeeded - VPN user" -p /etc/syslog-ng/patterndb.d/vpn-parser-up.xml -D -c Pattern matching part: user-identity: Add IP-User mapping @IPv4:VPN_IP=10.10.99.7@ - LOCAL\ @STRING:VPN_USER=alex@ Succeeded - VPN user Matching part: user-identity: Add IP-User mapping 10.10.99.7 - LOCAL\alex Succeeded - VPN user Values: MESSAGE=user-identity: Add IP-User mapping 10.10.99.7 - LOCAL\alex Succeeded - VPN user PROGRAM=%ASA-7-746012 .classifier.class=vpn.access_log .classifier.rule_id=019045a7383c252e57c20435ae5bf86c VPN_IP=10.10.99.7 VPN_USER=alex TAGS= Here's the xml file <patterndb version='4' pub_date='2015-12-22'> <ruleset id='04ba26e756011614c57cf469fed7b5c0' name='%ASA-7-746012'> <pattern>%ASA-7-746012</pattern> <rules> <rule class='vpn.access_log' id='019045a7383c252e57c20435ae5bf86c' provider='alex'> <patterns> <pattern>user-identity: Add IP-User mapping @IPv4:VPN_IP@ - LOCAL\@STRING:VPN_USER@ Succeeded - VPN user</pattern> </patterns> </rule> </rules> </ruleset> </patterndb> Now, the problem lies on the destination which is using the smtp driver. destination vpn_mail_up { smtp( host("x.x.x.x") port(25) from("alex(a)x.y" "alex(a)x.y") to("${VPN_USER}(a)x.y") subject("vpn connection") body("vpn connection from ${VPN_USER} with IP: ${VPN_IP}\n") ); }; The variable is functional inside the body() but not in the to() field. Here's a dump I extracted directly from the server on the tcp session to the mail server: RCPT.TO:<${VPN_USER}(a)x.y>.. BDAT.411.. X-Mailer:.syslog-ng.3.5.6.. Date:.Wed,.20.Jan.2016.21:51:51.+0100.. From:.alex@x.y.. Message-Id:.<1453323111.149975.19608@debian>.. To:."${VPN_USER}(a)x.y".<${VPN_USER}(a)x.y>.. Subject:.vpn.connection.. .BDAT.68..vpn.connection.from.alex..with.IP:.10.10.99.7.BDAT.2.LAST.. The variable is being populated in the body message but not the recipient. Is there any chance the variable could only be used once (!?) or not being able to be used inside the to() ? Regards, Alex

2 1

Syslog-ng mutual self cert authentication
by Ivan Adji - Krstev 18 Jan '16

18 Jan '16

Hi all, I'm truing to make mutual self cert authentication with Syslog-NG and using LogAnalyzer to see the logs. I have done the following: *Server configuration:* Creating a CA making directory and put the openssl.cnf file in the directory etc. do the following command: 1. openssl req -new -x509 -keyout private/cakey.pem -out cacert.pem -days 365 -config openssl.cnf 2. openssl req -nodes -new -x509 -keyout serverkey.pem -out serverreq.pem -days 365 -config openssl.cnf 3. openssl x509 -x509toreq -in serverreq.pem -signkey serverkey.pem -out tmp.pem 4. openssl ca -config openssl.cnf -policy policy_anything -out servercert.pem -infiles tmp.pem 5. In */etc/syslog-ng/* create directory *cert.d* and *ca.d* copy the *serverkey.pem* and *servercert.pem* to *cert.d* and copy *cacert.pem* to *ca.d*. Go to *ca.d* and issue the following command *"openssl x509 -noout -hash -in cacert.pem"* and do the link *"ln -s cacert.pem 6d2962a8.0" *6. On the syslog-ng.conf file i put the following changes: *************************************** source s_sys { system(); internal(); network( port(6514) file("/proc/kmsg" log_prefix("kernel: ")); transport("tls") tls( key_file("/etc/syslog-ng/cert.d/serverkey.pem") cert_file("/etc/syslog-ng/cert.d/servercert.pem") ca_dir("/etc/syslog-ng/ca.d")) ); }; ******************************************** And restart the service ================== *Client Configuration* Creating CA making directory and put the openssl.cnf file in the directory etc. do the following command 1. openssl req -new -x509 -keyout private/cakey.pem -out cacert.pem -days 365 -config openssl.cnf 2. openssl req -nodes -new -x509 -keyout clientkey.pem -out clientreq.pem -days 365 -config openssl.cnf 3. openssl x509 -x509toreq -in clientreq.pem -signkey clientkey.pem -out tmp.pem 4. openssl ca -config openssl.cnf -policy policy_anything -out clientcert.pem -infiles tmp.pem 5. In */etc/syslog-ng/* create directory *cert.d* and *ca.d* copy the *"clientkey.pem" *and *"clientcert.pem" *to *"cert.d" *and copy *"cacert.pem" *to *"ca.d" *and issue the following command in the *ca.d* directory. *openssl x509 -noout -hash -in cacert.pem *and *ln -s cacert.pem 6d2962a8.0 *6. add the following to the *syslog-ng.conf* file ****************************************************** destination d_tcp { tcp("192.168.111.176" port(6514) tls( ca_dir("/etc/syslog-ng/ca.d") key_file("/etc/syslog-ng/cert.d/clientkey.pem") cert_file("/etc/syslog-ng/cert.d/clientcert.pem") ) ); }; *************************************************************** And finally i have the following errors :) Jan 12 11:58:02 syslogserver syslog-ng[5507]: Syslog connection accepted; fd='16', client='AF_INET(192.168.111.104:45631)', local='AF_INET(0.0.0.0:6514)' Jan 12 11:58:02 syslogserver syslog-ng[5507]: SSL error while reading stream; tls_error='SSL routines:SSL3_READ_BYTES:tlsv1 alert unknown ca' Jan 12 11:58:02 syslogserver syslog-ng[5507]: I/O error occurred while reading; fd='16', error='Connection reset by peer (104)' Jan 12 11:58:02 syslogserver syslog-ng[5507]: Syslog connection closed; fd='16', client='AF_INET(192.168.111.104:45631)', local='AF_INET(0.0.0.0:6514)' Jan 12 11:58:12 syslogserver syslog-ng[5507]: Syslog connection accepted; fd='16', client='AF_INET(192.168.111.104:58937)', local='AF_INET(0.0.0.0:6514)' Jan 12 11:58:12 syslogserver syslog-ng[5507]: SSL error while reading stream; tls_error='SSL routines:SSL3_READ_BYTES:tlsv1 alert unknown ca' Jan 12 11:58:12 syslogserver syslog-ng[5507]: I/O error occurred while reading; fd='16', error='Connection reset by peer (104)' Jan 12 11:58:12 syslogserver syslog-ng[5507]: Syslog connection closed; fd='16', client='AF_INET(192.168.111.104:58937)', local='AF_INET(0.0.0.0:6514)' Can someone help me with this and tell me what im doing wrong ? Thanks in advanced !

2 8

syslog-ng @ Southern California Linux Expo
by Czanik, Péter 18 Jan '16

18 Jan '16

Hi, This week I'll be at the Southern California Linux Expo in Los Angeles. If you want to talk or hear about syslog-ng, you have many chances: - Balabit booth #426 during Expo hours - syslog-ng BoF in room 104 at 7:00pm on Saturday - "syslog-ng: from raw data to Big Data" presentation in room 211, Sunday 11:30 You can also request an appointment at https://www.balabit.com/scale-trade-show-meeting and I'm happy to meet you if you are in LA also outside of Scale. Bye, Peter Czanik (CzP) <peter.czanik(a)balabit.com> Balabit / syslog-ng upstream http://czanik.blogs.balabit.com/ https://twitter.com/PCzanik

1 0

Issue with syslog-ng and Windows Server 2012
by Artyom Gavrichenkov 15 Jan '16

15 Jan '16

Hi all, Since Windows Vista (and Server 2012), in NTFS, file size is updated not when its contents are being flushed to disk, but when the last handle to the corresponding file object is closed. More on that here: https://blogs.msdn.microsoft.com/oldnewthing/20111226-00/?p=8813 What this means in practice is that an application log file might already be quite large, but its reported file size will still be zero unless some other application will open(), read() and then close() it. And seems like it breaks syslog-ng’s source() file() handler in case the filename in file() arguments contains wildcards. Could anyone please look into it? | Artyom Gavrichenkov | gpg: fa1c670e | mailto: ximaera(a)gmail.com | xmpp: ximaera(a)gmail.com | tel. no: +7 916 515 49 58

3 2

syslog-ng ose push to master
by Scheidler, Balázs 14 Jan '16

14 Jan '16

Hi, there were some fallouts from the recent merger of #842, which I've quickly fixed up by pushing two trivial patches to master bypassing review on github. My patches compile without warnings and make check is passed, but please review them. Cheers, -- Bazsi

2 1

Insider 2016-01: syslog-ng web GUIs; Scale; GSoC; Raspberry Pi 2; SSB 4 F3
by Czanik, Péter 14 Jan '16

14 Jan '16

Dear syslog-ng users, This is the 46th issue of the syslog-ng Insider, a monthly newsletter that brings you syslog-ng-related news. NEWS syslog-ng web GUIs ------------------ The number and importance of logging solutions with a web interface is growing, luckily many of these are syslog-ng based. Using the Elasticsearch destination, Kibana provides an easy-to-use web interface for log messages. Read more about feeding log messages to Elasticsearch with syslog-ng with at https://czanik.blogs.balabit.com/2015/12/web-interfaces-for-your-syslog-ser… Southern California Linux Expo ------------------------------ Next week, Peter Czanik will present “syslog-ng: from raw data to Big Data” at the Southern California Linux Expo. There will be also a booth (#426) where you can talk about syslog-ng, and we also plan a syslog-ng BoF where you can share your experiences or ask your questions about syslog-ng. https://www.balabit.com/scale-trade-show-meeting Google Summer of Code --------------------- In the past four years syslog-ng participated in Google Summer of Code, where talented students spent their summers coding syslog-ng instead doing some less intellectually challenging summer jobs. Some of their results include the Redis destination or configuration management modules. Read more about our experiences and the upcoming “syslog-ng Universe” at https://syslog-ng.org/gsoc-2015-summary/ syslog-ng on the Raspberry Pi 2 ------------------------------- The performance of syslog-ng on the original Raspberry Pi was not that great. A quick test of the Raspberry Pi 2 revealed, that performance increased drastically, now you can use this $35 machine as a central log server: https://czanik.blogs.balabit.com/2016/01/syslog-ng-on-the-raspberry-pi-2/ syslog-ng Store Box 4F3 released -------------------------------- The latest version of syslog-ng Store Box (SSB), a turnkey appliance built around syslog-ng has been released. You can now search across multiple logspaces and across multiple SSB boxes from one search interface, while maintaining user profile-based access control. Read the full announcement at https://jluby.blogs.balabit.com/2015/12/15/syslog-ng-store-box-4-f3-release… Monitoring serial consoles -------------------------- The January issue of the Linux Journal contains an article from Fabien Wernli about using syslog-ng to monitor serial consoles. This month it is only available for subscribers: http://www.linuxjournal.com/content/january-2016-issue-linux-journal NEW RELEASES SSB 4 F3: https://jluby.blogs.balabit.com/2015/12/15/syslog-ng-store-box-4-f3-release… Your feedback and news tips about the next issue is welcome at documentation(a)balabit.com . To read this newsletter on-line, visit: http://insider.blogs.balabit.com/ Peter Czanik (CzP) <peter.czanik(a)balabit.com> Balabit / syslog-ng upstream http://czanik.blogs.balabit.com/ https://twitter.com/PCzanik

1 0

Changing a value after a match with patterndb
by Mark Shetka 12 Jan '16

12 Jan '16

Is it possible to substitute a value after a pattern match? For example, in the pattern below if I match the "23" then use value "telnet", but if it is a "22", then "ssh". %SEC_LOGIN-4-LOGIN_FAILED: Login failed [user: test] [Source: 131.212.1.1] [localport: 23] [Reason: Login Authentication Failed] at 10:38:41 CST Thu Jan 23 2014 I am trying to use one pattern to match both cases and we'd prefer to have service names rather than port numbers in our db. -- Mark Shetka Infrastructure Analyst - Network Team Information Technology Systems & Services University of Minnesota - Duluth (218) 726-7682

4 6

how to enable TLS on syslog-ng 3.2.5
by Jim Zapsalis 06 Jan '16

06 Jan '16

Hi guys, How do we enable TLS on OSE version of syslog-ng version 3.2.5? Or is it simpler to enable SSL ? Regards, Jim Zapsalis Senior Platform Engineer, IS Bupa, 33 Exhibition Street, Melbourne, 3000 T +61 3 9937 4028 M 0411 779 710 E jim.zapsalis(a)bupa.com.au<mailto:jim.zapsalis@bupa.com.au> W bupa.com.au/healthandcaring<http://www.bupa.com.au/healthandcaring/index.html> [Social Media Icons-01.png]<https://www.facebook.com/BupaAustralia>[Social Media Icons-02.png]<https://mobile.twitter.com/bupaaustralia>[Social Media Icons-04.png]<https://ea.linkedin.com/company/bupa-australia>[Social Media Icons-03.png]<https://www.youtube.com/user/BupaAustralia> Bupa Australia email disclaimer: The information contained in this email and any attachments is confidential and may be subject to copyright or other intellectual property protection. If you are not the intended recipient, you are not authorized to use or disclose this information, and we request that you notify us by reply mail or telephone and delete the original message from your mail system.

3 3

Proper formatting of incoming Log4j, Jetty events
by Clayton Dukes 16 Dec '15

16 Dec '15

Hi, I'm having trouble with getting "good" fields from Jetty. I found this: http://blog.trifork.com/2010/01/14/logging-to-the-syslog-from-a-java-applic… And had my user configure their log4j with the given recommendation, but the events still come in quite poorly. Our syslog-ng template uses the following template: @${R_UNIXTIME:--}.${R_USEC}\t${HOST:--}\t${PRI:--}\t${PROGRAM:--}\t${MSGONLY}\n"); Some sample events: @1450275007.912996 host001 173 2015-12-16 16:10:07,836 DEBUG [HeartbeatGeneratorTimer] <-- sessionKey=179: out>{"method":"Heartbeat","jsonrpc":"2.0"} As you can see - this event is sending the date as the program name. Another: @1450275007.924140 host002 13 521 <174>1 2015-12-16T16:10:07+02:00 host002 /openam 1000430466 AUTHENTICATION-303 [logRecord@36733 LoginID="id=2565a24-0829-11e2-b614-001e371e7e40,ou=user,dc=opensso,dc=java,dc=net" ContextID="b1de62b91530d5303" IPAddr="10.3.109.13" LogLevel="INFO" Domain="dc=opensso,dc=java,dc=net" ModuleName="ProfileMapper|Mandate|LDAP" HostName="10.3.109.13" LoggedBy="cn=dsameuser,ou=DSAME Users,dc=opensso,dc=java,dc=net" MessageID="AUTHENTICATION-303" NameID="" TIME="2015-12-16 16:10:07"] Logout|service|mailService This one is sending the program name as an integer (521) There are many others, the program names consist of the following (out of about 100k events): - " .0.9"},"parentID" ...80) ame" at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:275) del" ...e.common.AuthenticationFilter.doFilter(AuthenticationFilter.java ...ee.elion.smarthome.common.AuthenticationFilter.doFilter(AuthenticationFilter.java ...eJspFile(JspServlet.java EST in ,"is_recording" iWeather"},{"id" leRecordID" me" model" ng" ngth" ...rg.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java se,"is_recording" ....servlet.JspServlet.serviceJspFile(JspServlet.java tin Does anyone have a good way to fix this? ______________________________________________________________ Clayton Dukes ______________________________________________________________

2 1

rabbitmq errors
by Al Itchon 15 Dec '15

15 Dec '15

I'm seeing an error in my log indicating that messages aren't being sent to a queue in RabbitMQ. This happens a few times a week. Here's the error: Nov 24 13:29:28 syslog09 syslog-ng[7883]: Network error while inserting into AMQP server; time_reopen='10' I have about 1000 servers sending tcp messages through a call to /dev/tcp to port on the syslog-ng server. It seems messages aren't sent to the queue for about 1 or 2 minutes, and then it resumes sending again without error. Could this be a network issue or something within syslog-ng that's limiting the messages?

4 8