- syslog-ng - lists.balabit.hu

Destination program input
by Scappatura Rocco 30 Apr '18

30 Apr '18

Hello. I have written a C program that read the successful POP3 access log lines to my POP3 server. After get one line, it get the time and the POP3 account that logged in. These data are then stored in the account MySQL database. The program (named 'pop3_access') works fine: echo "Apr 10 12:00:17 mail1 pop3d: LOGIN, user= xxx(a)domain.tld, ip=[::ffff:xxx.yyy.zzz.vvv], port=[56196]"|./pop3_access Infact quering DB, I get: username pop3_last_access xxx(a)domain.tld 2018-04-10 12:00:17 I have configured syslog: filter f_pop3_access { facility(local7) and match ('LOGIN,'); }; destination d_pop3_last_access { program("/usr/local/bin/pop3_access >/dev/null" ); }; log { source(src); filter(f_pop3_access); destination(d_pop3_last_access); }; But no rows are inserted in the database.. >From log of destination program 'pop3_access' I see the it remains in a while loop waiting for an 'EOF' that never arrives: while (fgets(line, MAXSTR, stdin)) { if (sscanf(line, "%s %d %s %s %s %s %s %s %s", mese, &giorno, ora, s, proto, comando, u, ip, porta) != 9){ syslog (LOG_NOTICE, "Errore scanf"); exit(-1); } } How I have to do so that my program succeed in taking the syslog row correctly, and then returns after process the data as expected? Regards, Rocco Scappatura

3 5

insert timezone into log
by Asif Iqbal 26 Apr '18

26 Apr '18

How do I change the time in the log to append a timezone. Timestamp should be based on generateTime=1523620861 which is in the log So today the time in the shows like this Apr 25 16:46:51 host.example.net .., generateTime=1524674663, ... I like to change it like below 2018-04-25T16:46:55+0000 host.example.net ...,generateTime=1524674663, .... {"PROGRAM":"alarmLog,","PRIORITY":"notice","MESSAGE":"applianceName=KING-MER-50-PRI, tenantName=king, alarmType=vrrp-v3-proto-error, alarmKey=0|vni-0/3.0, generateTime=1523620861, applianceId=1, vsnId=0, tenantId=4, alarmCause=causeOther, alarmClearable=no, alarmClass=new, alarmKind=root, alarmEventType=equipmentAlarm, alarmSeverity=indeterminate, alarmOwner=tenant, alarmSeqNo=36657, alarmText=\"vni-0/3.0\", siteName=","HOST":"host.example.net","FACILITY":"user","DATE":"Apr 13 04:01:22"} -- Asif Iqbal PGP Key: 0xE62693C5 KeyServer: pgp.mit.edu A: Because it messes up the order in which people normally read text. Q: Why is top-posting such a bad thing?

2 4

GSoC 2018 participant: Queue Interface
by Mehul Prajapati 25 Apr '18

25 Apr '18

Hi All, I'm Mehul Prajapati, a prospective student in Masters of Applied Computer Science graduate program at Concordia University. In my spare time, I do competitive programming on HackerRank, online computer science courses (edx, Coursera, Udacity) and open source projects to improve myself as a software developer. I am going to work on the Queue Interface project using C. The aim of this project is to develop a Redis based queue interface which could be an alternative to the disk queue. Many thanks to Mr. Kókai Péter for selecting me. I am excited to be a part of this community. Looking forward to work on this project. -- Regards, Mehul Prajapati

1 0

GSoC 2018 participant : ElasticSearch Destination Client
by Videet Singhai 25 Apr '18

25 Apr '18

Hello all, I am Videet Singhai from India, currently pursuing Bachelor of Engineering in Information Technology. I am going to work on the ElasticSearch Destination REST Client using C. This client should be able to send the logs to ElasticSearch without depending on other languages. Thanks to Mr. Laszlo Budai for selecting me. Looking forward to having an amazing summer. Thank you

1 0

my syslog-ng-githead repo now depends on Java 1.8
by Czanik, Péter 25 Apr '18

25 Apr '18

Hi, Recently I updated my experimental syslog-ng repo built from git head for RHEL/CentOS 7 & Fedora to use openjdk 1.8 instead of 1.7, as it might support Oracle Java packages better (no need to install both openjdk and Oracle java). @Evan: please give it a try, if it fixes your problem: https://copr.fedorainfracloud.org/coprs/czanik/syslog-ng-githead/ @everyone: would you mind, if I update the dependency also for my unofficial packages for the next release? (3.15) As far as I can see, it's not openjdk 1.7 but 1.8 which is installed by default on CentOS 7. Bye, Peter Czanik (CzP) <peter.czanik(a)balabit.com> Balabit / syslog-ng upstream https://syslog-ng.com/blog/author/peterczanik/ https://twitter.com/PCzanik

1 0

Bug report: syslog-ng requests too many capabilities
by Russenberger Dominik 24 Apr '18

24 Apr '18

Hi List, I recently noticed something very strange: although I run syslog-ng as an unprivileged user (with -u log -g log), newly created logfiles were owned by root. syslog-ng shows up running as user log in ps, as expected. The culprit turned out to be linux capabilities. No problem I thought, I just add a "--caps 'CAP_NET_BIND_SERVICE=epi'" command line parameter (binding ports <1024 is the only capability I need). But then I was getting flooded by messages like this: syslog-ng[25657]: Error managing capability set, cap_set_proc returned an error; caps='= cap_net_bind_service+eip cap_fowner+e', error='Operation not permitted (1)' In my opinion, there are 2 bugs in syslog-ng: * if I tell a daemon to run as unprivileged user I do not expect it to write files as user root. What syslog-ng is doing basically is faking being an unprivileged user, while retaining capabilities which are equivalent to full root permissions. Syslog-ng should imho either run as root, with capabilities; OR as unprivileged user without capabilities (except those explicitly given in --caps) * syslog-ng drops to the capabilities it gets told in --caps, but later g_process_cap_modify() ignores what was specified. Some notes about my environment: - the OS is an Ubuntu remix. I'm recompiling syslog-ng 3.11.1 from Ubuntu Artful, with some changed configure flags (see below) - there are multiple syslog-ng instances running. One for the system itself (as root), and multiple other syslogs receiving incoming logs over udp/tcp (running as user log) My syslog-ng version: syslog-ng 3 (3.11.1) Installer-Version: 3.11.1 Revision: Compile-Date: Apr 10 2017 14:06:00 Module-Directory: /usr/lib/syslog-ng/3.11 Module-Path: /usr/lib/syslog-ng/3.11 Available-Modules: disk-buffer,afsocket,afprog,system-source,kvformat,cef,affile,date,cryptofuncs,basicfuncs,confgen,pseudofile,csvparser,afuser,syslogformat,dbparser,linux-kmsg-format Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-IPv6: on Enable-Spoof-Source: off Enable-TCP-Wrapper: off Enable-Linux-Caps: on Enable-Systemd: off Thanks, Dominik

4 4

Need clarification on different behavior of forwarding rules in syslog-ng
by Kavita Mohite (kmohite) 18 Apr '18

18 Apr '18

Hello Support Team, In the syslog-ng configuration, we have 2 forward rules as follows: 1. One rule reads from source (port 514) and writes syslogs to the file for all the syslogs greater than debug. 2. Second rule reads from source (port 514) and forwards the the syslog to the destination if a set of filter rules match. These set of filter rules are called whitelist filter rules are filter rules containing match filter and priority filter in each rule. As there are multiple filter rules, they are ORed to create a main filter rule. Now we have syslogs coming in at a rate of 400 eps. Following is our observation: A. If the syslog matches the filter in point 2 above, then following happens · It is forwarded to the destination provided in point 2. · Also at the same time it is written to a file as per rule in point 1. B. If the syslog does not match the filter in point 2 above, then following happens · It is NOT forwarded to the destination provided in point 2 as there is no match. · It is suppose to still write to file as the rule in 1 matches. But this does not happen. We are seeing that if the filter in point 2 matches then both the forward rules get executed immediately and we see the log written to file also immediately. But in case of B, if the syslog does not match we are seeing that the write to file is happening g in bulk for 30 minutes. Why is this ? Why does it not write to file immediately in case of (B) but does it in case of A ? Below is the snap of the config values we are using in the syslog-ng.conf options { flush_lines (100); log_fetch_limit(1000); log_iw_size(10000); time_reopen (10); log_fifo_size (10000); chain_hostnames (off); use_dns (no); use_fqdn (no); create_dirs (yes); keep_hostname (yes); threaded(yes); }; Thanks Kavita

2 1

Need clarification on different behavior of forwarding rules in syslog-ng
by Kavita Salve 18 Apr '18

18 Apr '18

Hello Support Team, In the syslog-ng configuration, we have 2 forward rules as follows: 1. One rule reads from source (port 514) and writes syslogs to the file for all the syslogs greater than debug. 2. Second rule reads from source (port 514) and forwards the the syslog to the destination if a set of filter rules match. These set of filter rules are called whitelist filter rules are filter rules containing match filter and priority filter in each rule. As there are multiple filter rules, they are ORed to create a main filter rule. Now we have syslogs coming in at a rate of 400 eps. Following is our observation: A. If the syslog matches the filter in point 2 above, then following happens - It is forwarded to the destination provided in point 2. - Also at the same time it is written to a file as per rule in point 1. B. If the syslog does not match the filter in point 2 above, then following happens - It is NOT forwarded to the destination provided in point 2 as there is no match. - It is suppose to still write to file as the rule in 1 matches. *But this does not happen.* We are seeing that if the filter in point 2 matches then both the forward rules get executed immediately and we see the log written to file also immediately. But in case of B, if the syslog does not match we are seeing that the write to file is happening g in bulk for 30 minutes. *Why is this ? Why does it not write to file immediately in case of (B) but does it in case of A ?* Below is the snap of the config values we are using in the syslog-ng.conf options { flush_lines (100); log_fetch_limit(1000); log_iw_size(10000); time_reopen (10); log_fifo_size (10000); chain_hostnames (off); use_dns (no); use_fqdn (no); create_dirs (yes); keep_hostname (yes); threaded(yes); }; Thanks Kavita Mohite

1 1

Re: [syslog-ng] Support for Open Source Syslog-ng
by Naveen Bhalla (nbhalla) 14 Apr '18

14 Apr '18

Team, Could you pls help us on below P1 situation for us? Regards, [http://wwwin.cisco.com/c/dam/cec/organizations/gmcc/services-tools/signatur…] Naveen Bhalla | Manager.Technical Support CMS Platform Operations Cell: +91-9880362157 Desk: +91-80-44260795 From: Naveen Bhalla (nbhalla) Sent: 13 April 2018 09:14 PM To: 'support(a)balabit.com' <support(a)balabit.com<mailto:support@balabit.com>> Subject: Support for Open Source Syslog-ng Hello Support Team, We have a situation in our platform where syslog-ng is dropping part of syslog traffic coming into our server. The syslog-ng has around 750 match rules in its configuration. Based on these rules the syslogs are forwarded to the destinations. Also, there is one rule to write all the received syslogs to disk. We are receiving syslogs at the rate of 300 eps. The issue is that we are seeing that syslog-ng is not able to process the syslogs and forward them to the destinations. It is not writing to the disk also. We are seeing that there is a big delay after which some syslogs are getting written to the disk. We are seeing loss of UDP packets. The UDP buffer size is big enough. net.ipv4.tcp_rmem = 4096 4194304 16777216 net.ipv4.tcp_wmem = 98304 4194304 16777216 net.core.rmem_default = 234217728 net.core.wmem_default = 234217728 net.core.rmem_max = 234217728 net.core.wmem_max = 234217728 net.ipv4.tcp_window_scaling = 1 net.ipv4.ip_local_port_range = 32768 61000 fs.file-max = 2097152 net.core.optmem_max = 40960 net.core.netdev_max_backlog = 50000 net.ipv4.udp_rmem_min = 8192 net.ipv4.udp_wmem_min = 8192 We need help to resolve this issue. We are using open-source syslog-ng in our setup. Regards, [http://wwwin.cisco.com/c/dam/cec/organizations/gmcc/services-tools/signatur…] Naveen Bhalla | Manager.Technical Support CMS Platform Operations Cell: +91-9880362157 Desk: +91-80-44260795

3 2

Insider 2018-04: RHEL6; Windows Event Log; Patched RPM; HEC; Conferences;
by Czanik, Péter 12 Apr '18

12 Apr '18

Dear syslog-ng users, This is the 66th issue of syslog-ng Insider, a monthly newsletter that brings you syslog-ng-related news. NEWS syslog-ng is available for RHEL 6 / CentOS 6 again -------------------------------------------------- After more than a year, the latest version of syslog-ng compiles again on Red Hat Enterprise Linux version 6. This is thanks to a patch from Balázs Scheidler which resolves compatibility problems with older glib releases by implementing the missing functions within syslog-ng. https://syslog-ng.com/blog/latest-syslog-ng-available-rhel-6-centos-6/ Agentless Windows logging ------------------------- With syslog-ng Premium Edition, you can collect Windows event logs without installing any third-party application on your Windows-based computer. Learn more about it from our blog at https://syslog-ng.com/blog/how-to-collect-windows-event-logs-with-syslog-ng… or join our webinar where you can also ask questions: https://www.brighttalk.com/webcast/16207/314789?utm_source=OSE/ Build your own syslog-ng RPM ---------------------------- When a user reports a problem with syslog-ng, developers create code to fix it. The fix is called a patch or a pull request (PR) in Git terminology. Often this code does not enter the main source code automatically but developers ask the user to test the fix before the code is merged. A new challenge starts here: figuring out how to test a proposed fix. This blog shows you how to download and patch the syslog-ng source code and build easy-to-install (and remove) RPM packages from it. https://syslog-ng.com/blog/build-syslog-ng-rpm-patched-git-sources-rhel-cen… While this blog introduces you to package building in depth, a lot simpler method is also available for Ubuntu packages and should be working soon again for RHEL / CentOS as well: https://github.com/balabit/syslog-ng/tree/master/dbld Sending logs to Splunk HEC -------------------------- Recently Splunk started to recommend the use of the HTTP Event Collector (HEC) instead of forwarders. syslog-ng supports this in multiple ways. In a previous blog, we showed you how to use the http() destination of syslog-ng. This time, we introduce you to another possibility: using an external python script to send logs to HEC. Read our blog at https://syslog-ng.com/blog/splunk-hec-sending-logs-using-program-destinatio… or watch the webinar at https://www.brighttalk.com/webcast/16207/308061 Conferences: CentOS Dojo, FOSDEM and SCALE ------------------------------------------ Recently syslog-ng was presented at some popular open source conferences. Read about the presentations and experiences in related blogs: CentOS Dojo & FOSDEM: https://syslog-ng.com/blog/centos-dojo-fosdem-2018/ SCALE: https://syslog-ng.com/blog/syslog-ng-scale-2018/ UPCOMING EVENTS LOADays: http://loadays.org/ (Antwerp) Red Hat Summit: https://www.redhat.com/en/summit/2018 (San Francisco) Your feedback and news, or tips about the next issue are welcome at documentation(a)balabit.com. To read this newsletter online, visit: https://syslog-ng.com/blog/ Peter Czanik (CzP) <peter.czanik(a)balabit.com> Balabit / syslog-ng upstream https://syslog-ng.com/blog/author/peterczanik/ https://twitter.com/PCzanik

1 0