- syslog-ng - lists.balabit.hu

Issue about using perl script in Receiving messages from external applications
by Jian Zhu 10 Aug '12

10 Aug '12

Hello, Could anyone please help me on this below issue? my syslog-ng.conf file look like below: The test.sh script work fine, I can see it's output in the test.sh.log file. But the test.pl doesn't work. the test.sh and test.pl have the same function of printing something to STDOUT. ------------------------------ ------------------------------------------------------------------------------------------------------------------- @version: 3.0 #Default configuration file for syslog-ng. # # For a description of syslog-ng configuration file directives, please read # the syslog-ng Administrator’s guide at: # # http://www.balabit.com/dl/html/syslog-ng-admin-guide_en.html/bk01-toc.html # options { # Number of syslog lines stored in memory before being written to files flush_lines (0); log_fifo_size (2048); create_dirs (yes); perm (0640); dir_perm (0750); }; source ppp { program("/root/perl/test.pl"); }; destination ppp { file ("/var/log/syslog-ng/$HOST/perl.pl.log" create_dirs(yes)); }; log { source(ppp); destination(ppp); }; source sss { program("/root/shell/test.sh"); }; destination sss { file ("/var/log/syslog-ng/$HOST/test.sh.log" create_dirs(yes)); }; log { source(sss); destination(sss); }; ------------------------------------------------------------------------------------------------------------------------------------------------- -- *Thanks & Best Regards* *James Zhu* *Mobile**:* *+86 18**6 8482 2818* *Email: *zhujian0805(a)gmail.com

1 0

Query On configuring Centralized Audit server with Auditd daemon
by Koresh... 09 Aug '12

09 Aug '12

Hi Folks, Need your help ! Want to configure a centralized Audit server (Currently the centralized server is running Octopussy Web interface, which receives logs from remote hosts by Rsyslog ). The challenge and confusion here is .. all my linux clients are configured with syslog-ng and the daemon is sending all the system logs and kernel logs like messages,secure,cron logs etc ... with out any trouble. The problem is the syslog-ng daemon is not able to send the auidtd logs (/var/log/audit.log) to the Rsyslog server, Hence request your help to guide me how to setup the syslog-ng to forward the audit.log to the remote Rsyslog server. It would be great if i can get client side and server side configuration guidelines. -- Thanks in Advance - Koresh

3 4

syslog-ng service not starting
by James McDonald 08 Aug '12

08 Aug '12

I returned from being out of the office for week and realized something was "off" with syslog-ng. I was unable to restart the service. I did a 'service syslog-ng status' and it says it's stopped. When I try 'service syslog-ng start' I get the following error: Error binding socket; addr='AF_INET(0.0.0.0:443)', error='Address already in use (98)' Error initializing source driver; source='inputs', id='inputs#3' Error initializing message pipeline; There is only instance of syslog-ng installed on the server but I did a reboot to try and clear any conflicting settings, but I still get the same error. Can someone help me clear this error and get the service started again? Thanks, James McDonald System Administrator RHCSA, Security+ jmcdonald(a)lce.com

5 8

query about dropped messages...
by Vijendra Agarwal 06 Aug '12

06 Aug '12

Hi, I am using syslog-ng 3.1.4 open source edition. my setup details are.. I have Log application running on MC1 with syslog-ng configured as client on it. My log application is communicating witg syslog-ng on unix-stream socket. I have machine M2 where my syslog-ng server is running. My syslog-ng client is coomunicating with syslog-ng server on TCP stream. It writes all the log messages received from my log application to the /var/log/messages file. My log application sends bulk of message around 600000 p/s , I have noticed from the syslog-statistics that the syslog-ng client dropped some log messages. I have a query here Can I store the dopped messages to some file by doing some configration setting? Or Is there a configyration setting by which I can avoide the dropping of messages by syslog-ng client? Kindly help me out to resolve the issue. Thanks Vijendra. -- Thanks Vij M No: 9960771025.

1 0

Syslog-ng OSE 3.3.5 - High CPU usage
by satish pattar 31 Jul '12

31 Jul '12

Hi All, I am using syslog-ng OSE & seeing a high CPU usage ? Could you please point out if this could be a config/rules issue or syslog-ng internal issue. CPU usage in sorted order : 0.0 11.9 16.6 22.2 25.0 25.0 25.0 25.0 33.3 33.3 50.0 ------------------------------------------ # syslog-ng --version syslog-ng 3.3.5 Installer-Version: 3.3.5 Revision: Compile-Date: Apr 20 2012 04:47:44 Default-Modules: afsocket,affile,afprog,afuser,basicfuncs,syslogformat,dbparser Available-Modules: syslogformat,dbparser,afuser,basicfuncs,afmongodb,convertfuncs,afprog,dummy,afsocket,confgen,affile,csvparser Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-IPv6: on Enable-Spoof-Source: off Enable-TCP-Wrapper: off Enable-Linux-Caps: off Enable-Pcre: off ----------- Here is the code snippet to send syslog messages to syslogger for(i=0; i<500;i++) { // this is the message format we are following syslog(LOG_INFO, "%s, %s, %d, %s, %s, %s %d, %s", "00:00:18","XYZ",getpid(),"Info","xyz.c", "Func1", 100, "WelCome to Syslog" ); } --------------------- # cat /proc/cpuinfo Processor : ARMv7 Processor rev 10 (v7l) processor : 0 BogoMIPS : 1988.29 processor : 1 BogoMIPS : 1988.29 Features : swp half thumb fastmult vfp edsp neon vfpv3 CPU implementer : 0x41 CPU architecture: 7 CPU variant : 0x2 CPU part : 0xc09 CPU revision : 10 ------------------------------ syslog-ng-confg: @version: 3.3 ###### # options options { # disable the chained hostname format in logs # (default is enabled) chain_hostnames(0); # the time to wait before a died connection is re-established # (default is 60) time_reopen(10); # the time to wait before an idle destination file is closed # (default is 60) time_reap(360); # the number of lines buffered before written to file # you might want to increase this if your disk isn't catching with # all the log messages you get or if you want less disk activity flush_lines(5); flush_timeout(500); # the number of lines fitting in the output queue log_fifo_size(10000); # enable or disable directory creation for destination files create_dirs(yes); # default owner, group, and permissions for log files # (defaults are 0, 0, 0600) #owner(root); group(root); perm(0640); frac_digits(4); # Override the global timestamp format #(set in the global ts_format() parameter) for the specific destination. ts_format(iso); # default owner, group, and permissions for created directories # (defaults are 0, 0, 0700) #dir_owner(root); dir_perm(0755); # enable or disable DNS usage # syslog-ng blocks on DNS queries, so enabling DNS may lead to # a Denial of Service attack # (default is yes) use_dns(no); # maximum length of message in bytes # this is only limited by the program listening on the /dev/log Unix # socket, glibc can handle arbitrary length log messages, but -- for # example -- syslogd accepts only 1024 bytes # (default is 2048) #log_msg_size(2048); # print statistic information only once every hour (it is not really needed) stats_freq(3600); }; ###### # sources # all known message sources source s_all { # message generated by Syslog-NG internal(); # standard Linux log source (this is the default place for the syslog() # function to send logs to) unix-dgram("/dev/log" program_override("mylogger: ")); # messages from the kernel file("/proc/kmsg" program_override("kernel: ")); # use the following line if you want to receive remote UDP logging messages # (this is equivalent to the "-r" syslogd flag) # udp(); }; ######## #templates #template t_default { template("${ISODATE} ${HOST} ${MSGHDR}${MSG}\n"); template_escape(no); }; #template t_logtty { template("${MSG}\n"); template_escape(no); }; template t_restmsg_notempty { template("${PROGRAM} ${date} ${cname}[${processId}] ${type} (${source}:${line}) ${Restmsg}\n"); }; template t_restmsg_empty { template("${PROGRAM} ${FACILITY} ${LEVEL} ${MSG}\n"); }; ###### # destinations destination d_log_notempty { file("/var/my_log" template(t_restmsg_notempty)); }; #destination d_log_notempty { file("/dev/console" template(t_restmsg_notempty)); }; destination d_log_empty { file("/var/my_log" template(t_restmsg_empty)); }; destination authlog { file("/var/log/auth.log"); }; destination debug { file("/var/log/debug"); }; destination messages { file("/var/log/messages"); }; destination _syslog { file("/var/log/syslog"); }; destination cron { file("/var/log/cron.log"); }; destination daemon { file("/var/log/daemon.log"); }; destination lpr { file("/var/log/lpr.log"); }; destination user { file("/var/log/user.log"); }; destination uucp { file("/var/log/uucp.log"); }; destination mail { file("/var/log/mail.log"); }; destination kern { file("/var/log/kern.log"); }; destination mailinfo { file("/var/log/mail.info"); }; destination mailwarn { file("/var/log/mail.warn"); }; destination mailerr { file("/var/log/mail.err"); }; destination newscrit { file("/var/log/news/news.crit"); }; destination newserr { file("/var/log/news/news.err"); }; destination newsnotice { file("/var/log/news/news.notice"); }; ###### # filters # all messages from the auth and authpriv facilities filter f_restmsg_notempty {match(".+" value("Restmsg"));}; filter f_restmsg_empty {not match(".+" value("Restmsg"));}; # all messages except from the auth and authpriv facilities filter f_auth { facility(auth); }; filter f_authpriv { not facility(auth, authpriv); }; #filter f_debug { level(notice) and not facility(auth, authpriv, news, mail); }; filter f_emergency { level(emerg); }; filter f_info { level(info); }; filter f_notice { level(notice); }; filter f_warn { level(warn); }; filter f_crit { level(crit); }; filter f_err { level(err); }; #filter f_syslog { not facility(authpriv, mail ); }; filter f_syslog { program(syslog-ng); }; filter f_cron { facility(cron); }; filter f_daemon { facility(daemon); }; filter f_kern { facility(kern); }; filter f_mail { facility(mail); }; filter f_user { facility(user); and level(error) }; filter f_uucp { facility(uucp); }; filter f_news { facility(news); }; filter f_messages { level(info..warn) and not facility(auth, authpriv, mail, news); }; filter f_all_info { not match("Info", value("type"), type("string")); }; filter f_all_generic{ not match("Generic", value("cname"), type("string")); }; parser p_db {db_parser( file("/etc/syslog-ng/rules/rules.xml") );}; log {source(s_all);parser(p_db);filter(f_restmsg_notempty);destination(d_log_notempty); }; #log { source(s_all); filter(f_authpriv); destination(authlog); }; log { source(s_all); filter(f_syslog); destination(_syslog); }; log { source(s_all); filter(f_cron); destination(cron); }; log { source(s_all); filter(f_daemon); destination(daemon); }; log { source(s_all); filter(f_kern); destination(kern); }; log { source(s_all); filter(f_mail); destination(mail); }; log { source(s_all); filter(f_user); destination(user); }; #log { source(s_all); filter(f_debug); destination(debug); }; ------------------------------------------------------------------------------------- rules.xml: <patterndb version='4' pub_date='2012-02-17'> <ruleset name='root' id='123456678'> <pattern>mylogger</pattern> <rules> <rule provider='test' id='182437592347598' class='system'> <patterns> <pattern>@ESTRING:date:,@ @ESTRING:cname:,@ @ESTRING:processId:,@ @ESTRING:type:,@ @ESTRING:source:,@ @ESTRING:line:,@ @ANYSTRING:Restmsg: @</pattern> </patterns> </rule> </rules> </ruleset> </patterndb> --------------------------------------------------------------------------------------------- Thanks, pattarsatish

2 1

Installation Error
by kibirango moses 28 Jul '12

28 Jul '12

Hullo users, I am installing syslog-ng ver.3.3.5 on slackware linux 13.1 and i am getting the errors below, when i do a make usr/lib64/gcc/x86_64-slackware-linux/4.4.4/../../../../x86_64-slackware-linux/bin/ld: /usr/local/lib64/libssl.a(s23_srvr.o): relocation R_X86_64_32 against `.rodata' can not be used when making a shared object; recompile with -fPIC /usr/local/lib64/libssl.a: could not read symbols: Bad value Could somebody helpme with a fix to the problem thanx in advnace

2 1

Logging To MongoDB and specifying data types
by Michael Dillon 27 Jul '12

27 Jul '12

I'm using the syslog-ng windows client to forward IIS logs to syslog-ng. I'm using the CSV parser to split it into pairs and pick and choose my pairs. I'm not able to figure out how to insert a field from these pairs (or any others) into MongoDB as a date time field. They are being stored as strings. Also I'd like to store quite a few of the other columns/elements as integers. Obviously I could run a job to do conversions after they are inserted into the database I'd just hate to setup a rube golberg machine if I don't have to. -- Michael Dillon

2 1

Able to connect but not logging
by kibirango moses 27 Jul '12

27 Jul '12

Hullo Users; I am using syslog-ng 3.3.4 and my client is able to connect but not logging to the central logging server. my syslog-ng.conf on server side is @version: 3.3 # Default syslog-ng configuration for Slackware # For info about the format of this file, see "man syslog-ng.conf" # Converted from sysklogd (syslog.conf) by mario(a)slackverse.org options { flush_lines (0); time_reopen (10); time_reap(360); # log_fifo_size (10240); # log_msg_size (8192); chain_hostnames (no); use_dns (no); use_fqdn (no); create_dirs (yes); keep_hostname (yes); owner("root"); group("root"); dir_owner(root); # dir_group(logs); dir_perm(0750); perm(0640); stats_freq (3600); check_hostname (yes); dns_cache (yes); threaded(yes); mark_freq(600); }; source system { internal(); unix-stream("/dev/log"); udp(); tcp(max_connections(100)); }; filter f_messages { level(info,notice) and not facility(authpriv,cron,mail,news); }; filter f_syslog { level(warn..emerg) and not facility(authpriv,cron,mail,news); }; filter f_debug { level(debug); }; filter f_authpriv { facility(authpriv); }; filter f_cron { facility(cron); }; filter f_mail { facility(mail); }; filter f_emerg { level(emerg); }; filter f_uucp { facility(uucp); }; destination d_messages { file("/var/log/messages"); }; destination d_syslog { file("/var/log/syslog"); }; destination d_debug { file("/var/log/debug"); }; destination d_secure { file("/var/log/secure"); }; destination d_cron { file("/var/log/cron"); }; destination d_maillog { file("/var/log/maillog"); }; destination d_usertty { usertty("*"); }; destination d_spooler { file("/var/log/spooler"); }; # Log anything 'info' or higher, but lower than 'warn'. # Exclude authpriv, cron, mail, and news. These are logged elsewhere. log { source(system); filter(f_messages); destination(d_messages); }; # Log anything 'warn' or higher. # Exclude authpriv, cron, mail, and news. These are logged elsewhere. log { source(system); filter(f_syslog); destination(d_syslog); }; # Debugging information is logged here. log { source(system); filter(f_debug); destination(d_debug); }; # Private authentication message logging: log { source(system); filter(f_authpriv); destination(d_secure); }; # Cron related logs: log { source(system); filter(f_cron); destination(d_cron); }; # Mail related logs: log { source(system); filter(f_mail); destination(d_maillog); }; # Emergency level messages go to all users: log { source(system); filter(f_emerg); destination(d_usertty); }; # This log is for news and uucp errors: log { source(system); filter(f_uucp); destination(d_spooler); }; # Uncomment this to see kernel messages on the console. #filter f_kern { facility(kern); }; #destination d_console { file("/dev/console"); }; #log { source(system); filter(f_kern); destination(d_console); }; # Uncomment these if you'd like INN to keep logs on everything. # You won't need this if you don't run INN (the InterNetNews daemon). #filter f_news_crit { facility(news) and level(crit); }; #filter f_news_err { facility(news) and level(err); }; #filter f_news_notice { facility(news) and level(notice); }; #destination d_news_crit { file("/var/log/news/news.crit"); }; #destination d_news_err { file("/var/log/news/news.err"); }; #destination d_news_notice { file("/var/log/news/news.notice"); }; #log { source(system); filter(f_news_crit); destination(d_news_crit); }; #log { source(system); filter(f_news_err); destination(d_news_err); }; #log { source(system); filter(f_news_notice); destination(f_news_notice); }; # Remote logging source s_remote { tcp(ip(0.0.0.0) port(514)); udp(ip(0.0.0.0) port(514)); }; destination logpile { file("/var/log/$HOST/$YEAR/$MONTH/$FACILITY.$YEAR$MONTH$DAY" owner(root) group(root) perm(0600) create_dirs(yes) dir_perm(0700)); }; log { source(system); destination(logpile); }; Cleint syslog-ng.conf I just added the lines belo to the server file above # Remote logging destination logpile { file("/var/log/$HOST/$YEAR/$MONTH/$FACILITY.$YEAR$MONTH$DAY" owner(root) group(root) perm(0600) create_dirs(yes) dir_perm(0700)); }; destination remote { tcp("196.43.133.98" port(514)); }; log { source(system); destination(logpile); }; log { source(system); destination(remote); }; testing for connection root@client:/# lsof -i | grep syslog-ng syslog-ng 18339 root 9u IPv4 37485820 0t0 UDP *:syslog syslog-ng 18339 root 10u IPv4 37485822 0t0 TCP *:shell (LISTEN) syslog-ng 18339 root 11u IPv4 37485824 0t0 TCP client:48305-> syslog server:shell (ESTABLISHED) I am not seeing any logging being registered . How can i solve this and where have i gone wrong. Thanx i advance

2 1

[Bug 187] New: Syslog-ng truncates SDATA PARAM VALUE to 256 chars
by bugzilla＠bugzilla.balabit.com 27 Jul '12

27 Jul '12

https://bugzilla.balabit.com/show_bug.cgi?id=187 Summary: Syslog-ng truncates SDATA PARAM VALUE to 256 chars Product: syslog-ng Version: 3.2.x Platform: All OS/Version: All Status: NEW Severity: major Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: cloun-rulez(a)yandex.ru QAContact: cloun-rulez(a)yandex.ru Type of the Report: enhancement Estimated Hours: 0.0 As you can see there are no limitations of SDATA PARAM VALUE length in rfc5424 http://tools.ietf.org/html/rfc5424 As I have saw in tests/unit/test_msgparse.c this is default behavior of syslog-ng. // too long sdata value gets truncated testcase("<132>1 2006-10-29T01:59:59.156+01:00 mymachine evntslog - - [a i=\"aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa\"] An application event log entry...", LP_SYSLOG_PROTOCOL, NULL, 132, // pri 1162083599, 156000, 3600, // timestamp (sec/usec/zone) "mymachine", // host "evntslog", //app "An application event log entry...", // msg "[a i=\"aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa\"]", //sd_str "",//processid "",//msgid expected_sd_pairs_test_5b ); I'm trying to increase limit of truncation but still haven't success. Need help. -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 5

non-UTF-8 data in mongodb
by Lucas, Sascha 25 Jul '12

25 Jul '12

Hi, I was looking for a way to have some log-rotate functionality (move logs older then x into another collection). I heard of TTL collections, which is fine for deletion. To keep collections small, I tried following mongo-shell code: for( var c = db.messages.find({DATE: {$lt: "1342688698"}}); c.hasNext(); ) { db.messages.old.save(c.next()); } Being totally new to mongodb, I don’t know if this is the right solution, however I got the following error: Mon Jul 23 12:30:34 decode failed. probably invalid utf-8 string [opx2 opx2-dispatch.exe(8100) 2 OPX-20001 Attempt to do an array operation on #26\META-SUPER-HYPER-� which is not an array.] Mon Jul 23 12:30:34 why: TypeError: malformed UTF-8 character sequence at offset 98 Mon Jul 23 12:30:34 Error: invalid utf8 shell/collection.js:128 Asking on mongodb-user Tad Marshall explained that running mongodb with --objcheck prevents this problem. The mogodb config file has following comment: # Inspect all client data for validity on receipt (useful for # developing drivers) #objcheck = true Is the syslog-ng shipped driver the root cause? Should syslog-ng utf8-encode the strings? Is this a bug, or do I miss something? TIA, Sascha. Vorsitzender des Aufsichtsrates: Ralf Hiltenkamp Geschäftsführung: Michael Krüger (Sprecher), Stephan Drescher Sitz der Gesellschaft: Halle/Saale Registergericht: Amtsgericht Stendal | Handelsregister-Nr. HRB 208414 UST-ID-Nr. DE 158253683 Diese E-Mail enthält vertrauliche und/oder rechtlich geschützte Informationen. Wenn Sie nicht der richtige Empfänger sind oder diese E-Mail irrtümlich erhalten haben, informieren Sie bitte sofort den Absender und vernichten Sie diese Mail. Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser Mail oder des Inhalts dieser Mail sind nicht gestattet. Diese Kommunikation per E-Mail ist nicht gegen den Zugriff durch Dritte geschützt. Die GISA GmbH haftet ausdrücklich nicht für den Inhalt und die Vollständigkeit von E-Mails und den gegebenenfalls daraus entstehenden Schaden. Sollte trotz der bestehenden Viren-Schutzprogramme durch diese E-Mail ein Virus in Ihr System gelangen, so haftet die GISA GmbH - soweit gesetzlich zulässig - nicht für die hieraus entstehenden Schäden.

2 2

Glob character ranges
by T. A. Smooth 24 Jul '12

24 Jul '12

Any one have an example of a glob character range for matching a value? match("*[0-9]" value ("MSG") type ("glob") ) The above should match Dog 1 Dog 2 Dog 3

5 7

Syslog-ng MongoDB inserting date data type
by Michael Dillon 24 Jul '12

24 Jul '12

I'm using Epilog (Snare) to push syslog messages from my IIS logs to syslog-ng. I'm using the CSV parser to parse the message body and create nv pairs. I can't seem to figure out how to get the messages inserted with a datetime data type. Any insights? -- Michael Dillon

2 1

prevent duplicate messages in logs
by Sergey Naumov 23 Jul '12

23 Jul '12

Hello. I would like to ask what mechanism syslog-ng provides to prevent duplicate messages in a log. In most common scenarios I can use "final" flag, for example, when I want only logging on a local machine (so when I have a single destination). But imagine, that user can specify any combination of facilities/priorities and destinations, for example, user want to log auth/>=info and all/>=notice to local machine and auth/>=debug to remote. Then I have a problem: I can't use "final" flag for these entries, because some messages (say auth/>=info) will not be logged to remote host, but only auth/debug, and if I don't use final flag, there will be duplicates of all auth/>=notice messages in the local log. Is there a way to solve this problem in general? One way is to make a separate log entry for each facility/priority combination and group all destinations there, but it won't work if user specify some additional custom filters. Thanks in advance, Sergey Naumov.

2 2

Question regarding lib/nvtable.c (FAIL: test_nvtable)
by John Skopis 20 Jul '12

20 Jul '12

Hello, I just tried to rebuild the EPEL6 SRPM for syslog-ng. The build fails on make check because the test_nvtable test fails. Forgive me if the solution has already been posted. I was able to find a post, but it linked to a few RH bugs suggesting improper use of g_hash_table_insert in nv_registry_alloc_handle(nvtable.c). I looked at nvtable.c and noticed the hash is created: g_hash_table_new_full(g_str_hash, g_str_equal, NULL, NULL) Then, in nv_registry_free: for (i = 0; i < self->names->len; i++) g_free(g_array_index(self->names, NVHandleDesc, i).name); g_array_free(self->names, TRUE); I am wondering if there is a good reason not to pass in g_free as the "key_destroy_func" to g_hash_table_new_full (http://developer.gnome.org/glib/2.29/glib-Hash-Tables.html#g-hash-table-new…) ? Thanks, John

1 0

what's next for syslog-ng?
by Peter Czanik 20 Jul '12

20 Jul '12

Hello, It was in last weeks syslog-ng newsletter, but as I'd like to see more input, I post it again: Sources for some previously Premium Edition-only features have been available for a while now, but have not been merged into the Open Source Edition yet. As we want to release OSE 3.4 soon, we are unlikely to have the necessary resources to merge all of these. We would like to ask Your opinion on which of these features would you like to see in the upcoming release. Please vote and share your reasoning in the BalaBit community forum at http://communities.balabit.com/balabit/topics/poll_which_of_these_features_… Thanks for your feedback in these hot summer days! :) Bye, -- Peter Czanik (CzP) <czanik(a)balabit.hu> BalaBit IT Security / syslog-ng upstream http://czanik.blogs.balabit.com/

2 2

mongodb with auth in OSE 3.3/3.4
by Lucas, Sascha 20 Jul '12

20 Jul '12

Hi, I would like to have mongodb support with auth enabled. With syslog-ng-OSE 3.3.5 this seems not supported? destination d_mongo { mongodb( username("XXXXXXX") password("XXXXXXXXXXXX")); }; results in not connecting right to mongodb (despite no configuration error). As an excuse, not being a dev, I grep-ed these parameters from the source, but not from the docs :). ATM I have no luck with 3.4.0alpha3 and mongodb (even without auth). But what I'm interested in, is to know if "mongodb with auth" will be supported in the 3.4 release? Or what future plans exists? Thanks Sascha. Vorsitzender des Aufsichtsrates: Ralf Hiltenkamp Geschäftsführung: Michael Krüger (Sprecher), Stephan Drescher Sitz der Gesellschaft: Halle/Saale Registergericht: Amtsgericht Stendal | Handelsregister-Nr. HRB 208414 UST-ID-Nr. DE 158253683 Diese E-Mail enthält vertrauliche und/oder rechtlich geschützte Informationen. Wenn Sie nicht der richtige Empfänger sind oder diese E-Mail irrtümlich erhalten haben, informieren Sie bitte sofort den Absender und vernichten Sie diese Mail. Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser Mail oder des Inhalts dieser Mail sind nicht gestattet. Diese Kommunikation per E-Mail ist nicht gegen den Zugriff durch Dritte geschützt. Die GISA GmbH haftet ausdrücklich nicht für den Inhalt und die Vollständigkeit von E-Mails und den gegebenenfalls daraus entstehenden Schaden. Sollte trotz der bestehenden Viren-Schutzprogramme durch diese E-Mail ein Virus in Ihr System gelangen, so haftet die GISA GmbH - soweit gesetzlich zulässig - nicht für die hieraus entstehenden Schäden.

2 1

reliable log transfer with syslog-ng-OSE (a bit like PEs' diskbuffering)
by Lucas, Sascha 19 Jul '12

19 Jul '12

Hi, I'm trying to have reliable log transfer (in the sense of network failure) with syslog-ng-OSE. Here my idea (syslog-ng.conf): ## version='3.3.5' ## first save every message to a file source s_relay { tcp( max-connections(10) port(4321) ); udp( port(4321) ); }; destination d_buffer { file("/data/syslog-buffer" template("$ISODATE $HOST $SOURCEIP $PROGRAM $PID $PRI $MSG\n") ); }; log { source(s_relay); destination(d_buffer); }; ## then follow this file and parse it with flow-control enabled source s_buffer { file("/data/syslog-buffer" flags(no-parse) follow_freq(0.1)); }; parser p_buffer { csv-parser( columns("buffer.isodate", "buffer.host", "buffer.srcip", "buffer.program", "buffer.pid", "buffer.pri", "buffer.msg") delimiters(" ") flags(greedy) ); }; destination d_mongo_unrel { mongodb(...); }; log { source(s_buffer); parser(p_buffer); destination(d_mongo_unrel); flags(flow-control); }; What do you think about this solution? Are there objections? I need to add follow_freq() in s_buffer. Without it, the mongodb fills a bit slow. But I don't understand why. Can someone give me a hint, how to get the actual reading position from s_buffer (i.e. from syslog-ng.persist?) and then to truncate it (shell commands preferred)? TIA, Sascha. Vorsitzender des Aufsichtsrates: Ralf Hiltenkamp Geschäftsführung: Michael Krüger (Sprecher), Stephan Drescher Sitz der Gesellschaft: Halle/Saale Registergericht: Amtsgericht Stendal | Handelsregister-Nr. HRB 208414 UST-ID-Nr. DE 158253683 Diese E-Mail enthält vertrauliche und/oder rechtlich geschützte Informationen. Wenn Sie nicht der richtige Empfänger sind oder diese E-Mail irrtümlich erhalten haben, informieren Sie bitte sofort den Absender und vernichten Sie diese Mail. Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser Mail oder des Inhalts dieser Mail sind nicht gestattet. Diese Kommunikation per E-Mail ist nicht gegen den Zugriff durch Dritte geschützt. Die GISA GmbH haftet ausdrücklich nicht für den Inhalt und die Vollständigkeit von E-Mails und den gegebenenfalls daraus entstehenden Schaden. Sollte trotz der bestehenden Viren-Schutzprogramme durch diese E-Mail ein Virus in Ihr System gelangen, so haftet die GISA GmbH - soweit gesetzlich zulässig - nicht für die hieraus entstehenden Schäden.

2 1

Draft and incomplete documentation for syslog-ng OSE 3.4
by Fekete Robert 19 Jul '12

19 Jul '12

Hi Everyone, I have started to update the Administrator Guide for version 3.4. It is far from finished, and several things are still missing, but as I won't be able to work on it in August, I have released it as it is, since it already contains some updates and useful information. You can find it at http://www.balabit.com/sites/default/files/documents/syslog-ng-ose-3.4-guid… The list of new stuff that is already documented is available at: http://www.balabit.com/sites/default/files/documents/syslog-ng-ose-3.4-guid… Some other smaller changes and corrections is listed here: http://www.balabit.com/sites/default/files/documents/syslog-ng-ose-3.4-guid… I hope to get back to finishing the updates early September, but this depends on other things as well. Regards, Robert

1 0

Fwd: Re: Following a file to generate one syslog message per appended line
by Terry Burton 18 Jul '12

18 Jul '12

---------- Forwarded message ---------- From: "Terry Burton" <tez(a)terryburton.co.uk> Date: 17 Jul 2012 19:52 Subject: Re: [syslog-ng] Following a file to generate one syslog message per appended line To: "Balazs Scheidler" <bazsi77(a)gmail.com> On 17 Jul 2012 19:30, "Balazs Scheidler" <bazsi77(a)gmail.com> wrote: > > ----- Original message ----- > > On 17 July 2012 15:20, Jim Hendrick <jrhendri(a)maine.rr.com> wrote: > > > hmmm - nothing obvious to me. > > > > > > Questions that might help > > > - do you know what the message rate for that source is? > > > > The source is the alerts from an IDS where the messages arrive at > > irregular intervals and often in surges of many messages. > > > > > - is there anything possibly unusual about the messages themselves? > > > (how is end of line demarked? what are the maximum line lengths?) > > > > The end of line is demarked by a LF (0x0a). > > > > $ xxd /srv/snort/snort.fast > > > > 0000000: 3037 2f31 362d 3134 3a30 333a 3431 2e33 07/16-14:03:41.3 > > 0000010: 3837 3339 3020 205b 2a2a 5d20 5b31 3a31 87390 [**] [1:1 > > 0000020: 3535 3137 3a31 305d 203c 6574 6831 3e20 5517:10] <eth1> > > 0000030: 5745 422d 434c 4945 4e54 204d 6963 726f WEB-CLIENT Micro > > 0000040: 736f 6674 2057 696e 646f 7773 2041 5649 soft Windows AVI > > 0000050: 2044 6972 6563 7453 686f 7720 5175 6963 DirectShow Quic > > 0000060: 6b54 696d 6520 7061 7273 696e 6720 6f76 kTime parsing ov > > 0000070: 6572 666c 6f77 2061 7474 656d 7074 205b erflow attempt [ > > 0000080: 2a2a 5d20 5b43 6c61 7373 6966 6963 6174 **] [Classificat > > 0000090: 696f 6e3a 2041 7474 656d 7074 6564 2055 ion: Attempted U > > 00000a0: 7365 7220 5072 6976 696c 6567 6520 4761 ser Privilege Ga > > 00000b0: 696e 5d20 5b50 7269 6f72 6974 793a 2031 in] [Priority: 1 > > 00000c0: 5d20 7b54 4350 7d20 322e 3232 2e32 3238 ] {TCP} 2.22.228 > > 00000d0: 2e34 303a 3830 202d 3e20 3134 332e 3231 .40:80 -> 143.21 > > 00000e0: 302e 3936 2e31 3138 3a36 3238 3732 0a30 0.96.118:62872.0 > > 00000f0: 372f 3136 2d31 343a 3033 3a34 312e 3537 7/16-14:03:41.57 > > 0000100: 3836 3535 2020 5b2a 2a5d 205b 313a 3135 8655 [**] [1:15 > > 0000110: 3531 373a 3130 5d20 3c65 7468 313e 2057 517:10] <eth1> W > > 0000120: 4542 2d43 4c49 454e 5420 4d69 6372 6f73 EB-CLIENT Micros > > 0000130: 6f66 7420 5769 6e64 6f77 7320 4156 4920 oft Windows AVI > > 0000140: 4469 7265 6374 5368 6f77 2051 7569 636b DirectShow Quick > > 0000150: 5469 6d65 2070 6172 7369 6e67 206f 7665 Time parsing ove > > 0000160: 7266 6c6f 7720 6174 7465 6d70 7420 5b2a rflow attempt [* > > 0000170: 2a5d 205b 436c 6173 7369 6669 6361 7469 *] [Classificati > > 0000180: 6f6e 3a20 4174 7465 6d70 7465 6420 5573 on: Attempted Us > > 0000190: 6572 2050 7269 7669 6c65 6765 2047 6169 er Privilege Gai > > 00001a0: 6e5d 205b 5072 696f 7269 7479 3a20 315d n] [Priority: 1] > > 00001b0: 207b 5443 507d 2032 2e32 322e 3232 382e {TCP} 2.22.228. > > 00001c0: 3430 3a38 3020 2d3e 2031 3433 2e32 3130 40:80 -> 143.210 > > 00001d0: 2e39 362e 3131 383a 3632 3837 320a 3037 .96.118:62872.07 > > 00001e0: 2f31 362d 3134 3a30 333a 3530 2e39 3534 /16-14:03:50.954 > > 00001f0: 3939 3620 205b 2a2a 5d20 5b31 3a35 3930 996 [**] [1:590 > > 0000200: 333a 395d 203c 6574 6831 3e20 5350 5957 3:9] <eth1> SPYW > > 0000210: 4152 452d 5055 5420 4164 7761 7265 2064 ARE-PUT Adware d > > 0000220: 6f77 6e6c 6f61 6420 6163 6365 6c65 7261 ownload accelera > > 0000230: 746f 7220 706c 7573 2072 756e 7469 6d65 tor plus runtime > > 0000240: 2064 6574 6563 7469 6f6e 202d 2067 6574 detection - get > > 0000250: 2061 6473 205b 2a2a 5d20 5b43 6c61 7373 ads [**] [Class > > 0000260: 6966 6963 6174 696f 6e3a 204d 6973 6320 ification: Misc > > 0000270: 6163 7469 7669 7479 5d20 5b50 7269 6f72 activity] [Prior > > 0000280: 6974 793a 2033 5d20 7b54 4350 7d20 3134 ity: 3] {TCP} 14 > > 0000290: 332e 3231 302e 3139 332e 3639 3a34 3937 3.210.193.69:497 > > 00002a0: 3535 202d 3e20 3231 322e 3134 332e 3232 55 -> 212.143.22 > > 00002b0: 2e31 3130 3a38 300a 3037 2f31 362d 3134 .110:80.07/16-14 > > 00002c0: 3a30 333a 3536 2e34 3536 3034 3320 205b :03:56.456043 [ > > 00002d0: 2a2a 5d20 5b31 3a31 3535 3137 3a31 305d **] [1:15517:10] > > 00002e0: 203c 6574 6831 3e20 5745 422d 434c 4945 <eth1> WEB-CLIE > > 00002f0: 4e54 204d 6963 726f 736f 6674 2057 696e NT Microsoft Win > > 0000300: 646f 7773 2041 5649 2044 6972 6563 7453 dows AVI DirectS > > 0000310: 686f 7720 5175 6963 6b54 696d 6520 7061 how QuickTime pa > > 0000320: 7273 696e 6720 6f76 6572 666c 6f77 2061 rsing overflow a > > 0000330: 7474 656d 7074 205b 2a2a 5d20 5b43 6c61 ttempt [**] [Cla > > 0000340: 7373 6966 6963 6174 696f 6e3a 2041 7474 ssification: Att > > 0000350: 656d 7074 6564 2055 7365 7220 5072 6976 empted User Priv > > 0000360: 696c 6567 6520 4761 696e 5d20 5b50 7269 ilege Gain] [Pri > > 0000370: 6f72 6974 793a 2031 5d20 7b54 4350 7d20 ority: 1] {TCP} > > 0000380: 3233 2e36 372e 3235 352e 3536 3a38 3020 23.67.255.56:80 > > 0000390: 2d3e 2031 3433 2e32 3130 2e31 3030 2e37 -> 143.210.100.7 > > 00003a0: 333a 3535 3539 330a 3037 2f31 362d 3134 3:55593.07/16-14 > > 00003b0: 3a30 343a 3434 2e36 3232 3930 3820 205b :04:44.622908 [ > > 00003c0: 2a2a 5d20 5b31 3a31 3131 3932 3a31 325d **] [1:11192:12] > > <...snip...> > > > > > I would personally try removing the flags (from source and log lines) > > > and one by one add them back - looking at the changes in behavior (if > > > any), then taking that one back out and adding the other. > > > > I've tried that to no useful effect unfortunately. It appears that > > flush_lines(1) is the default setting. > > > > I am wondering whether the file source driver does not treat a lone LF > > as a new line and therefore log_fetch_limit(1) is ineffective? I am > > about to test this. > > it should treat one NL (aka LF, ASCII 10) as line terminator. what version of syslog-ng are you using? Version 3.1.3 on Debian Squeeze, 64 bit.

3 2

Following a file to generate one syslog message per appended line
by Terry Burton 17 Jul '12

17 Jul '12

Hi, I am looking to use syslog-ng to follow a file and create one syslog UDP message for each line that is appended to the file. So far I have the following, however this will put multiple lines into a single syslog message when they arrive together: source s_tail_snort { file("/srv/snort/snort.fast" flags(no-parse) ); }; destination to_splunk { udp("143.210.16.141" port(1514) template("$MSG\n")); }; log {source(s_tail_snort); destination(to_splunk); flags(flow-control); }; I have tried adding log_fetch_limit(1) to the source and flush_lines(1) to the destination, but I still get multiple lines per syslog message: source s_tail_snort { file("/srv/snort/snort.fast" flags(no-parse) log_fetch_limit(1) ); }; destination to_splunk { udp("143.210.16.141" port(1514) template("$MSG\n") flush_lines(1)); }; log {source(s_tail_snort); destination(to_splunk); flags(flow-control); }; Am I missing something simple? Thanks, Terry

3 3

syslog-ng Insider - July 2012
by Peter Czanik 12 Jul '12

12 Jul '12

Dear syslog-ng users, This is the 15th issue of the syslog-ng Insider, a monthly newsletter that brings you syslog-ng related news. Your feedback and news tips about the next issue is welcome at documentation(a)balabit.com FEATURED NEWS What's next for syslog-ng ------------------------- Sources for some previously Premium Edition-only features have been available for a while now, but have not been merged into the Open Source Edition yet. As we want to release OSE 3.4 soon, we are unlikely to have the necessary resources to merge all of these. We would like to ask Your opinion on which of these features would you like to see in the upcoming release. Please vote and share your reasoning in the BalaBit community forum at http://communities.balabit.com/balabit/topics/poll_which_of_these_features_… syslog-ng 3.4 alpha3 is released -------------------------------- Summer is here, when usually not much is happening. This is not the case with syslog-ng, where the third alpha of the upcoming version was released. Changes since the previous alpha version include new tag related rewrite rules, a marker option for the JSON parser to ease implementing CEE and many smaller features, like support for Cisco extended timestamp format, ported over from the Premium Edition. Of course there were also many smaller fixes, some of them merged from the 3.3 version. For a complete list, check the announcement at https://lists.balabit.hu/pipermail/syslog-ng-announce/2012-June/000144.html As the changes since v3.3 are less drastic, than were between 3.2 and 3.3, we hope that more people will install it on their test or non critical systems. We hope to gather some feedback, how existing configurations work with syslog-ng 3.4 (so no existing feature was broken), and experiences with the new features. Syslog-ng 3.3.6 is coming ------------------------- This release is a bug-fix release mostly, with a few minor - yet useful - feature enhancements. The most important one is that the patched ivykis syslog-ng 3.3 shipped with until now is no more. We build against upstream ivykis now (still included for convenience, though). Other highlights include much improved systemd support, and an enhancement to the @include feature, and many bugfixes. It will be available in the coming days from http://www.balabit.com/downloads/files?path=/syslog-ng/open-source-edition/… LogZilla 4.0 is released ------------------------ A new major version of LogZilla, previously known as php-syslog-ng, was released last month. Next to its outstanding Cisco network device support, the new release features easier, even unattended installation, enhanced access control and search possibilities. Detailed information about the new release is available at http://www.logzilla.pro/news/releases/4.0 OTHER SHORT NEWS ELSA tested with syslog-ng Premium Edition: http://czanik.blogs.balabit.com/2012/06/elsa-and-syslog-ng-pe/ NEW RELEASES: syslog-ng OSE 3.4 alpha3: https://lists.balabit.hu/pipermail/syslog-ng-announce/2012-June/000144.html syslog-ng PE 4.2.3: https://lists.balabit.hu/pipermail/syslog-ng-announce/2012-July/000145.html syslog-ng OSE 3.3.6: http://www.balabit.com/downloads/files?path=/syslog-ng/open-source-edition/… (once available) ARCHIVE http://insider.blogs.balabit.com/ -- Peter Czanik (CzP) <czanik(a)balabit.hu> BalaBit IT Security / syslog-ng upstream http://czanik.blogs.balabit.com/

1 0

feature request?: monitor open logs for deletion/rename
by Patrick Hemmer 12 Jul '12

12 Jul '12

Something which I think would be an awesome feature would be to monitor open file destinations for the file being renamed or deleted, and then reopen the file in such an event. The benefit of this is that when doing log rotation, you don't have to SIGHUP syslog-ng to make it re-open files. It would also make it so that all destination buffers don't have to be flushed and reopened, just the single file destination. The only downside is portability as not all OSs support the same way of doing this. On linux this can be easily done through inotify, and it looks like the BSD equivalent is kqueue (though I have pretty much no BSD experience). For platforms which there isn't a good method, we could instead fall back to a simple polling. -Patrick

5 9

AUTO: James B Horwath/TheGuardian is out of the office. (returning 07/14/2012)
by James B Horwath 12 Jul '12

12 Jul '12

I am out of the office until 07/14/2012. I am out of the office until Monday July 16, 2012 with limited access to e-mail. If there are Nitro issues please contact Dante Rodino at 212-919-2819 or Sunil Gupta at 610-807-6710. Any other issues please contact Al Limone manager of IT Security Services & Governance. Thanks, Jim Note: This is an automated response to your message "syslog-ng Digest, Vol 87, Issue 11" sent on 7/12/2012 6:00:02. This is the only notification you will receive while this person is away. ----------------------------------------- This message, and any attachments to it, may contain information that is privileged, confidential, and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are notified that any use, dissemination, distribution, copying, or communication of this message is strictly prohibited. If you have received this message in error, please notify the sender immediately by return e-mail and delete the message and any attachments. Thank you.

1 0

Debugging messages correlation in online mode??
by C. L. Martinez 05 Jul '12

05 Jul '12

Hi all, Is it possible to enable some flag to debug patterndb.xml, contexts defined, variables, etc in online mode when correlation is used?? Thanks.

2 1

syslog-ng 3.4.0alpha3 has been released
by devel＠balabit.hu 05 Jul '12

05 Jul '12

------------------------------------------------------------------------------ PACKAGE : syslog-ng VERSION : 3.4.0alpha3 SUMMARY : new stable release DATE : Jun 30, 2012 ------------------------------------------------------------------------------ DESCRIPTION: A new version of syslog-ng Open Source Edition (3.4.0alpha3) has been released. For latest fixes in the 3.4.x branch you are recommended to upgrade to this version. CHANGES: 3.4.0alpha3 Fri, 29 Jun 2012 15:32:54 +0200 Features: ========= * Added set-tag() and clear-tag() rewrite rules. * Added a marker() option to the json-parser(), which indicates that a message is indeed has a json payload. Used to implement a cee-style payload parser. http://bazsi.blogs.balabit.com/2012/05/cee-prototype-and-a-show-case-for-th… * The system source automatically detectd systemd and defines the proper log socket accordingly. Also, if the system() source is not used, a workaround to use the systemd supplied log socket instead of /dev/log was applied. This kicks in if syslog-ng is running under systemd and its config version is 3.3 or less. * Support for FreeBSD's utmpx database. Featured ported from the Premium Edition ======================================== * Added 12 hour time related macros $AMPM and $HOUR12 * Added $USEC and $MSEC macros for micro and milliseconds respectively. * Added $SYSUPTIME that expands to the time since syslog-ng started in the number of 1/100th of a second (just like in the SNMP sysUptime value). * db-parser(): added field extraction to the matching or program names * Added support for Cisco extended timestamp format, one that includes time synchronity information and sequence numbers. https://github.com/bazsi/syslog-ng-3.4/commit/046f90e76137f31e61412d24b9920… * Added support for mark-mode() that allows the customization how MARK messages are generated. https://github.com/bazsi/syslog-ng-3.4/commit/c38d7ad9a2d40650e243ec29c6b3e… Bugfixes: ========= * Fixed a possible segmentation fault in threaded mode when dns-cache-hosts() is used. * Incorrect encoding of data caused UDP and unix-dgram() receives to be erroneously closed, so no further data would be received. This has been fixed. Incorrect encoding of incoming data causes the connection for connection oriented transports (like tcp() and unix-stream()) to be closed, but the same condition is ignored for datagram oriented transports (like udp() or unix-dgram()). * Accept empty source and destination object in the configuration file. * Fixed a linking issue when using the SMTP destination. * Ignore SIGHUP in the supervisor process. * Fixed fsync() behaviour for destination files. * Fixed a possible NULL deref in db-parser() correllation when context-scope is not defined. Other changes: ============== * Added a --disable-smtp configure option. * Integrated unit test refactorization changes from the BalaBit syslog-ng and SSB teams. * Added systemd alias "syslog" to the unit file. Credits: ======== syslog-ng is developed as a community project, and as such it relies on volunteers to do the work necessarily to produce syslog-ng. Reporting bugs, testing changes, writing code or simply providing feedback are all important contributions, so please if you are a user of syslog-ng, contribute. These people have helped in this release: Alexander Komyagin <komyagin(a)altell.ru> EgonB <egon(a)local.ee> Evan Rempel <erempel(a)uvic.ca> Imre Lazar <imre(a)balabit.hu> Jose Pedro Oliveira <jpo(a)di.uminho.pt> Mark Ulmer <mark.ulmer(a)apollogrp.edu> Patrick Hemmer Peter Czanik <czanik(a)balabit.hu> Attila Magyar <athos(a)balabit.hu> Attila Nagy <naat(a)balabit.hu> Balazs Scheidler <bazsi(a)balabit.hu> Gergely Nagy <algernon(a)balabit.hu> Juhasz Viktor <jviktor(a)balabit.hu> Peter Gyorko <gyorkop(a)balabit.hu> Viktor Juhasz <jviktor(a)balabit.hu> DOWNLOAD: You can download the source packages from: http://www.balabit.com/network-security/syslog-ng/opensource-logging-system… The documentation of the syslog-ng Open Source Edition is available in The syslog-ng Open Source Edition Administrator's Guide at http://www.balabit.com/support/documentation/

3 6