[syslog-ng] file() sourcing application logs

[Chris Pratt]

Hi syslog-ng,

Been all over google and still not found what I need so I'm hoping to
get some help here...

I have syslog-ng capturing and duplicating all of my *normal*
/var/logs.  I still need to capture some application specific logs that
are not using syslog at all currently but dumping to text in application
log directories.  I wanted to use the file() source but I am not getting
the results I wanted.  I have tried both of these options to get these
files logged:

# See http://www.syslog.org/syslog-ng/v2/#id2535001
# tail -f [file log] | logger -p local#.info
#
# See
http://www.balabit.com/dl/html/syslog-ng-admin-guide_en.html/ch09s01.html
# source s_tail { file("/var/log/apache/access.log" follow_freq(1)
flags(no-parse)); };

but I am running into two issues:

1.) Either above way of sourcing the files results in each line in my
application log showing up as a separate syslog event even though it is
a single application event that wraps lines in its own application log. 
For example:

 {Heap before GC invocations=1087 (full 0):
  par new generation   total 307200K, used 219907K [0x5af10000,
0x73f10000, 0x73f10000)
   eden space 204800K, 100% used [0x5af10000, 0x67710000, 0x67710000)
   from space 102400K,  14% used [0x6db10000, 0x6e9d0d38, 0x73f10000)
   to   space 102400K,   0% used [0x67710000, 0x67710000, 0x6db10000)
  concurrent mark-sweep generation total 991232K, used 116200K
[0x73f10000, 0xb0710000, 0xb0710000)
  concurrent-mark-sweep perm gen total 71680K, used 46017K [0xb0710000,
0xb4d10000, 0xb4d10000)
 27877.615: [GC 27877.615: [ParNew
 Desired survivor size 94371840 bytes, new threshold 15 (max 15)
 - age   1:   32802608 bytes,   32802608 total
 - age   2:     604752 bytes,   33407360 total
 - age   3:      40968 bytes,   33448328 total
 - age   4:      50728 bytes,   33499056 total
 - age   5:      54952 bytes,   33554008 total
 - age   6:      36096 bytes,   33590104 total
 - age   7:      47128 bytes,   33637232 total
 - age   8:      44048 bytes,   33681280 total
 - age   9:      40976 bytes,   33722256 total
 - age  10:      53912 bytes,   33776168 total
 - age  11:      63680 bytes,   33839848 total
 - age  12:      90944 bytes,   33930792 total
 - age  13:      33560 bytes,   33964352 total
 - age  14:      60552 bytes,   34024904 total
 - age  15:      59328 bytes,   34084232 total
 : 219907K->33808K(307200K), 0.0730690 secs] 336108K->150064K(1298432K),
0.0732270 secs]
 Heap after GC invocations=1088 (full 0):
  par new generation   total 307200K, used 33808K [0x5af10000,
0x73f10000, 0x73f10000)
   eden space 204800K,   0% used [0x5af10000, 0x5af10000, 0x67710000)
   from space 102400K,  33% used [0x67710000, 0x69814218, 0x6db10000)
   to   space 102400K,   0% used [0x6db10000, 0x6db10000, 0x73f10000)
  concurrent mark-sweep generation total 991232K, used 116256K
[0x73f10000, 0xb0710000, 0xb0710000)
  concurrent-mark-sweep perm gen total 71680K, used 46017K [0xb0710000,
0xb4d10000, 0xb4d10000)
 }

becomes something like...

2008-06-03T14:27:40-04:00 s_catalinag at tomcat {Heap before GC
invocations=1087 (full 0):
2008-06-03T14:27:40-04:00 s_catalinag at tomcat par new generation   total
307200K, used 219907K [0x5af10000, 0x73f10000, 0x73f10000)
2008-06-03T14:27:40-04:00 s_catalinag at tomcat eden space 204800K, 100%
used [0x5af10000, 0x67710000, 0x67710000)
2008-06-03T14:27:40-04:00 s_catalinag at tomcat from space 102400K,  14%
used [0x6db10000, 0x6e9d0d38, 0x73f10000)
2008-06-03T14:27:40-04:00 s_catalinag at tomcat to   space 102400K,   0%
used [0x67710000, 0x67710000, 0x6db10000)
2008-06-03T14:27:40-04:00 s_catalinag at tomcat concurrent mark-sweep
generation total 991232K, used 116200K [0x73f10000, 0xb0710000, 0xb0710000)
2008-06-03T14:27:40-04:00 s_catalinag at tomcat concurrent-mark-sweep perm
gen total 71680K, used 46017K [0xb0710000, 0xb4d10000, 0xb4d10000)
2008-06-03T14:27:40-04:00 s_catalinag at tomcat 27877.615: [GC 27877.615:
[ParNew
2008-06-03T14:27:40-04:00 s_catalinag at tomcat Desired survivor size
94371840 bytes, new threshold 15 (max 15)
2008-06-03T14:27:40-04:00 s_catalinag at tomcat - age   1:   32802608
bytes,   32802608 total
2008-06-03T14:27:40-04:00 s_catalinag at tomcat - age   2:     604752
bytes,   33407360 total
2008-06-03T14:27:40-04:00 s_catalinag at tomcat - age   3:      40968
bytes,   33448328 total
2008-06-03T14:27:40-04:00 s_catalinag at tomcat - age   4:      50728
bytes,   33499056 total
2008-06-03T14:27:40-04:00 s_catalinag at tomcat - age   5:      54952
bytes,   33554008 total
2008-06-03T14:27:40-04:00 s_catalinag at tomcat - age   6:      36096
bytes,   33590104 total
2008-06-03T14:27:40-04:00 s_catalinag at tomcat - age   7:      47128
bytes,   33637232 total
2008-06-03T14:27:40-04:00 s_catalinag at tomcat - age   8:      44048
bytes,   33681280 total
2008-06-03T14:27:40-04:00 s_catalinag at tomcat - age   9:      40976
bytes,   33722256 total
2008-06-03T14:27:40-04:00 s_catalinag at tomcat - age  10:      53912
bytes,   33776168 total
2008-06-03T14:27:40-04:00 s_catalinag at tomcat - age  11:      63680
bytes,   33839848 total
2008-06-03T14:27:40-04:00 s_catalinag at tomcat - age  12:      90944
bytes,   33930792 total
2008-06-03T14:27:41-04:00 s_catalinag at tomcat - age  13:      33560
bytes,   33964352 total
2008-06-03T14:27:41-04:00 s_catalinag at tomcat - age  14:      60552
bytes,   34024904 total
2008-06-03T14:27:41-04:00 s_catalinag at tomcat - age  15:      59328
bytes,   34084232 total
2008-06-03T14:27:41-04:00 s_catalinag at tomcat : 219907K->33808K(307200K),
0.0730690 secs] 336108K->150064K(1298432K), 0.0732270 secs]
2008-06-03T14:27:41-04:00 s_catalinag at tomcat Heap after GC
invocations=1088 (full 0):
2008-06-03T14:27:41-04:00 s_catalinag at tomcat par new generation   total
307200K, used 33808K [0x5af10000, 0x73f10000, 0x73f10000)
2008-06-03T14:27:41-04:00 s_catalinag at tomcat eden space 204800K,   0%
used [0x5af10000, 0x5af10000, 0x67710000)
2008-06-03T14:27:41-04:00 s_catalinag at tomcat from space 102400K,  33%
used [0x67710000, 0x69814218, 0x6db10000)
2008-06-03T14:27:41-04:00 s_catalinag at tomcat to   space 102400K,   0%
used [0x6db10000, 0x6db10000, 0x73f10000)
2008-06-03T14:27:41-04:00 s_catalinag at tomcat concurrent mark-sweep
generation total 991232K, used 116256K [0x73f10000, 0xb0710000, 0xb0710000)
2008-06-03T14:27:41-04:00 s_catalinag at tomcat concurrent-mark-sweep perm
gen total 71680K, used 46017K [0xb0710000, 0xb4d10000, 0xb4d10000)
2008-06-03T14:27:41-04:00 s_catalinag at tomcat }

2.) I also have a couple of log files that use the date in their name
but from what I can tell macros only work with destinations, not
sources?  I'd like to do something like:

#source s_tomcat {
file("/usr/local/server/apache-tomcat-5.5.20/logs/tomcat.${YEAR}-${MONTH}-${DAY}.log");
};

to accommodate this.  Anyone have any similar app log setup and a
solution to these issues? 

TIA,

Chris