[syslog-ng] Following a file to generate one syslog message per appended line

[Terry Burton]

On 17 July 2012 15:20, Jim Hendrick <jrhendri at maine.rr.com> wrote:
> hmmm - nothing obvious to me.
>
> Questions that might help
> - do you know what the message rate for that source is?

The source is the alerts from an IDS where the messages arrive at
irregular intervals and often in surges of many messages.

> - is there anything possibly unusual about the messages themselves? (how
> is end of line demarked? what are the maximum line lengths?)

The end of line is demarked by a LF (0x0a).

$ xxd /srv/snort/snort.fast

0000000: 3037 2f31 362d 3134 3a30 333a 3431 2e33  07/16-14:03:41.3
0000010: 3837 3339 3020 205b 2a2a 5d20 5b31 3a31  87390  [**] [1:1
0000020: 3535 3137 3a31 305d 203c 6574 6831 3e20  5517:10] <eth1>
0000030: 5745 422d 434c 4945 4e54 204d 6963 726f  WEB-CLIENT Micro
0000040: 736f 6674 2057 696e 646f 7773 2041 5649  soft Windows AVI
0000050: 2044 6972 6563 7453 686f 7720 5175 6963   DirectShow Quic
0000060: 6b54 696d 6520 7061 7273 696e 6720 6f76  kTime parsing ov
0000070: 6572 666c 6f77 2061 7474 656d 7074 205b  erflow attempt [
0000080: 2a2a 5d20 5b43 6c61 7373 6966 6963 6174  **] [Classificat
0000090: 696f 6e3a 2041 7474 656d 7074 6564 2055  ion: Attempted U
00000a0: 7365 7220 5072 6976 696c 6567 6520 4761  ser Privilege Ga
00000b0: 696e 5d20 5b50 7269 6f72 6974 793a 2031  in] [Priority: 1
00000c0: 5d20 7b54 4350 7d20 322e 3232 2e32 3238  ] {TCP} 2.22.228
00000d0: 2e34 303a 3830 202d 3e20 3134 332e 3231  .40:80 -> 143.21
00000e0: 302e 3936 2e31 3138 3a36 3238 3732 0a30  0.96.118:62872.0
00000f0: 372f 3136 2d31 343a 3033 3a34 312e 3537  7/16-14:03:41.57
0000100: 3836 3535 2020 5b2a 2a5d 205b 313a 3135  8655  [**] [1:15
0000110: 3531 373a 3130 5d20 3c65 7468 313e 2057  517:10] <eth1> W
0000120: 4542 2d43 4c49 454e 5420 4d69 6372 6f73  EB-CLIENT Micros
0000130: 6f66 7420 5769 6e64 6f77 7320 4156 4920  oft Windows AVI
0000140: 4469 7265 6374 5368 6f77 2051 7569 636b  DirectShow Quick
0000150: 5469 6d65 2070 6172 7369 6e67 206f 7665  Time parsing ove
0000160: 7266 6c6f 7720 6174 7465 6d70 7420 5b2a  rflow attempt [*
0000170: 2a5d 205b 436c 6173 7369 6669 6361 7469  *] [Classificati
0000180: 6f6e 3a20 4174 7465 6d70 7465 6420 5573  on: Attempted Us
0000190: 6572 2050 7269 7669 6c65 6765 2047 6169  er Privilege Gai
00001a0: 6e5d 205b 5072 696f 7269 7479 3a20 315d  n] [Priority: 1]
00001b0: 207b 5443 507d 2032 2e32 322e 3232 382e   {TCP} 2.22.228.
00001c0: 3430 3a38 3020 2d3e 2031 3433 2e32 3130  40:80 -> 143.210
00001d0: 2e39 362e 3131 383a 3632 3837 320a 3037  .96.118:62872.07
00001e0: 2f31 362d 3134 3a30 333a 3530 2e39 3534  /16-14:03:50.954
00001f0: 3939 3620 205b 2a2a 5d20 5b31 3a35 3930  996  [**] [1:590
0000200: 333a 395d 203c 6574 6831 3e20 5350 5957  3:9] <eth1> SPYW
0000210: 4152 452d 5055 5420 4164 7761 7265 2064  ARE-PUT Adware d
0000220: 6f77 6e6c 6f61 6420 6163 6365 6c65 7261  ownload accelera
0000230: 746f 7220 706c 7573 2072 756e 7469 6d65  tor plus runtime
0000240: 2064 6574 6563 7469 6f6e 202d 2067 6574   detection - get
0000250: 2061 6473 205b 2a2a 5d20 5b43 6c61 7373   ads [**] [Class
0000260: 6966 6963 6174 696f 6e3a 204d 6973 6320  ification: Misc
0000270: 6163 7469 7669 7479 5d20 5b50 7269 6f72  activity] [Prior
0000280: 6974 793a 2033 5d20 7b54 4350 7d20 3134  ity: 3] {TCP} 14
0000290: 332e 3231 302e 3139 332e 3639 3a34 3937  3.210.193.69:497
00002a0: 3535 202d 3e20 3231 322e 3134 332e 3232  55 -> 212.143.22
00002b0: 2e31 3130 3a38 300a 3037 2f31 362d 3134  .110:80.07/16-14
00002c0: 3a30 333a 3536 2e34 3536 3034 3320 205b  :03:56.456043  [
00002d0: 2a2a 5d20 5b31 3a31 3535 3137 3a31 305d  **] [1:15517:10]
00002e0: 203c 6574 6831 3e20 5745 422d 434c 4945   <eth1> WEB-CLIE
00002f0: 4e54 204d 6963 726f 736f 6674 2057 696e  NT Microsoft Win
0000300: 646f 7773 2041 5649 2044 6972 6563 7453  dows AVI DirectS
0000310: 686f 7720 5175 6963 6b54 696d 6520 7061  how QuickTime pa
0000320: 7273 696e 6720 6f76 6572 666c 6f77 2061  rsing overflow a
0000330: 7474 656d 7074 205b 2a2a 5d20 5b43 6c61  ttempt [**] [Cla
0000340: 7373 6966 6963 6174 696f 6e3a 2041 7474  ssification: Att
0000350: 656d 7074 6564 2055 7365 7220 5072 6976  empted User Priv
0000360: 696c 6567 6520 4761 696e 5d20 5b50 7269  ilege Gain] [Pri
0000370: 6f72 6974 793a 2031 5d20 7b54 4350 7d20  ority: 1] {TCP}
0000380: 3233 2e36 372e 3235 352e 3536 3a38 3020  23.67.255.56:80
0000390: 2d3e 2031 3433 2e32 3130 2e31 3030 2e37  -> 143.210.100.7
00003a0: 333a 3535 3539 330a 3037 2f31 362d 3134  3:55593.07/16-14
00003b0: 3a30 343a 3434 2e36 3232 3930 3820 205b  :04:44.622908  [
00003c0: 2a2a 5d20 5b31 3a31 3131 3932 3a31 325d  **] [1:11192:12]
<...snip...>

> I would personally try removing the flags (from source and log lines)
> and one by one add them back - looking at the changes in behavior (if
> any), then taking that one back out and adding the other.

I've tried that to no useful effect unfortunately. It appears that
flush_lines(1) is the default setting.

I am wondering whether the file source driver does not treat a lone LF
as a new line and therefore log_fetch_limit(1) is ineffective? I am
about to test this.


Thanks,

Terry


> On 07/17/2012 09:21 AM, Terry Burton wrote:
>> Hi,
>>
>> I am looking to use syslog-ng to follow a file and create one syslog
>> UDP message for each line that is appended to the file.
>>
>> So far I have the following, however this will put multiple lines into
>> a single syslog message when they arrive together:
>>
>> source s_tail_snort { file("/srv/snort/snort.fast" flags(no-parse) ); };
>> destination to_splunk { udp("143.210.16.141" port(1514) template("$MSG\n")); };
>> log {source(s_tail_snort); destination(to_splunk); flags(flow-control); };
>>
>> I have tried adding log_fetch_limit(1) to the source and
>> flush_lines(1) to the destination, but I still get multiple lines per
>> syslog message:
>>
>> source s_tail_snort { file("/srv/snort/snort.fast" flags(no-parse)
>> log_fetch_limit(1) ); };
>> destination to_splunk { udp("143.210.16.141" port(1514)
>> template("$MSG\n") flush_lines(1)); };
>> log {source(s_tail_snort); destination(to_splunk); flags(flow-control); };
>>
>> Am I missing something simple?
>>
>>
>> Thanks,
>>
>> Terry