[syslog-ng] Converting multiline text input to concatanated single-line syslog format

[SZIGETVÁRI János]

Dear Brendan,

Please be aware that under normal circumstances when following the standard
support process, your questions/problems should be forwarder to BalaBit's
partner, who is providing end-user support for you and your organization.

With that being said, I think that:
- the multi-line-prefix() option looks okay in your config
- is there any specific reason you have set the multi-line-garbage()
option? It simply throws away everything between the garbage, and the next
prefix regex pattern.
- I would check the file format of the Oracle logs, whether they have UNIX
or Windows-style line endings. (CRLFs or plain LFs)
- the no-multi-line flag looks okay in your config, it should flatten the
logs read in, to one-line messages

Also, the 5.0.6b is a rather old version. Currently we are at 5.0.14 with
the 5.0 LTS line of syslog-ng PE.
Do you have the possibility to test the behavior of the latest release? I
tried to check our internal bug-tracker, but failed to find any relevant
bugs to this case.

Thank You!

Best Regards,
János Szigetvári

-- 
Janos SZIGETVARI
RHCE, License no. 150-053-692
<https://www.redhat.com/rhtapps/verify/?certId=150-053-692>

__ at __˚V˚
Make the switch to open (source) applications, protocols, formats now:
- windows -> Linux, iexplore -> Firefox, msoffice -> LibreOffice
- msn -> jabber protocol (Pidgin, Google Talk)
- mp3 -> ogg, wmv -> ogg, jpg -> png, doc/xls/ppt -> odt/ods/odp


2016-09-29 16:04 GMT+02:00 Newport, Brendan (Contractor - Security
Operations - Development & Support) <Brendan.Newport at lloydsbanking.com>:

> *Classification: Public*
>
> Hi!
>
>
>
> My first posting on the mailing list.
>
>
>
> I’ve run into a snag trying to get multiline logs concatenated onto one
> line and written as syslog-format messages.
>
>
>
> Actually the second part is working fine; I can get the first line
> identified and incorporated into a syslog message, but all subsequent lines
> aren’t included.
>
>
>
> Just for testing I’ve some simple input;
>
>
>
>
>
> ‘IMP-386: ORACLE error 386 encountered
>
> ORA-01017: invalid username/password; logon deniedUsername:
>
> Connected to: Oracle Database 10g Enterprise Edition Release 10.2.0.3.0 -
> 64bit Production
>
> With the Partitioning, OLAP and Data Mining options
>
>
>
> Export file created by EXPORT:V10.02.01 via conventional path
>
>
>
> Warning: the objects were exported by FALCON, not by you
>
>
>
> import done in WE8ISO8859P1 character set and AL16UTF16 NCHAR character set
>
> IMP-00034: Warning: FromUser "FALBOS" not found in export file’
>
>
>
> What I’m aiming to do is log only the Oracle errors, commencing at the
> string ‘IMP’ and ending only when the next line is found with ‘IMP’ (this
> log only sees errors)
>
>
>
> So in syslog-ng.conf (version 5.0.6b PE)
>
>
>
> source s_table { file("/path to Table.log" multi-line-prefix("IMP")
> multi-line-garbage("set$") flags(no-p
>
> arse) flags(no-multi-line) program_override("Table")
> default-facility(local5) default-priority(info)); };
>
>
>
> Just to try to get things working, I’ve hard-coded the final string
> present in my sample input - ‘set’
>
>
>
> The output is scheduled to go off-server, but for the moment I want to see
> the transformed messages in their own file;
>
>
>
> destination d_table { file("/auditsox/table.log"); };
>
> log { source(s_table); destination(d_table); };
>
>
>
> What I get from the above (in d_table) is;
>
>
>
>
>
> Sep 29 14:07:15 p14425dev022 Table: IMP-386: ORACLE error 386
> encountered     < Ok, first line transformed into syslog messages
>
> ORA-01017: invalid username/password; logon deniedUsername:
>
> Connected to: Oracle Database 10g Enterprise Edition Release 10.2.0.3.0 -
> 64bit Production
>
> With the Partitioning, OLAP and Data Mining options
>
>
>
> Export file created by EXPORT:V10.02.01 via conventional path
>
>
>
> Warning: the objects were exported by FALCON, not by you
>
>
>
> import done in WE8ISO8859P1 character set and AL16UTF16 NCHAR character
>
> Sep 29 14:07:26 p14425dev022 Table: IMP-00034: Warning: FromUser "FALBOS"
> not found in export file
>
>
>
> _____________________
>
>
>
> The snag is, the lines after the first line in the input aren’t
> concatenated, but are rather individual lines, with an LF.
>
>
>
> So I’m doing something wrong.
>
>
>
> With this source;
>
>
>
> source s_table { file("/path to Table.log " flags(no-multi-line)
> flags(no-parse) program_override("
>
> Table") default-facility(local5) default-priority(info)); };
>
>
>
> I can get every line of input to transform into a syslog message. Not much
> use though!
>
>
>
> Can anyone provide a pointer as to what I’m doing wonky?
>
>
>
> Thanks
>
>
>
>
>
> Brendan
>
> Lloyds Banking Group plc. Registered Office: The Mound, Edinburgh EH1 1YZ.
> Registered in Scotland no. SC95000. Telephone: 0131 225 4555. Lloyds Bank
> plc. Registered Office: 25 Gresham Street, London EC2V 7HN. Registered in
> England and Wales no. 2065. Telephone 0207626 1500. Bank of Scotland plc.
> Registered Office: The Mound, Edinburgh EH1 1YZ. Registered in Scotland no.
> SC327000. Telephone: 03457 801 801. Cheltenham & Gloucester plc. Registered
> Office: Barnett Way, Gloucester GL4 3RL. Registered in England and Wales
> 2299428. Telephone: 0345 603 1637
>
> Lloyds Bank plc, Bank of Scotland plc are authorised by the Prudential
> Regulation Authority and regulated by the Financial Conduct Authority and
> Prudential Regulation Authority.
>
> Cheltenham & Gloucester plc is authorised and regulated by the Financial
> Conduct Authority.
>
> Halifax is a division of Bank of Scotland plc. Cheltenham & Gloucester
> Savings is a division of Lloyds Bank plc.
>
> HBOS plc. Registered Office: The Mound, Edinburgh EH1 1YZ. Registered in
> Scotland no. SC218813.
>
> This e-mail (including any attachments) is private and confidential and
> may contain privileged material. If you have received this e-mail in error,
> please notify the sender and delete it (including any attachments)
> immediately. You must not copy, distribute, disclose or use any of the
> information in it or any attachments. Telephone calls may be monitored or
> recorded.
>
> ____________________________________________________________
> __________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation: http://www.balabit.com/support/documentation/?
> product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.balabit.hu/pipermail/syslog-ng/attachments/20160930/1503444f/attachment-0001.htm