Converting multiline text input to concatanated single-line syslog format
Classification: Public Hi! My first posting on the mailing list. I've run into a snag trying to get multiline logs concatenated onto one line and written as syslog-format messages. Actually the second part is working fine; I can get the first line identified and incorporated into a syslog message, but all subsequent lines aren't included. Just for testing I've some simple input; 'IMP-386: ORACLE error 386 encountered ORA-01017: invalid username/password; logon deniedUsername: Connected to: Oracle Database 10g Enterprise Edition Release 10.2.0.3.0 - 64bit Production With the Partitioning, OLAP and Data Mining options Export file created by EXPORT:V10.02.01 via conventional path Warning: the objects were exported by FALCON, not by you import done in WE8ISO8859P1 character set and AL16UTF16 NCHAR character set IMP-00034: Warning: FromUser "FALBOS" not found in export file' What I'm aiming to do is log only the Oracle errors, commencing at the string 'IMP' and ending only when the next line is found with 'IMP' (this log only sees errors) So in syslog-ng.conf (version 5.0.6b PE) source s_table { file("/path to Table.log" multi-line-prefix("IMP") multi-line-garbage("set$") flags(no-p arse) flags(no-multi-line) program_override("Table") default-facility(local5) default-priority(info)); }; Just to try to get things working, I've hard-coded the final string present in my sample input - 'set' The output is scheduled to go off-server, but for the moment I want to see the transformed messages in their own file; destination d_table { file("/auditsox/table.log"); }; log { source(s_table); destination(d_table); }; What I get from the above (in d_table) is; Sep 29 14:07:15 p14425dev022 Table: IMP-386: ORACLE error 386 encountered < Ok, first line transformed into syslog messages ORA-01017: invalid username/password; logon deniedUsername: Connected to: Oracle Database 10g Enterprise Edition Release 10.2.0.3.0 - 64bit Production With the Partitioning, OLAP and Data Mining options Export file created by EXPORT:V10.02.01 via conventional path Warning: the objects were exported by FALCON, not by you import done in WE8ISO8859P1 character set and AL16UTF16 NCHAR character Sep 29 14:07:26 p14425dev022 Table: IMP-00034: Warning: FromUser "FALBOS" not found in export file _____________________ The snag is, the lines after the first line in the input aren't concatenated, but are rather individual lines, with an LF. So I'm doing something wrong. With this source; source s_table { file("/path to Table.log " flags(no-multi-line) flags(no-parse) program_override(" Table") default-facility(local5) default-priority(info)); }; I can get every line of input to transform into a syslog message. Not much use though! Can anyone provide a pointer as to what I'm doing wonky? Thanks Brendan Lloyds Banking Group plc. Registered Office: The Mound, Edinburgh EH1 1YZ. Registered in Scotland no. SC95000. Telephone: 0131 225 4555. Lloyds Bank plc. Registered Office: 25 Gresham Street, London EC2V 7HN. Registered in England and Wales no. 2065. Telephone 0207626 1500. Bank of Scotland plc. Registered Office: The Mound, Edinburgh EH1 1YZ. Registered in Scotland no. SC327000. Telephone: 03457 801 801. Cheltenham & Gloucester plc. Registered Office: Barnett Way, Gloucester GL4 3RL. Registered in England and Wales 2299428. Telephone: 0345 603 1637 Lloyds Bank plc, Bank of Scotland plc are authorised by the Prudential Regulation Authority and regulated by the Financial Conduct Authority and Prudential Regulation Authority. Cheltenham & Gloucester plc is authorised and regulated by the Financial Conduct Authority. Halifax is a division of Bank of Scotland plc. Cheltenham & Gloucester Savings is a division of Lloyds Bank plc. HBOS plc. Registered Office: The Mound, Edinburgh EH1 1YZ. Registered in Scotland no. SC218813. This e-mail (including any attachments) is private and confidential and may contain privileged material. If you have received this e-mail in error, please notify the sender and delete it (including any attachments) immediately. You must not copy, distribute, disclose or use any of the information in it or any attachments. Telephone calls may be monitored or recorded.
Hi, This seems to be a pe customer. Helping him publicly is a great way to push publicity. Basically he is already a reference with a great company name. Any takers on his problem? Cheers Bazsi ---------- Forwarded message ---------- From: "Newport, Brendan (Contractor - Security Operations - Development & Support)" <Brendan.Newport@lloydsbanking.com> Date: Sep 29, 2016 15:04 Subject: [syslog-ng] Converting multiline text input to concatanated single-line syslog format To: "syslog-ng@lists.balabit.hu" <syslog-ng@lists.balabit.hu> Cc: *Classification: Public*
Hi!
My first posting on the mailing list.
I’ve run into a snag trying to get multiline logs concatenated onto one line and written as syslog-format messages.
Actually the second part is working fine; I can get the first line identified and incorporated into a syslog message, but all subsequent lines aren’t included.
Just for testing I’ve some simple input;
‘IMP-386: ORACLE error 386 encountered
ORA-01017: invalid username/password; logon deniedUsername:
Connected to: Oracle Database 10g Enterprise Edition Release 10.2.0.3.0 - 64bit Production
With the Partitioning, OLAP and Data Mining options
Export file created by EXPORT:V10.02.01 via conventional path
Warning: the objects were exported by FALCON, not by you
import done in WE8ISO8859P1 character set and AL16UTF16 NCHAR character set
IMP-00034: Warning: FromUser "FALBOS" not found in export file’
What I’m aiming to do is log only the Oracle errors, commencing at the string ‘IMP’ and ending only when the next line is found with ‘IMP’ (this log only sees errors)
So in syslog-ng.conf (version 5.0.6b PE)
source s_table { file("/path to Table.log" multi-line-prefix("IMP") multi-line-garbage("set$") flags(no-p
arse) flags(no-multi-line) program_override("Table") default-facility(local5) default-priority(info)); };
Just to try to get things working, I’ve hard-coded the final string present in my sample input - ‘set’
The output is scheduled to go off-server, but for the moment I want to see the transformed messages in their own file;
destination d_table { file("/auditsox/table.log"); };
log { source(s_table); destination(d_table); };
What I get from the above (in d_table) is;
Sep 29 14:07:15 p14425dev022 Table: IMP-386: ORACLE error 386 encountered < Ok, first line transformed into syslog messages
ORA-01017: invalid username/password; logon deniedUsername:
Connected to: Oracle Database 10g Enterprise Edition Release 10.2.0.3.0 - 64bit Production
With the Partitioning, OLAP and Data Mining options
Export file created by EXPORT:V10.02.01 via conventional path
Warning: the objects were exported by FALCON, not by you
import done in WE8ISO8859P1 character set and AL16UTF16 NCHAR character
Sep 29 14:07:26 p14425dev022 Table: IMP-00034: Warning: FromUser "FALBOS" not found in export file
_____________________
The snag is, the lines after the first line in the input aren’t concatenated, but are rather individual lines, with an LF.
So I’m doing something wrong.
With this source;
source s_table { file("/path to Table.log " flags(no-multi-line) flags(no-parse) program_override("
Table") default-facility(local5) default-priority(info)); };
I can get every line of input to transform into a syslog message. Not much use though!
Can anyone provide a pointer as to what I’m doing wonky?
Thanks
Brendan
Lloyds Banking Group plc. Registered Office: The Mound, Edinburgh EH1 1YZ. Registered in Scotland no. SC95000. Telephone: 0131 225 4555. Lloyds Bank plc. Registered Office: 25 Gresham Street, London EC2V 7HN. Registered in England and Wales no. 2065. Telephone 0207626 1500. Bank of Scotland plc. Registered Office: The Mound, Edinburgh EH1 1YZ. Registered in Scotland no. SC327000. Telephone: 03457 801 801. Cheltenham & Gloucester plc. Registered Office: Barnett Way, Gloucester GL4 3RL. Registered in England and Wales 2299428. Telephone: 0345 603 1637
Lloyds Bank plc, Bank of Scotland plc are authorised by the Prudential Regulation Authority and regulated by the Financial Conduct Authority and Prudential Regulation Authority.
Cheltenham & Gloucester plc is authorised and regulated by the Financial Conduct Authority.
Halifax is a division of Bank of Scotland plc. Cheltenham & Gloucester Savings is a division of Lloyds Bank plc.
HBOS plc. Registered Office: The Mound, Edinburgh EH1 1YZ. Registered in Scotland no. SC218813.
This e-mail (including any attachments) is private and confidential and may contain privileged material. If you have received this e-mail in error, please notify the sender and delete it (including any attachments) immediately. You must not copy, distribute, disclose or use any of the information in it or any attachments. Telephone calls may be monitored or recorded.
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/? product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Obviously I never intended to post this publicly, just wanted to motivate BalaBit support team to engage, even though this is not an official support forum. Cheers Bazsi
Novice user but it looks like your using a file source with no-multi-line have you tried sending using any of the suported drivers ? - *no-multi-line*: The no-multi-line flag disables line-breaking in the messages; the entire message is converted to a single line. Note that this happens only if the underlying transport method actually supports multi-line messages. Currently the ,syslog(), network(), unix-dgram() drivers support multi-line messages -
Files do support multi line, so this should be reflected by the manual. On Sep 30, 2016 1:38 AM, "Scot" <scotrn@gmail.com> wrote:
Novice user but it looks like your using a file source with no-multi-line have you tried sending using any of the suported drivers ?
-
*no-multi-line*: The no-multi-line flag disables line-breaking in the messages; the entire message is converted to a single line. Note that this happens only if the underlying transport method actually supports multi-line messages. Currently the ,syslog(), network(), unix-dgram() drivers support multi-line messages -
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/? product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Hi, For what it's worth, I helped a collegue at work integrate various oracle logs with syslog-ng, using quite a few syslog-ng features like patterndb, multiline and last but not least python. If you like, I can hook you cheers
This would be a great addition to our scl library to make this an out-of-the-box experience. Do you think he would be willing to work in that? I would personally thank him with a t-shirt and syslog-ng swag. On Sep 29, 2016 19:35, "Fabien Wernli" <wernli@in2p3.fr> wrote:
Hi,
For what it's worth, I helped a collegue at work integrate various oracle logs with syslog-ng, using quite a few syslog-ng features like patterndb, multiline and last but not least python.
If you like, I can hook you
cheers
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/? product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Dear Brendan, Please be aware that under normal circumstances when following the standard support process, your questions/problems should be forwarder to BalaBit's partner, who is providing end-user support for you and your organization. With that being said, I think that: - the multi-line-prefix() option looks okay in your config - is there any specific reason you have set the multi-line-garbage() option? It simply throws away everything between the garbage, and the next prefix regex pattern. - I would check the file format of the Oracle logs, whether they have UNIX or Windows-style line endings. (CRLFs or plain LFs) - the no-multi-line flag looks okay in your config, it should flatten the logs read in, to one-line messages Also, the 5.0.6b is a rather old version. Currently we are at 5.0.14 with the 5.0 LTS line of syslog-ng PE. Do you have the possibility to test the behavior of the latest release? I tried to check our internal bug-tracker, but failed to find any relevant bugs to this case. Thank You! Best Regards, János Szigetvári -- Janos SZIGETVARI RHCE, License no. 150-053-692 <https://www.redhat.com/rhtapps/verify/?certId=150-053-692> __@__˚V˚ Make the switch to open (source) applications, protocols, formats now: - windows -> Linux, iexplore -> Firefox, msoffice -> LibreOffice - msn -> jabber protocol (Pidgin, Google Talk) - mp3 -> ogg, wmv -> ogg, jpg -> png, doc/xls/ppt -> odt/ods/odp 2016-09-29 16:04 GMT+02:00 Newport, Brendan (Contractor - Security Operations - Development & Support) <Brendan.Newport@lloydsbanking.com>:
*Classification: Public*
Hi!
My first posting on the mailing list.
I’ve run into a snag trying to get multiline logs concatenated onto one line and written as syslog-format messages.
Actually the second part is working fine; I can get the first line identified and incorporated into a syslog message, but all subsequent lines aren’t included.
Just for testing I’ve some simple input;
‘IMP-386: ORACLE error 386 encountered
ORA-01017: invalid username/password; logon deniedUsername:
Connected to: Oracle Database 10g Enterprise Edition Release 10.2.0.3.0 - 64bit Production
With the Partitioning, OLAP and Data Mining options
Export file created by EXPORT:V10.02.01 via conventional path
Warning: the objects were exported by FALCON, not by you
import done in WE8ISO8859P1 character set and AL16UTF16 NCHAR character set
IMP-00034: Warning: FromUser "FALBOS" not found in export file’
What I’m aiming to do is log only the Oracle errors, commencing at the string ‘IMP’ and ending only when the next line is found with ‘IMP’ (this log only sees errors)
So in syslog-ng.conf (version 5.0.6b PE)
source s_table { file("/path to Table.log" multi-line-prefix("IMP") multi-line-garbage("set$") flags(no-p
arse) flags(no-multi-line) program_override("Table") default-facility(local5) default-priority(info)); };
Just to try to get things working, I’ve hard-coded the final string present in my sample input - ‘set’
The output is scheduled to go off-server, but for the moment I want to see the transformed messages in their own file;
destination d_table { file("/auditsox/table.log"); };
log { source(s_table); destination(d_table); };
What I get from the above (in d_table) is;
Sep 29 14:07:15 p14425dev022 Table: IMP-386: ORACLE error 386 encountered < Ok, first line transformed into syslog messages
ORA-01017: invalid username/password; logon deniedUsername:
Connected to: Oracle Database 10g Enterprise Edition Release 10.2.0.3.0 - 64bit Production
With the Partitioning, OLAP and Data Mining options
Export file created by EXPORT:V10.02.01 via conventional path
Warning: the objects were exported by FALCON, not by you
import done in WE8ISO8859P1 character set and AL16UTF16 NCHAR character
Sep 29 14:07:26 p14425dev022 Table: IMP-00034: Warning: FromUser "FALBOS" not found in export file
_____________________
The snag is, the lines after the first line in the input aren’t concatenated, but are rather individual lines, with an LF.
So I’m doing something wrong.
With this source;
source s_table { file("/path to Table.log " flags(no-multi-line) flags(no-parse) program_override("
Table") default-facility(local5) default-priority(info)); };
I can get every line of input to transform into a syslog message. Not much use though!
Can anyone provide a pointer as to what I’m doing wonky?
Thanks
Brendan
Lloyds Banking Group plc. Registered Office: The Mound, Edinburgh EH1 1YZ. Registered in Scotland no. SC95000. Telephone: 0131 225 4555. Lloyds Bank plc. Registered Office: 25 Gresham Street, London EC2V 7HN. Registered in England and Wales no. 2065. Telephone 0207626 1500. Bank of Scotland plc. Registered Office: The Mound, Edinburgh EH1 1YZ. Registered in Scotland no. SC327000. Telephone: 03457 801 801. Cheltenham & Gloucester plc. Registered Office: Barnett Way, Gloucester GL4 3RL. Registered in England and Wales 2299428. Telephone: 0345 603 1637
Lloyds Bank plc, Bank of Scotland plc are authorised by the Prudential Regulation Authority and regulated by the Financial Conduct Authority and Prudential Regulation Authority.
Cheltenham & Gloucester plc is authorised and regulated by the Financial Conduct Authority.
Halifax is a division of Bank of Scotland plc. Cheltenham & Gloucester Savings is a division of Lloyds Bank plc.
HBOS plc. Registered Office: The Mound, Edinburgh EH1 1YZ. Registered in Scotland no. SC218813.
This e-mail (including any attachments) is private and confidential and may contain privileged material. If you have received this e-mail in error, please notify the sender and delete it (including any attachments) immediately. You must not copy, distribute, disclose or use any of the information in it or any attachments. Telephone calls may be monitored or recorded.
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/? product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
participants (6)
-
Balazs Scheidler
-
Fabien Wernli
-
Newport, Brendan (Contractor - Security Operations - Development & Support)
-
Scheidler, Balázs
-
Scot
-
SZIGETVÁRI János