WANTED: feature owner of syslog-ng command line tool :)
Hello! A new feature is going to be developed, as a part of GSOC, so syslog-ng can be started as a command line tool. What could the CLI do for you? - pipelines of complex filtering, rewriting could be tested more easily, thus configuration could become simpler than before - existing logs could be transformed The format of the new command line options is not decided yet. So your help and insight is needed to design the interface and the usage of the tool that you and other people around the world would use. :) If you are intereted in contributing to the CLI, thus contributing to the syslog-ng community, please send an email to this thread. :) I am available at gitter for further info. :) Regards kvch
Hi, This is an email where I tried to describe what I had in mind with the original idea: http://comments.gmane.org/gmane.comp.syslog-ng/17449 It would be great to have this implemented. Cheers, Bazsi -- Bazsi On Fri, May 20, 2016 at 2:10 PM, Noémi Ványi <sitbackandwait@gmail.com> wrote:
Hello!
A new feature is going to be developed, as a part of GSOC, so syslog-ng can be started as a command line tool.
What could the CLI do for you?
- pipelines of complex filtering, rewriting could be tested more easily, thus configuration could become simpler than before - existing logs could be transformed
The format of the new command line options is not decided yet. So your help and insight is needed to design the interface and the usage of the tool that you and other people around the world would use. :)
If you are intereted in contributing to the CLI, thus contributing to the syslog-ng community, please send an email to this thread. :) I am available at gitter for further info. :)
Regards kvch
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Hi, This is a great one, it will help sysadmins tremendously. Some of it is available in "pbdtool match", but it's not a very flexible as the available syslog-ng facilities (filter, template) are hardwired. On the other hand, it helps with gathering user stories, I've been using this for the following tasks: - patterndb building process -> testing the current patterndb coverage and identify outliers in existing log data - "offline" processing log messages, without running a syslog-ng daemon - basic filtering, "grep on steroids" - advanced, message parser (e.g. patterndb) based filtering - rewrite, adding metadata and anonymisation - transform output using templates - postprocessing old raw logs with new patterndb rules What is not available in pbdtool is the heuristics in the initial message parsing of the source drivers, it would be great to have that implemented in the stdin driver, as it would save a great deal of time and resources if you don't have to transform the source material to a format that pdbtool likes. I for myself see specifying the configuration in the command line a bit clumsy, I would not want to deal with escaping quotation marks and parentheses and rather use an input file. Also, the supplied config snippet can grow very fast to a level where it is not quite well readable without indentation. Nevertheless this might be useful for quick, one-shot tests. It would also be great if this tool could be used to test very complex log routing setups by being able to use your _existing_ syslog-ng configuration and feed a test message to it. It could prefix the output with the name of the chosen destination or it you want to take the idea further, you could specify which source and destination should be attached to stdin and stdout respectively, overriding the configured source and destination with the new drivers. The closest thing to this, that comes to my mind, is when you are stubbing out input and output streams of a library function within a unit test. I'm not sure that this fits in the scope of the GSoC project, but would save quite a lot of debug time on some setups I've encountered. Best regards, Balint On 05/21/2016 03:02 PM, Scheidler, Balázs wrote:
Hi,
This is an email where I tried to describe what I had in mind with the original idea:
http://comments.gmane.org/gmane.comp.syslog-ng/17449
It would be great to have this implemented.
Cheers, Bazsi
-- Bazsi
On Fri, May 20, 2016 at 2:10 PM, Noémi Ványi <sitbackandwait@gmail.com <mailto:sitbackandwait@gmail.com>> wrote:
Hello!
A new feature is going to be developed, as a part of GSOC, so syslog-ng can be started as a command line tool.
What could the CLI do for you?
* pipelines of complex filtering, rewriting could be tested more easily, thus configuration could become simpler than before * existing logs could be transformed
The format of the new command line options is not decided yet. So your help and insight is needed to design the interface and the usage of the tool that you and other people around the world would use. :)
If you are intereted in contributing to the CLI, thus contributing to the syslog-ng community, please send an email to this thread. :) I am available at gitter for further info. :)
Regards kvch
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Really good ideas here. Thanks for sharing. Some answers inline. On May 23, 2016 6:06 PM, "Balint Kovacs" <kovacs.balint@balasys.hu> wrote:
Hi,
This is a great one, it will help sysadmins tremendously. Some of it is
available in "pbdtool match", but it's not a very flexible as the available syslog-ng facilities (filter, template) are hardwired. On the other hand, it helps with gathering user stories, I've been using this for the following tasks:
- patterndb building process -> testing the current patterndb coverage
and identify outliers in existing log data
- "offline" processing log messages, without running a syslog-ng daemon - basic filtering, "grep on steroids" - advanced, message parser (e.g. patterndb) based filtering - rewrite, adding metadata and anonymisation - transform output using templates - postprocessing old raw logs with new patterndb rules
What is not available in pbdtool is the heuristics in the initial message parsing of the source drivers, it would be great to have that implemented in the stdin driver, as it would save a great deal of time and resources if you don't have to transform the source material to a format that pdbtool likes.
I for myself see specifying the configuration in the command line a bit clumsy, I would not want to deal with escaping quotation marks and parentheses and rather use an input file.
Yeah, I think both are needed. Awk supports both specifying it via the command line and specifying a file. Using apostrophes you can specify parentheses without escaping. Anyway,we are in agreement here. Also, the supplied config snippet can grow very fast to a level where it is not quite well readable without indentation. Nevertheless this might be useful for quick, one-shot tests.
It would also be great if this tool could be used to test very complex
log routing setups by being able to use your _existing_ syslog-ng configuration and feed a test message to it. It could prefix the output with the name of the chosen destination or it you want to take the idea further, you could specify which source and destination should be attached to stdin and stdout respectively, overriding the configured source and destination with the new drivers. The closest thing to this, that comes to my mind, is when you are stubbing out input and output streams of a library function within a unit test. I'm not sure that this fits in the scope of the GSoC project, but would save quite a lot of debug time on some setups I've encountered. Good ideas. I am not sure how to handle complex source/destination setups. Also this could be extremely helpful when combined with the built-in interactive config debugger (eg -i in 3.7 and current master)
Best regards, Balint
On 05/21/2016 03:02 PM, Scheidler, Balázs wrote:
Hi,
This is an email where I tried to describe what I had in mind with the
original idea:
http://comments.gmane.org/gmane.comp.syslog-ng/17449
It would be great to have this implemented.
Cheers, Bazsi
-- Bazsi
On Fri, May 20, 2016 at 2:10 PM, Noémi Ványi <sitbackandwait@gmail.com>
wrote:
Hello!
A new feature is going to be developed, as a part of GSOC, so syslog-ng
can be started as a command line tool.
What could the CLI do for you? pipelines of complex filtering, rewriting could be tested more easily,
thus configuration could become simpler than before
existing logs could be transformed The format of the new command line options is not decided yet. So your help and insight is needed to design the interface and the usage of the tool that you and other people around the world would use. :)
If you are intereted in contributing to the CLI, thus contributing to the syslog-ng community, please send an email to this thread. :) I am available at gitter for further info. :)
Regards kvch
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Hi, We use syslog-ng in a very diverse environment of vlans. The majority of the config is driven from python against an ipam. (Maybe API to drive config from Ipam's too?) So for any given subnet we will have a filter, destination and log rule. we have over 700 vlans which makes for a large but simple set of conf files. I would like to see options that syslog-ng-ctl has plus the ability to see filter, log, and destination data in real-time. Even a counter would be helpful. Ideally this would be a socket connection to the running syslog-ng daemon similar to what NcFTP has ncftpd_spy and ncftpd_watch. more to come. Thanks Scot On Fri, May 20, 2016 at 8:10 AM, Noémi Ványi <sitbackandwait@gmail.com> wrote:
Hello!
A new feature is going to be developed, as a part of GSOC, so syslog-ng can be started as a command line tool.
What could the CLI do for you?
- pipelines of complex filtering, rewriting could be tested more easily, thus configuration could become simpler than before - existing logs could be transformed
The format of the new command line options is not decided yet. So your help and insight is needed to design the interface and the usage of the tool that you and other people around the world would use. :)
If you are intereted in contributing to the CLI, thus contributing to the syslog-ng community, please send an email to this thread. :) I am available at gitter for further info. :)
Regards kvch
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Hello! First, thanks for all of the proposals and suggestions. In my proposal I had written the same as Balint did about the command line configuration of syslog-ng (This can be found under the section "Configuration" of my proposal: https://drive.google.com/file/d/0B5rOtg14cnD0ek9TSklPSnZ4YWM/view). I would prefer that existing configuration is testable by adding the configuration elements to the stdin/stdout path. Complex configurations would take forever to type as command line parameters. So I agree with Balint. But I am not sure how big the feature would be if this way is chosen. I will discuss it with my mentor. Scot, what do you mean by "I would like to see options that syslog-ng-ctl has plus the ability to see filter, log, and destination data in real-time."? Do you want to see which messages and how got processed? kvch On 24 May 2016 at 02:35, Scot <scotrn@gmail.com> wrote:
Hi,
We use syslog-ng in a very diverse environment of vlans. The majority of the config is driven from python against an ipam. (Maybe API to drive config from Ipam's too?)
So for any given subnet we will have a filter, destination and log rule. we have over 700 vlans which makes for a large but simple set of conf files.
I would like to see options that syslog-ng-ctl has plus the ability to see filter, log, and destination data in real-time. Even a counter would be helpful.
Ideally this would be a socket connection to the running syslog-ng daemon similar to what NcFTP has ncftpd_spy and ncftpd_watch.
more to come.
Thanks Scot
On Fri, May 20, 2016 at 8:10 AM, Noémi Ványi <sitbackandwait@gmail.com> wrote:
Hello!
A new feature is going to be developed, as a part of GSOC, so syslog-ng can be started as a command line tool.
What could the CLI do for you?
- pipelines of complex filtering, rewriting could be tested more easily, thus configuration could become simpler than before - existing logs could be transformed
The format of the new command line options is not decided yet. So your help and insight is needed to design the interface and the usage of the tool that you and other people around the world would use. :)
If you are intereted in contributing to the CLI, thus contributing to the syslog-ng community, please send an email to this thread. :) I am available at gitter for further info. :)
Regards kvch
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
participants (4)
-
Balint Kovacs
-
Noémi Ványi
-
Scheidler, Balázs
-
Scot