Log to syslog file, filter from fifo
Topic says it. I think I have a pretty simple layout: log { source(s_local); filter(f_firewall); destination(d_file); destination(fifo); }; Can I change this to: log { source(s_local); destination(d_file); filter(f_firewall); destination(fifo); }; ? Will this log to file, then filter, then go to fifo? Thanks and hope this isn't a silly question...I read this from the admin guide: Log statements are processed in the order they appear in the configuration file, thus the order of log paths may influence what happens to a message, especially when using filters and log flags. So I'm hoping that is the case. Thank you. James
We are preparing to leverage the patterndb functionality in a very big way. A few questions before I jump. 1. Is there a public repository of pattern databases? 2. Is there some registry for creating the uniq IDs for rules and rule sets? For example, is there anything that prevents me from creating a rule with a duplicate rule ID that would result in an ID collision when merged with some other patterndb author? 3. Is there any registry for tag names, or key value pair names so that tags that I place on a message will be usable by others syslog-ng configuration filters? 4. Are there any best practices for tag names or key names to provide any kind of grouping? For example; user.name user.uid user.gid host.address.ipv4 host.address.ipv6 host.name Any pointers or discussion will be helpful because we are looking at producing a complete "artificial ignorance" infrastructure for our entire organization, from linux, research compute clusters, network gear, Windows hosts, web hosting, database services etc. Thanks for your time. Evan
No registry that I know of. A while back I think there was some discussion along these lines, but I don't know that anything official ever got hammered out. Since ELSA uses PatternDB, I'm also very interested in this direction, because if a standard were developed, ELSA could be made to both produce and consume these patterns with repository integration. On Fri, Nov 25, 2011 at 12:55 PM, Evan Rempel <erempel@uvic.ca> wrote:
We are preparing to leverage the patterndb functionality in a very big way. A few questions before I jump.
1. Is there a public repository of pattern databases?
2. Is there some registry for creating the uniq IDs for rules and rule sets? For example, is there anything that prevents me from creating a rule with a duplicate rule ID that would result in an ID collision when merged with some other patterndb author?
3. Is there any registry for tag names, or key value pair names so that tags that I place on a message will be usable by others syslog-ng configuration filters?
4. Are there any best practices for tag names or key names to provide any kind of grouping? For example;
user.name user.uid user.gid host.address.ipv4 host.address.ipv6 host.name
Any pointers or discussion will be helpful because we are looking at producing a complete "artificial ignorance" infrastructure for our entire organization, from linux, research compute clusters, network gear, Windows hosts, web hosting, database services etc.
Thanks for your time.
Evan ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Hi Evan, First of all, thank you very much for your lots of useful feedback and comments on patterndb, and our apologies for the slow reply (year-end rush hit us early this year). I try to answer your questions where I can, and hope that Bazsi'll find some time to correct me and answer the rest. See my answers inline. Regards, Robert On Friday, November 25, 2011 19:55 CET, Evan Rempel <erempel@uvic.ca> wrote:
We are preparing to leverage the patterndb functionality in a very big way. A few questions before I jump.
1. Is there a public repository of pattern databases?
Yes, though the number of patterns is rather limited. The current repository is available here: http://git.balabit.hu/?p=bazsi/syslog-ng-patterndb.git Some Windows patterns that are not yet in this repository are available here: http://people.balabit.hu/czanik/patterndb-win2k8.xml A bunch of logcheck rules have been also converted to patterndbs, but from what I have heard the results were not very good, but may or may not be useful for a start (http://www.balabit.com/downloads/files/patterndb-snapshot/).
2. Is there some registry for creating the uniq IDs for rules and rule sets? For example, is there anything that prevents me from creating a rule with a duplicate rule ID that would result in an ID collision when merged with some other patterndb author? No, but if you use some kind of hash, the chance for collision should be pretty low. For example, if you store your patterns under version control (git, whatever), the hash of the parent commit will probably do just fine.
3. Is there any registry for tag names, or key value pair names so that tags that I place on a message will be usable by others syslog-ng configuration filters?
Originally we started to develop a schema for the tags, you can still find it in the SCHEMAS.txt of the repository. Then we heard that MITRE is working (among others) on a standard description for log messages (called Common Event Expression, CEE (http://cee.mitre.org)), and hoped that we could use it instead of reinventing the wheel. Unfortunately, the development of CEE is going slow, so we are stuck in the middle: having stopped to create an own schema so we can use a standard one that'll become more widespread, but that is not here yet. Once a working CEE specification is out, we will convert the tagging of our currently published patterns (which use our own tagging schema) to the CEE schema, so (strictly IMHO) if you use our schema, we will be able to give you at least a mapping on how to convert the tags to the CEE standard.
4. Are there any best practices for tag names or key names to provide any kind of grouping? For example;
user.name user.uid user.gid host.address.ipv4 host.address.ipv6 host.name
See the above comment.
Any pointers or discussion will be helpful because we are looking at producing a complete "artificial ignorance" infrastructure for our entire organization, from linux, research compute clusters, network gear, Windows hosts, web hosting, database services etc.
That sounds really exciting (and a whole lot of work). Please keep us posted about our experiences, both good and bad. Robert
Thanks for your time.
Evan ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
On Friday, November 25, 2011 17:02 CET, "Lay, James" <james.lay@wincofoods.com> wrote:
Topic says it. I think I have a pretty simple layout:
log {
source(s_local);
filter(f_firewall);
destination(d_file);
destination(fifo);
};
Can I change this to:
log {
source(s_local);
destination(d_file);
filter(f_firewall);
destination(fifo);
};
? Will this log to file, then filter, then go to fifo? Thanks and hope this isn't a silly question...I read this from the admin guide:
Yes, it should work that way. Regards, Robert
Log statements are processed in the order they appear in the configuration file, thus the order of log paths may influence what happens to a message, especially when using filters and log flags.
So I'm hoping that is the case. Thank you.
James
Sorry for the late reply, I hope its still relevant: "Lay, James" <james.lay@wincofoods.com> writes:
Topic says it. I think I have a pretty simple layout: [...] log { source(s_local); filter(f_firewall); destination(d_file); destination(fifo); };
Can I change this to:
log { source(s_local); destination(d_file); filter(f_firewall); destination(fifo); };
? Will this log to file, then filter, then go to fifo?
To the best of my knowledge, yes, that is exactly what will happen. -- |8]
Sorry for the late reply, I hope its still relevant:
"Lay, James" <james.lay@wincofoods.com> writes:
Topic says it. I think I have a pretty simple layout: [...] log { source(s_local); filter(f_firewall); destination(d_file); destination(fifo); };
Can I change this to:
log { source(s_local); destination(d_file); filter(f_firewall); destination(fifo); };
? Will this log to file, then filter, then go to fifo?
To the best of my knowledge, yes, that is exactly what will happen.
I can confirm that is exactly how it works...thanks for the response and great app! James
On Fri, 2011-11-25 at 09:02 -0700, Lay, James wrote:
Topic says it. I think I have a pretty simple layout:
log {
source(s_local);
filter(f_firewall);
destination(d_file);
destination(fifo);
};
Can I change this to:
log {
source(s_local);
destination(d_file);
filter(f_firewall);
destination(fifo);
};
? Will this log to file, then filter, then go to fifo? Thanks and hope this isn’t a silly question…I read this from the admin guide:
Log statements are processed in the order they appear in the configuration file, thus the order of log paths may influence what happens to a message, especially when using filters and log flags.
So I’m hoping that is the case. Thank you.
Yes, it is. Starting with syslog-ng 3.0, you can organize your log paths in a tree, which might be more readable, but achieves the same: log { source(s_local); log { destination(d_file); }; log { filter(f_firewall); destination(fifo); }; }; The log {} statements on the 2nd level are branches of the tree rooted at the outmost log statement. You can have any number of embedded statements, and flags(final) is usable too. -- Bazsi
participants (6)
-
Balazs Scheidler
-
Evan Rempel
-
Fekete Róbert
-
Gergely Nagy
-
Lay, James
-
Martin Holste