Hello ELK solution is not suitable for us to collect all logs in one server . i prefer to use syslog-ng to collect all logs in Mysql database. is there possible to send all servers log to one syslog-ng server and that collect them to database ? anyone can give me a good configuration file or some examples in this case ? thank you
Hi Arash, On Tue, Sep 15, 2015 at 05:47:44AM +0000, Arash Shams wrote:
ELK solution is not suitable for us to collect all logs in one server . i prefer to use syslog-ng to collect all logs in Mysql database. is there possible to send all servers log to one syslog-ng server and that collect them to database ? anyone can give me a good configuration file or some examples in this case ?
The most straightforward would be to simply define a network() destination. If you want to transfer non-standard key-values (other than PID, PROGRAM, MSG, DATE, PRI, etc.) you will have to add more config though. Two good solutions: * using flags(syslog-protocol) and using .SDATA.* macros * using JSON, i.e. a template like "$(format-json --scope all-nv-pairs)" on one side, and a JSON parser on the remote end (json-parser()) There's lots of examples out there and I'd be happy to point you to some if you can't find them. Cheers
Hi, It is possible but note that there are performance limitations with using MySQL especially if you want to keep data for long term storage. You may want to check one of the nosql solutions (mongo for instance). If you want to stick to single box solutions, you might want to look into syslog-ng Store Box (note it is commercial) that indexes data at pretty high speeds but as of now is not scalable horizontally, but works out of the box for your usecase. Bazsi On Sep 15, 2015 7:47 AM, "Arash Shams" <ara4sh@hotmail.com> wrote:
Hello ELK solution is not suitable for us to collect all logs in one server . i prefer to use syslog-ng to collect all logs in Mysql database. is there possible to send all servers log to one syslog-ng server and that collect them to database ? anyone can give me a good configuration file or some examples in this case ? thank you
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Hi Arash, You could consider a very simple design where the log retention is kept as flat files on disk, with the "searchable" time period in Elasticsearch (or whatever) As I see the advantages: - you can use cheap storage for the retention (since they will not need to be regularly read back or searched) - since you are not adding anything to the data (as happens whenever a log is "parsed" into CEF, json or whatever) this is the most storage efficient form that still preserves all the information - flat files compress extremely well (8 - 10 times with standard gzip) - syslog-ng can easily create files by date, system, whatever so log retention, compression, rotation is trivial - if you need to search something outside your search time window, the files are readily available to be re-indexed or whatever - if you need to share logs with anyone (e.g. a vendor for troubleshooting, an investigation, etc.) gzipped text files are pretty much universally acceptable - you don't even need to backup your search tool, since you can always re-index if necessary - you can even add basic integrity checking by checksumming the files (maybe even before and after compression) Then the search tool (elasticsearch for example) can store a shorter time period (maybe 90 days) and roll off older indices. - makes the search tool faster (fewer logs to search) - simpler to implement (e.g. no backups required) - can use faster / more expensive storage since you need less of it I have found that even security investigations rarely need anything beyond 90 days (not *never* but less than 5% of investigations in my experience). This meets PCI (i.e. store a year at least with 3 months immediately available online) Not saying you can't (or shouldn't) use the search tool for retention - I just like to consider them separately (with correlation / SIEM being a third piece) when designing a logging & monitoring architecture) Just something to think about. Jim On 09/15/2015 01:47 AM, Arash Shams wrote:
Hello ELK solution is not suitable for us to collect all logs in one server . i prefer to use syslog-ng to collect all logs in Mysql database. is there possible to send all servers log to one syslog-ng server and that collect them to database ? anyone can give me a good configuration file or some examples in this case ? thank you
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Hi, Following up on Jim's reply, which I can back a 100%, another good reason not to have a large retention in Elasticsearch is that it's not doing very well with a large number of indices.
Hi, could you explain that? as far as know elastic scales horizontally, isn't that right? -- Bazsi On Fri, Sep 18, 2015 at 3:06 PM, Fabien Wernli <wernli@in2p3.fr> wrote:
Hi,
Following up on Jim's reply, which I can back a 100%, another good reason not to have a large retention in Elasticsearch is that it's not doing very well with a large number of indices.
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Hi Balázs, On Fri, Sep 18, 2015 at 11:12:04PM +0200, Scheidler, Balázs wrote:
could you explain that? as far as know elastic scales horizontally, isn't that right?
You are absolutely right, it *is* designed to scale horizontally. *But* when you have a lot of open indices, *and* queries are not optimized (e.g. if you still give users kibana 3) then you need a *lot* of nodes. So my point was: it's not all that magical as you might think, and needs quite a lot of babysitting from an ops point of view.
Has anyone noticed the segmentation fault in patterndb tool when testing some patterns. The pattern database worked fine under syslog-ng 3.6.3 (and earlier). [xerr@wolverine ~]$ /usr/local/bin/pdbtool --validate test --rule-id ea7cd8ed-0fb7-46d7-8a65-13f42bd8a939 pattern2.xml pattern2.xml validates Testing message: program='ageoutboundquota2' message='Updating S3:walkerh@uvic.ca:24.85.242.83 in (1, 1, 1, 1, 5, 7, 3, 3, 6, 7, 7, 7, 12, 12) -> (0, 0, 1, 1, 5, 7, 3, 3, 6, 7, 7, 7, 12, 12 (Quota 100,1000))' Segmentation fault I have trimmed the pattern database down to a single program with its patterns and tests. I have not yet determined exactly what it is that causes this. Comments welcomed! Evan Rempel.
So it looks like it is the EMAIL parser in patterndb that completely fails Any use of the @EMAIL@ parser causes a segmentation fault. On 09/21/2015 09:31 AM, Evan Rempel wrote:
Has anyone noticed the segmentation fault in patterndb tool when testing some patterns. The pattern database worked fine under syslog-ng 3.6.3 (and earlier).
[xerr@wolverine ~]$ /usr/local/bin/pdbtool --validate test --rule-id ea7cd8ed-0fb7-46d7-8a65-13f42bd8a939 pattern2.xml pattern2.xml validates Testing message: program='ageoutboundquota2' message='Updating S3:walkerh@uvic.ca:24.85.242.83 in (1, 1, 1, 1, 5, 7, 3, 3, 6, 7, 7, 7, 12, 12) -> (0, 0, 1, 1, 5, 7, 3, 3, 6, 7, 7, 7, 12, 12 (Quota 100,1000))' Segmentation fault
I have trimmed the pattern database down to a single program with its patterns and tests.
I have not yet determined exactly what it is that causes this.
Comments welcomed!
Evan Rempel. ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
-- Evan Rempel erempel@uvic.ca Senior Systems Administrator 250.721.7691 Data Centre Services, University Systems, University of Victoria
Patch https://github.com/ihrwein/syslog-ng/commit/50f46c30d3b67d1d4bec5ef0f0afc587... fixes the segmentation fault. On 09/21/2015 12:01 PM, Evan Rempel wrote:
So it looks like it is the EMAIL parser in patterndb that completely fails
Any use of the @EMAIL@ parser causes a segmentation fault.
On 09/21/2015 09:31 AM, Evan Rempel wrote:
Has anyone noticed the segmentation fault in patterndb tool when testing some patterns. The pattern database worked fine under syslog-ng 3.6.3 (and earlier).
[xerr@wolverine ~]$ /usr/local/bin/pdbtool --validate test --rule-id ea7cd8ed-0fb7-46d7-8a65-13f42bd8a939 pattern2.xml pattern2.xml validates Testing message: program='ageoutboundquota2' message='Updating S3:walkerh@uvic.ca:24.85.242.83 in (1, 1, 1, 1, 5, 7, 3, 3, 6, 7, 7, 7, 12, 12) -> (0, 0, 1, 1, 5, 7, 3, 3, 6, 7, 7, 7, 12, 12 (Quota 100,1000))' Segmentation fault
I have trimmed the pattern database down to a single program with its patterns and tests.
I have not yet determined exactly what it is that causes this.
Comments welcomed!
Evan Rempel. ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
-- Evan Rempel erempel@uvic.ca Senior Systems Administrator 250.721.7691 Data Centre Services, University Systems, University of Victoria
Hi Evan, Sounds similar to something which has been fixed a month ago [1]. Could you check out the patch? Cheers [1] https://github.com/balabit/syslog-ng/issues/658
Yes, this is the same patch that I reported fixed this. https://github.com/ihrwein/syslog-ng/commit/50f46c30d3b67d1d4bec5ef0f0afc587... I was just surprized that this was not in the 3.7.1 release ... missed it by 8 days :-( Thanks, On 09/21/2015 01:00 PM, Fabien Wernli wrote:
Hi Evan,
Sounds similar to something which has been fixed a month ago [1]. Could you check out the patch?
Cheers
The release notes for Syslog-ng 3.7.1 include a bug fix to the patternDB as shown below. PatternDB Earlier, in patterndb, the first applicable rule won, even if it was only a partial match. This means that when rules overlapped, the shorter match would have been found, if it was the first to be loaded. A strong preference introduced for rules that match the input string completely. The load order is still applicable though, it is possible to create two distinct rules that would match the same input, in those cases the first one to be loaded wins. ------------- I think that the choice of using the longest match does not reflect the intention of the person making the pattern. Consider these two messages and patterns 1. MESSAGE: Failed to open /var/local/tmp/mailguard/mailguard_timestamp for writing: OS error 5 PATTERN: Failed to open @STRING:workingfile:/_@ for writing: 2. MESSAGE: Failed to open /var/local/tmp/mailguard/mailguard_timestamp PATTERN: Failed to open @ANYSTRING:workingfile@ The first pattern does not match the entire line because it already identifies that it was an error for writing. The second pattern matches the entire line, but the intent was just to match fie file name. ESTRING can not be used to limit the amount of text matched in the #1 message because there is no trailing space on the #2 message. What I am leading up to is the discussion that the first pattern is more DEFINITIVE because it has more STATIC content in the PATTERN. It does not matter that the second pattern matched more of the message. When a human writes a pattern, they try to make the pattern as specific as possible. They do that by including as much STATIC content in the pattern. The more variable content that a pattern matches implies that the pattern is more of a "catch all" than patterns with less variable content. To make the current implimentation of PatternDB work, EVERY pattern must match the ENTIRE message. This means that many patterns will have a @ANYSTRING@ needed at the end of them. I propose that the PatternDB preference be changed from the pattern with the longest MATCH to the pattern with the largest amount of static content. Discussion/comments? -- Evan Rempel
Hi Evan, On Tue, Sep 22, 2015 at 09:49:43AM -0700, Evan Rempel wrote:
I propose that the PatternDB preference be changed from the pattern with the longest MATCH to the pattern with the largest amount of static content.
I fully agree with Evan here: it should work as described in this sentence. That being said, I'm not so sure about the Status quo with 3.7.1. Maybe Balázs can give some more details on the change?
What if we grabbed the tail of the message you are matching with @ANYSTRING@ to a name-value pair automatically, so you don't need anything in your rule, making it a shorter match than the other. What do you think? On Sep 22, 2015 10:43 PM, "Fabien Wernli" <wernli@in2p3.fr> wrote:
Hi Evan,
On Tue, Sep 22, 2015 at 09:49:43AM -0700, Evan Rempel wrote:
I propose that the PatternDB preference be changed from the pattern with the longest MATCH to the pattern with the largest amount of static content.
I fully agree with Evan here: it should work as described in this sentence. That being said, I'm not so sure about the Status quo with 3.7.1. Maybe Balázs can give some more details on the change?
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
That does not really address the issue. By adding @ANYSTRNG@ to then end, we just end up with equally long matches, and then the pattern database load order needs to be controlled to define preference. When merging pattern databases there is no real control over load order. Additionally, if you and an @ANYSTRING@ to the end of a pattern that is designed to match the whole line, you fail to get a match at all because @ANYSTRING@ can not match a zero length string. The real issue is that the most specific pattern needs to be the preferred pattern when there are multiple matches. The discussion is how to define a pattern as being more specific. Consider glob file patterns as an example. Listed from least to most specific, or stated differently, matching the most number of files, to matching the least number of files. * *.doc contract*.doc* contract*2015*.doc contract-purchase-2015.doc The reason that this list is least specific to most specific is because there is more static content to match on. If these were mail filtering rules to store e-mail in different folders, you would never expect the document "contract-sale-2015.doc" to be stored in a folder associated with the rule for the pattern "contract*.doc*". You would expect the rule "contract*2015*.doc" to have priority. It is the amount of static content in the pattern that defines how specific a pattern is. In the above example, it looks easy because the longest pattern is the most specific. That is misleading because using Syslog-ng pattern syntax, a user variable name can be introduced which can make a longer pattern actauly match a shorter message. For example some @STRING:my.variable.matched:"@ here compared to some @STRING@ here @STRING@ done The second is more specific. At my site we had already done this because we store all of our patterns in a database and programmatically create our pattern database by ordering the patterns by the amount of static content. Now that the patterndb was "fixed" in 3.7 we can't use this work around any more, which makes 3.7 break our pattern database :-( Evan. On 09/25/2015 02:35 PM, Scheidler, Balázs wrote:
What if we grabbed the tail of the message you are matching with @ANYSTRING@ to a name-value pair automatically, so you don't need anything in your rule, making it a shorter match than the other.
What do you think?
On Sep 22, 2015 10:43 PM, "Fabien Wernli" <wernli@in2p3.fr <mailto:wernli@in2p3.fr>> wrote:
Hi Evan,
On Tue, Sep 22, 2015 at 09:49:43AM -0700, Evan Rempel wrote: > I propose that the PatternDB preference be changed from the pattern with the longest MATCH to the pattern with the largest amount of static content.
I fully agree with Evan here: it should work as described in this sentence. That being said, I'm not so sure about the Status quo with 3.7.1. Maybe Balázs can give some more details on the change?
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
This body part will be downloaded on demand.
Hi, It is simple to revert to the old behaviour, and maybe we should just do that. Using the amount of literal text in the pattern as the sort order of specifism is a good idea, this could perhaps be added to pdbtool merge. The question is whether the preference over full matches over partial ones should stay as an option or be dropped entirely. What do you think? On Sep 22, 2015 10:43 PM, "Fabien Wernli" <wernli@in2p3.fr> wrote:
Hi Evan,
On Tue, Sep 22, 2015 at 09:49:43AM -0700, Evan Rempel wrote:
I propose that the PatternDB preference be changed from the pattern with the longest MATCH to the pattern with the largest amount of static content.
I fully agree with Evan here: it should work as described in this sentence. That being said, I'm not so sure about the Status quo with 3.7.1. Maybe Balázs can give some more details on the change?
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
There is a work around to this current issue where patterns can be changed to have @ANYSTRING@ at the end. Explicitly doing this, rather than having patterndb do it automatically gives the user the most control/flexibility. Because there is a work around available, I would NOT revert this latest change. If you look at the original bug report https://bugzilla.balabit.com/show_bug.cgi?id=211 it was to use the amount of literal text in the pattern for the order of preference, so I would add this to the patterndb, NOT pdbtool merge, because many people will not use pdbtool merge and the results of a merged and manual pattern database should be consistent. A change request for literal text pattern preference should be added to the TODO list. With regards to "whether the preference over full matches over partial ones should stay as an option", I don't see this as valuable. Using full matches is really just an edge case of longer matches. It does not make sense for the length of a message (rather than the pattern) to influence the order of matching preference. The order of matching needs to be consistent regardless of the log stream input, which is unknown at the time of making the pattern database. Does that make sense? Evan. On 09/26/2015 11:33 AM, Scheidler, Balázs wrote:
Hi,
It is simple to revert to the old behaviour, and maybe we should just do that.
Using the amount of literal text in the pattern as the sort order of specifism is a good idea, this could perhaps be added to pdbtool merge.
The question is whether the preference over full matches over partial ones should stay as an option or be dropped entirely.
What do you think?
On Sep 22, 2015 10:43 PM, "Fabien Wernli" <wernli@in2p3.fr <mailto:wernli@in2p3.fr>> wrote:
Hi Evan,
On Tue, Sep 22, 2015 at 09:49:43AM -0700, Evan Rempel wrote: > I propose that the PatternDB preference be changed from the pattern with the longest MATCH to the pattern with the largest amount of static content.
I fully agree with Evan here: it should work as described in this sentence. That being said, I'm not so sure about the Status quo with 3.7.1. Maybe Balázs can give some more details on the change?
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
This body part will be downloaded on demand.
Is anyone else noticing that under heavy load and with complex configurations that 3.7.1 reload (HUP) that syslog-ng stops reading any sources? It does not die, it just stops processing data. Evan.
Hi, We did not noticed such working yet. Can you give some information to reproduce this issue, for example: OS, syslog-ng configuration (sources/destinations), active inputs, load? Maybe also an strace output will help while reload happened. Best regards, Andrew On Wed, Sep 30, 2015 at 8:10 PM, Evan Rempel <erempel@uvic.ca> wrote:
Is anyone else noticing that under heavy load and with complex configurations that 3.7.1 reload (HUP) that syslog-ng stops reading any sources?
It does not die, it just stops processing data.
Evan.
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Does it stop forever or resume normal processing after X seconds?
participants (7)
-
Arash Shams
-
Balazs Scheidler
-
Evan Rempel
-
Fabien Wernli
-
Jim Hendrick
-
Mitzki, András
-
Scheidler, Balázs