Is there a standard for naming tag/value pairs when parsing
There was a project by Mitre (https://www.mitre.org/) called the Common Event Expression (https://cee.mitre.org/) that was going to be the official standard for metadata names for events, but that project has been stopped. Other than the two references that the CEE project has for logging standardization efforts, does anyone know of any major efforts by any group to define a standard for metadata naming? Evan.
There's common information model at splunk or the field dictionary of CEF, of arcsight fame. I would probably use the splunk one, except if you plan to use arcsight at the end. On Jun 11, 2016 18:32, "Evan Rempel" <erempel@uvic.ca> wrote:
There was a project by Mitre (https://www.mitre.org/) called the Common Event Expression (https://cee.mitre.org/) that was going to be the official standard for metadata names for events, but that project has been stopped.
Other than the two references that the CEE project has for logging standardization efforts, does anyone know of any major efforts by any group to define a standard for metadata naming?
Evan.
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
You are the last person I thought would point me toward the splunk CIM. Given the support that Balabit has put behind CEE and then lumberjack and even the experimental patternDB schema (https://github.com/balabit/syslog-ng-patterndb/blob/master/SCHEMAS.txt) I was sure you would steer me toward lumberjack. At first glance the splunk CIM appears to be structured around and partially dependant on some of the data flows of the splunk product. I'll continue to review it but at this point I am still open to alternate suggestions. Evan. On 06/11/2016 11:45 AM, Scheidler, Balázs wrote:
There's common information model at splunk or the field dictionary of CEF, of arcsight fame.
I would probably use the splunk one, except if you plan to use arcsight at the end.
On Jun 11, 2016 18:32, "Evan Rempel" <erempel@uvic.ca <mailto:erempel@uvic.ca>> wrote:
There was a project by Mitre (https://www.mitre.org/) called the Common Event Expression (https://cee.mitre.org/) that was going to be the official standard for metadata names for events, but that project has been stopped.
Other than the two references that the CEE project has for logging standardization efforts, does anyone know of any major efforts by any group to define a standard for metadata naming?
Evan.
Well, CEE is pretty much dead, and I didn't see too much activity wrt lumberjack either. I would rather see consolidation instead of further fragmentation in this area. Cheers Bazsi You are the last person I thought would point me toward the splunk CIM. Given the support that Balabit has put behind CEE and then lumberjack and even the experimental patternDB schema ( https://github.com/balabit/syslog-ng-patterndb/blob/master/SCHEMAS.txt) I was sure you would steer me toward lumberjack. At first glance the splunk CIM appears to be structured around and partially dependant on some of the data flows of the splunk product. I'll continue to review it but at this point I am still open to alternate suggestions. Evan. On 06/11/2016 11:45 AM, Scheidler, Balázs wrote: There's common information model at splunk or the field dictionary of CEF, of arcsight fame. I would probably use the splunk one, except if you plan to use arcsight at the end. On Jun 11, 2016 18:32, "Evan Rempel" <erempel@uvic.ca> wrote:
There was a project by Mitre (https://www.mitre.org/) called the Common Event Expression (https://cee.mitre.org/) that was going to be the official standard for metadata names for events, but that project has been stopped.
Other than the two references that the CEE project has for logging standardization efforts, does anyone know of any major efforts by any group to define a standard for metadata naming?
Evan.
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Thanks, I just wanted to see your reasoning behind your decision. Does anyone know of any patternDB parsing that was intended to conform to the Splunk CIM that I could take a look at. I'm just trying to shorten the learning curve. Evan. On 06/12/2016 02:44 AM, Scheidler, Balázs wrote:
Well, CEE is pretty much dead, and I didn't see too much activity wrt lumberjack either.
I would rather see consolidation instead of further fragmentation in this area.
Cheers Bazsi
You are the last person I thought would point me toward the splunk CIM. Given the support that Balabit has put behind CEE and then lumberjack and even the experimental patternDB schema (https://github.com/balabit/syslog-ng-patterndb/blob/master/SCHEMAS.txt) I was sure you would steer me toward lumberjack.
At first glance the splunk CIM appears to be structured around and partially dependant on some of the data flows of the splunk product. I'll continue to review it but at this point I am still open to alternate suggestions.
Evan.
On 06/11/2016 11:45 AM, Scheidler, Balázs wrote:
There's common information model at splunk or the field dictionary of CEF, of arcsight fame.
I would probably use the splunk one, except if you plan to use arcsight at the end.
On Jun 11, 2016 18:32, "Evan Rempel" <erempel@uvic.ca <mailto:erempel@uvic.ca>> wrote:
There was a project by Mitre (https://www.mitre.org/) called the Common Event Expression (https://cee.mitre.org/) that was going to be the official standard for metadata names for events, but that project has been stopped.
Other than the two references that the CEE project has for logging standardization efforts, does anyone know of any major efforts by any group to define a standard for metadata naming?
Evan.
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
-- Evan Rempel erempel@uvic.ca Senior Systems Administrator 250.721.7691 Data Centre Services, University Systems, University of Victoria
The way I would do this is to put all conforming fields under the .cim namespace, and then forward it to splunk using $(format-welf --subkeys .cim) This would remove the .cim prefix when forwarding. Other name value pairs are not included by default. But you can always have everything using: $(format-welf *) Cheers, Bazsi Bazsi On Jun 13, 2016 6:31 PM, "Evan Rempel" <erempel@uvic.ca> wrote:
Thanks, I just wanted to see your reasoning behind your decision.
Does anyone know of any patternDB parsing that was intended to conform to the Splunk CIM that I could take a look at. I'm just trying to shorten the learning curve.
Evan.
On 06/12/2016 02:44 AM, Scheidler, Balázs wrote:
Well, CEE is pretty much dead, and I didn't see too much activity wrt lumberjack either.
I would rather see consolidation instead of further fragmentation in this area.
Cheers Bazsi You are the last person I thought would point me toward the splunk CIM. Given the support that Balabit has put behind CEE and then lumberjack and even the experimental patternDB schema ( https://github.com/balabit/syslog-ng-patterndb/blob/master/SCHEMAS.txt) I was sure you would steer me toward lumberjack.
At first glance the splunk CIM appears to be structured around and partially dependant on some of the data flows of the splunk product. I'll continue to review it but at this point I am still open to alternate suggestions.
Evan.
On 06/11/2016 11:45 AM, Scheidler, Balázs wrote:
There's common information model at splunk or the field dictionary of CEF, of arcsight fame.
I would probably use the splunk one, except if you plan to use arcsight at the end. On Jun 11, 2016 18:32, "Evan Rempel" <erempel@uvic.ca> wrote:
There was a project by Mitre (https://www.mitre.org/) called the Common Event Expression (https://cee.mitre.org/) that was going to be the official standard for metadata names for events, but that project has been stopped.
Other than the two references that the CEE project has for logging standardization efforts, does anyone know of any major efforts by any group to define a standard for metadata naming?
Evan.
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
-- Evan Rempel erempel@uvic.ca Senior Systems Administrator 250.721.7691 Data Centre Services, University Systems, University of Victoria
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Hi, There is also an attempt at http://metrics20.org/ that focuses on structure rather than key-name/content
participants (3)
-
Evan Rempel
-
Fabien Wernli
-
Scheidler, Balázs