[syslog-ng] Help parsing logs

[John Norton]

Hi All,



I have a panorama system that sends logs from several devices (dvc_name).
What I want to do is have it write to files based on the dvc_name. The
dvc_name is in the following message as "ip-10-37-12-142".  These are not
static as they are spun up as needed and the dvc_name may change.



Can you help me figure out how to create a parser or filter to take the
following message, where it will read in the dvc_name "ip-10-37-12-142" and
write logs that include that dvc_name to a separate file (and other
dvc_names it sees in messages to their own file?:



*Jun 27 19:08:15 TERL-PANORAMA-01.times.com
<http://terl-panorama-01.times.com/> 1,2024/06/27
15:08:14,005655000463255,SYSTEM,dhcp,2817,2024/06/27
15:08:09,,if-renew-trigger,,0,0,general,informational,"DHCP RENEW:
interface eth0, ip 10.37.12.142 netmask 255.255.255.224 dhcp server:
10.37.12.129",7383625404611793457,0x8000000000000000,0,0,0,0,,ip-10-37-12-142,0,0,2024-06-27T15:08:09.224-04:00*



I've tried a few different things but can't figure out how to do it
dynamically. This rewrite worked, but it requires a static entry which
won't scale and all potential dvc_names cannot be known:



rewrite r_panorama      { set("ip-10-37-12-142", value("HOST")
condition(message(".*ip-10-37-12-142"))); };





I've also tried creating a filter, but it doesn't actually store what the
match finds in a macro that I can replace something like $HOST with.



filter f_panorama                            { match("ip-\d+-\d+-\d+-\d+"
value("MESSAGE")); };



I was also thinking that I could do something with the csv parser, but it
didn't work like I was expecting, but am thinking a parser might work if I
could get it right (I think partially it doesn't work because it is a
syslog message and the first part of the message interferes with the csv
parsing, not sure):



parser p_panorama {

                csv-parser(


columns("future_use1","receive_time","serial_number","log_type","log_subtype","version","generated_time","vsys","event_id","object","future_use3","future_use4","module","severity","description","sequence_number","action_flags","devicegroup_level1","devicegroup_level2","devicegroup_level3","devicegroup_level4","vsys_name","dvc_name")

                                delimiters(",")

                                quote-pairs('""[]')

                );

};



destination d_panorama
{file("`mypath`/panorama/$dvc_name/$dvc_name.log"); };



log { source(s_panother); parser(p_panorama); destination(d_panorama); };



Is this something someone could help me with?



Thank you,

John
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.balabit.hu/pipermail/syslog-ng/attachments/20240628/b264a87c/attachment.htm>