[syslog-ng]Sorting data from msg field into correct columns?

Tue, 6 May 2003 08:53:18 +0200

This is a multi-part message in MIME format.

--------------InterScan_NT_MIME_Boundary
Content-Type: multipart/alternative;
	boundary="----_=_NextPart_001_01C3139C.2CE135C0"

------_=_NextPart_001_01C3139C.2CE135C0
Content-Type: text/plain;
	charset="US-ASCII"
Content-Transfer-Encoding: quoted-printable

Hi list,
=20
 I can be more specific.
=20
I have in a DMZ proxies, routers and firewalls. Theses report into a
syslog server in the DMZ.
=20
I want to have the messages in one central place so I send them to my
central syslog server with a MYSQL DB.
=20
There they arrive but they look like following example.=20
a.a.a.a =3D Where it comes from originally (at least what I think ;-))
b.b.b.b =3D NAME of the syslog server
=20
A router message
| a.a.a.a | syslog5   | info     | info  | ae   | 2003-05-06 | 04:21:38
| b.b.b.b | b.b.b.b : [ID 72119 syslog5.info]  04:21:36 drop   rgw2
>eth0 reason: ACL: Ingress filter dropped packet ;  src: a.b.c.d;
s_port: 933; dst: f.g.h.i; d_port: 514;  | 1015510 |
=20
A firewall message
| c.c.c.c | syslog5   | info     | info  | ae   | 2003-05-06 | 04:21:38
| b.b.b.b | b.b.b.b: [ID 72119 syslog5.info]  04:21:37 drop   fwgw >eth1
product: SmartDefense; TCP sequence validator: dropped packet with
invalid ACK number; attack: Bad TCP sequence; src:  i.j.k.l; s_port: 80;
dst: m.n.o.p; service: shell; proto: tcp; | 1015511 |
So I would like to have the machine name as rgw2 or fwgw on the hosts or
facility's place. Maybe the word 'drop' as level so I can create an
index for fast search.=20
Ideal would be, if I could do a search in the $MSG and then filter the
entry and putting parts of the $MSG into different variabls e.g. $HOST
etc.=20
=20
I guess this could already be done with a regexp search in the $MSG
field then assigning the different variables. But I guess I have to
hardcode all cases and I'd just like to take something like $MSG.$1,
$MSG.$2 etc.
=20
This allowed me to do some kind of "message normalisation". Maybe that's
impossible for some reasons (speed).
=20
Regards,
=20
Michael
=20
-----Original Message-----
From: Balazs Scheidler [mailto:bazsi@balabit.hu]=20
Sent: Montag, 5. Mai 2003 14:39
To: syslog-ng@lists.balabit.hu
Subject: Re: [syslog-ng]Sorting data from msg field into correct
columns?
=20
On Mon, May 05, 2003 at 10:55:07AM +0200, Michael.Semling@swisscom.com
wrote:
> Hi all!
>=20
> As I have now added hardware to report into a central syslog but I
have
> to uses a relay. (Another SYSLOG-NG)
> Now  I get messages containing
>=20
> Host: IP/Name of the relay
> Facility, Priority, Level, tag, date, time and program is just all
info
> from the relay.
>=20
> In the msg I have all the data as the original host, time, msg. etc.
>=20
> Is there any way to "grep/sed/awk" the data out of the msg field and
> sort them into the right columns?
>=20
> Is there an example for such a filter?

Can you be more specific, can you perhaps provide examples (what you
would
expect and what happens in reality)

--
Bazsi
PGP info: KeyID 9AF8D0A9 Fingerprint CD27 CFB0 802C 0944 9CFD 804E C82C
8EB1
_______________________________________________
syslog-ng maillist  -  syslog-ng@lists.balabit.hu
https://lists.balabit.hu/mailman/listinfo/syslog-ng
Frequently asked questions at http://www.campin.net/syslog-ng/faq.html

------_=_NextPart_001_01C3139C.2CE135C0
Content-Type: text/html;
	charset="US-ASCII"
Content-Transfer-Encoding: quoted-printable

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2//EN">
<html xmlns:o=3D"urn:schemas-microsoft-com:office:office" =
xmlns:w=3D"urn:schemas-microsoft-com:office:word" =
xmlns:st1=3D"urn:schemas-microsoft-com:office:smarttags" =
xmlns=3D"http://www.w3.org/TR/REC-html40">

<head>
<meta http-equiv=3DContent-Type content=3D"text/html; =
charset=3Dus-ascii">
<meta name=3DProgId content=3DWord.Document>
<meta name=3DGenerator content=3D"Microsoft Word 10">
<meta name=3DOriginator content=3D"Microsoft Word 10">
<link rel=3DFile-List href=3D"cid:filelist.xml@01C313AC.F0B5E120">
<title>Re: [syslog-ng]Sorting data from msg field into correct =
columns?</title>
<o:SmartTagType =
namespaceuri=3D"urn:schemas-microsoft-com:office:smarttags"
 name=3D"time"/>
<o:SmartTagType =
namespaceuri=3D"urn:schemas-microsoft-com:office:smarttags"
 name=3D"PersonName"/>
<!--[if gte mso 9]><xml>
 <o:OfficeDocumentSettings>
  <o:DoNotRelyOnCSS/>
 </o:OfficeDocumentSettings>
</xml><![endif]--><!--[if gte mso 9]><xml>
 <w:WordDocument>
  <w:GrammarState>Clean</w:GrammarState>
  <w:DocumentKind>DocumentEmail</w:DocumentKind>
  <w:EnvelopeVis/>
  <w:BrowserLevel>MicrosoftInternetExplorer4</w:BrowserLevel>
 </w:WordDocument>
</xml><![endif]--><!--[if !mso]>
<style>
st1\:*{behavior:url(#default#ieooui) }
</style>
<![endif]-->
<style>
<!--
 /* Font Definitions */
 @font-face
	{font-family:Tahoma;
	panose-1:2 11 6 4 3 5 4 4 2 4;
	mso-font-charset:0;
	mso-generic-font-family:swiss;
	mso-font-pitch:variable;
	mso-font-signature:553679495 -2147483648 8 0 66047 0;}
 /* Style Definitions */
 p.MsoNormal, li.MsoNormal, div.MsoNormal
	{mso-style-parent:"";
	margin:0cm;
	margin-bottom:.0001pt;
	mso-pagination:widow-orphan;
	font-size:12.0pt;
	font-family:"Times New Roman";
	mso-fareast-font-family:"Times New Roman";}
a:link, span.MsoHyperlink
	{color:blue;
	text-decoration:underline;
	text-underline:single;}
a:visited, span.MsoHyperlinkFollowed
	{color:blue;
	text-decoration:underline;
	text-underline:single;}
p
	{mso-margin-top-alt:auto;
	margin-right:0cm;
	mso-margin-bottom-alt:auto;
	margin-left:0cm;
	mso-pagination:widow-orphan;
	font-size:12.0pt;
	font-family:"Times New Roman";
	mso-fareast-font-family:"Times New Roman";}
span.EmailStyle18
	{mso-style-type:personal-reply;
	mso-style-noshow:yes;
	mso-ansi-font-size:10.0pt;
	mso-bidi-font-size:10.0pt;
	font-family:Arial;
	mso-ascii-font-family:Arial;
	mso-hansi-font-family:Arial;
	mso-bidi-font-family:Arial;
	color:navy;}
span.GramE
	{mso-style-name:"";
	mso-gram-e:yes;}
@page Section1
	{size:612.0pt 792.0pt;
	margin:72.0pt 90.0pt 72.0pt 90.0pt;
	mso-header-margin:35.4pt;
	mso-footer-margin:35.4pt;
	mso-paper-source:0;}
div.Section1
	{page:Section1;}
-->
</style>
<!--[if gte mso 10]>
<style>
 /* Style Definitions */=20
 table.MsoNormalTable
	{mso-style-name:"Table Normal";
	mso-tstyle-rowband-size:0;
	mso-tstyle-colband-size:0;
	mso-style-noshow:yes;
	mso-style-parent:"";
	mso-padding-alt:0cm 5.4pt 0cm 5.4pt;
	mso-para-margin:0cm;
	mso-para-margin-bottom:.0001pt;
	mso-pagination:widow-orphan;
	font-size:10.0pt;
	font-family:"Times New Roman";}
</style>
<![endif]-->
</head>

<body lang=3DEN-US link=3Dblue vlink=3Dblue =
style=3D'tab-interval:36.0pt'>

<div class=3DSection1>

<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;mso-bidi-font-family:Arial;color:navy'><!-- =
Converted from text/plain format -->Hi
list,<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;mso-bidi-font-family:Arial;color:navy'><o:p>&nbs=
p;</o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;mso-bidi-font-family:Arial;color:navy'><span
style=3D'mso-spacerun:yes'>&nbsp;</span>I can be more =
specific.<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;mso-bidi-font-family:Arial;color:navy'><o:p>&nbs=
p;</o:p></span></font></p>

I have =
in a DMZ
proxies, routers and firewalls. Theses report into a syslog server in =
the DMZ.<o:p></o:p>

<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;mso-bidi-font-family:Arial;color:navy'><o:p>&nbs=
p;</o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;mso-bidi-font-family:Arial;color:navy'>I want =
to have
the messages in one central place so I send them to my central syslog =
server
with a MYSQL DB.<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;mso-bidi-font-family:Arial;color:navy'><o:p>&nbs=
p;</o:p></span></font></p>

There =
they
arrive but they look like following example. =
<o:p></o:p>

<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;mso-bidi-font-family:Arial;color:navy'>a.a.a.a =
=3D Where
it comes from originally (at least what I think =
;-))<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;mso-bidi-font-family:Arial;color:navy'>b.b.b.b =
=3D NAME of
the syslog server<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;mso-bidi-font-family:Arial;color:navy'><o:p>&nbs=
p;</o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;mso-bidi-font-family:Arial;color:navy'>A router
message<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;mso-bidi-font-family:Arial;color:navy'>| =
a.a.a.a | syslog5<span
style=3D'mso-spacerun:yes'>&nbsp;&nbsp; </span>| info<span
style=3D'mso-spacerun:yes'>&nbsp;&nbsp;&nbsp;&nbsp; </span>| <span =
class=3DGramE>info<span
style=3D'mso-spacerun:yes'>&nbsp; </span>|</span> ae<span
style=3D'mso-spacerun:yes'>&nbsp;&nbsp; </span>| 2003-05-06 | =
</span></font><st1:time
Hour=3D"4" Minute=3D"21"><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
 =
10.0pt;font-family:Arial;mso-bidi-font-family:Arial;color:navy'>04:21:38<=
/span></font></st1:time><font
size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:10.0pt;font-family:Arial;
mso-bidi-font-family:Arial;color:navy'> | b.b.b.b | b.b.b.b : [ID 72119 =
syslog5.info]<span
style=3D'mso-spacerun:yes'>&nbsp; </span></span></font><st1:time =
Hour=3D"4"
Minute=3D"21"><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:10.0pt;
 =
font-family:Arial;mso-bidi-font-family:Arial;color:navy'>04:21:36</span><=
/font></st1:time><font
size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:10.0pt;font-family:Arial;
mso-bidi-font-family:Arial;color:navy'> drop<span
style=3D'mso-spacerun:yes'>&nbsp;&nbsp; </span>rgw2 &gt;eth0 reason: =
ACL: Ingress
filter dropped packet ; <span =
style=3D'mso-spacerun:yes'>&nbsp;</span>src: a.b.c.d;
s_port: 933; dst: f.g.h.i; d_port: 514; <span
style=3D'mso-spacerun:yes'>&nbsp;</span>| 1015510 =
|<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;mso-bidi-font-family:Arial;color:navy'><o:p>&nbs=
p;</o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;mso-bidi-font-family:Arial;color:navy'>A =
firewall
message<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;mso-bidi-font-family:Arial;color:navy'>| =
c.c.c.c | syslog5<span
style=3D'mso-spacerun:yes'>&nbsp;&nbsp; </span>| info<span
style=3D'mso-spacerun:yes'>&nbsp;&nbsp;&nbsp;&nbsp; </span>| <span =
class=3DGramE>info<span
style=3D'mso-spacerun:yes'>&nbsp; </span>|</span> ae<span
style=3D'mso-spacerun:yes'>&nbsp;&nbsp; </span>| 2003-05-06 | =
</span></font><st1:time
Hour=3D"4" Minute=3D"21"><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
 =
10.0pt;font-family:Arial;mso-bidi-font-family:Arial;color:navy'>04:21:38<=
/span></font></st1:time><font
size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:10.0pt;font-family:Arial;
mso-bidi-font-family:Arial;color:navy'> | b.b.b.b | b.b.b.b: [ID 72119 =
syslog5.info]<span
style=3D'mso-spacerun:yes'>&nbsp; </span></span></font><st1:time =
Hour=3D"4"
Minute=3D"21"><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:10.0pt;
 =
font-family:Arial;mso-bidi-font-family:Arial;color:navy'>04:21:37</span><=
/font></st1:time><font
size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:10.0pt;font-family:Arial;
mso-bidi-font-family:Arial;color:navy'> drop<span
style=3D'mso-spacerun:yes'>&nbsp;&nbsp; </span>fwgw &gt;eth1 product:
SmartDefense; TCP sequence validator: dropped packet with invalid ACK =
number;
attack: Bad TCP sequence; src: <span =
style=3D'mso-spacerun:yes'>&nbsp;</span>i.j.k.l;
s_port: 80; dst: m.n.o.p; service: shell; proto: tcp; | 1015511 =
|<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;mso-bidi-font-family:Arial;color:navy'><o:p></o:=
p></span></font></p>

So I =
would like
to have the machine name as rgw2 or fwgw on the hosts or =
facility&#8217;s place.
Maybe the word &#8216;drop&#8217; as level so I can create an index for =
fast
search. <o:p></o:p>

Ideal =
would be,
if I could do a search in the $MSG and then filter the entry and putting =
parts
of the $MSG into different variabls e.g. $HOST etc. =
<o:p></o:p>

<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;mso-bidi-font-family:Arial;color:navy'><o:p>&nbs=
p;</o:p></span></font></p>

I guess =
this
could already be done with a regexp search in the $MSG field then =
assigning the
different variables. But I guess I have to hardcode all cases and =
I&#8217;d
just like to take something like $MSG.$1, =
$MSG.$2 etc.<o:p></o:p>

<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;mso-bidi-font-family:Arial;color:navy'><o:p>&nbs=
p;</o:p></span></font></p>

This =
allowed me
to do some kind of &#8220;message normalisation&#8221;. Maybe =
that&#8217;s
impossible for some reasons (speed).<o:p></o:p>

<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;mso-bidi-font-family:Arial;color:navy'><o:p>&nbs=
p;</o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;mso-bidi-font-family:Arial;color:navy'>Regards,<=
o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;mso-bidi-font-family:Arial;color:navy'><o:p>&nbs=
p;</o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;mso-bidi-font-family:Arial;color:navy'>Michael<o=
:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;mso-bidi-font-family:Arial;color:navy'><o:p>&nbs=
p;</o:p></span></font></p>

-----Original =
Message----- 
From: Balazs Scheidler
[mailto:bazsi@balabit.hu] 
Sent: Montag, 5. Mai 2003 14:39 
To: =
<st1:PersonName>syslog-ng@lists.balabit.hu<=
/span></st1:PersonName> 
Subject: Re: =
[syslog-ng]Sorting data from msg field into correct =
columns?

<p class=3DMsoNormal style=3D'margin-left:36.0pt'><font size=3D3
face=3D"Times New Roman"><span =
style=3D'font-size:12.0pt'><o:p>&nbsp;</o:p></span></font></p>

<p =
style=3D'mso-margin-top-alt:5.0pt;margin-right:0cm;margin-bottom:12.0pt;
margin-left:36.0pt'><font size=3D2 face=3D"Times New Roman"><span =
style=3D'font-size:
10.0pt'>On Mon, May 05, 2003 at 10:55:07AM +0200, =
Michael.Semling@swisscom.com
wrote:<br>
&gt; Hi all!<br>
&gt;<span class=3DGramE>&nbsp;<br>
&gt;</span> As I have now added hardware to report into a central syslog =
but I
have<br>
&gt; <span class=3DGramE>to</span> uses a relay. (Another SYSLOG-NG)<br>
&gt; Now<span class=3DGramE>&nbsp; I</span> get messages containing<br>
&gt;<span class=3DGramE>&nbsp;<br>
&gt;</span> Host: IP/Name of the relay<br>
&gt; Facility, Priority, Level, tag, date, time and program is just all =
info<br>
&gt; <span class=3DGramE>from</span> the relay.<br>
&gt;<span class=3DGramE>&nbsp;<br>
&gt;</span> In the msg I have all the data as the original host, time, =
msg.
etc.<br>
&gt;<span class=3DGramE>&nbsp;<br>
&gt;</span> Is there any way to &quot;grep/sed/awk&quot; the data out of =
the msg
field and<br>
&gt; sort them into the right columns?<br>
&gt;<span class=3DGramE>&nbsp;<br>
&gt;</span> Is there an example for such a filter?<br>
<br>
Can you be more specific, can you perhaps provide examples (what you =
would<br>
<span class=3DGramE>expect</span> and what happens in reality)<br>
<br>
--<br>
Bazsi<br>
PGP info: KeyID 9AF8D0A9 Fingerprint CD27 CFB0 802C 0944 9CFD 804E C82C =
8EB1<br>
_______________________________________________<br>
<span class=3DGramE>syslog-ng</span> maillist&nbsp; -&nbsp; =
</span></font><st1:PersonName><font
 size=3D2><span =
style=3D'font-size:10.0pt'>syslog-ng@lists.balabit.hu</span></font></st1:=
PersonName><font
size=3D2><span style=3D'font-size:10.0pt'><br>
<a =
href=3D"https://lists.balabit.hu/mailman/listinfo/syslog-ng">https://list=
s.balabit.hu/mailman/listinfo/syslog-ng</a><br>
Frequently asked questions at <a =
href=3D"http://www.campin.net/syslog-ng/faq.html">http://www.campin.net/s=
yslog-ng/faq.html</a></span></font><o:p></o:p></p>

</div>

</body>

</html>
=00
------_=_NextPart_001_01C3139C.2CE135C0--

--------------InterScan_NT_MIME_Boundary--