Adding specific information from Syslog-ng to Redis
I am trying to send information from Syslog-ng to Redis. In /etc/syslog-ng/syslog-ng.conf I added this: destination d_redis { redis( host("127.0.0.1") port(6379) command("RPUSH", "sensor_name", "${sensor_name}")); }; I am trying to create a list of variables, hopefully one being timestamp. Here is an example of my syslog that I am trying to pull: http://pastebin.com/Hx5vW4VA Here's is syslog-ng.conf, for reference: http://pastebin.com/2VQFBNmK Those are logs being sent from Snort to Syslog-ng through Snort. I want to connect to Redis. I saw that the command parameters are: comma-separated list of strings ("<redis-command>", "<first-command-parameter>", "<second-command-parameter>", "<third-command-parameter>") from: https://www.balabit.com/documents/syslog-ng-ose-latest-guides/en/syslog-ng-o... I'm assuming I'd say RPUSH <something>... However, I am unsure of how to find the correct parameters. Specifically, for now, I want to create a list (RPUSH) of timestamps, IP addresses (to and from), and event type (ICMP, for example). I did find this list of parameters: https://www.balabit.com/documents/syslog-ng-ose-latest-guides/en/syslog-ng-o... A good first try, I'd like to make a list of timestamps. How can I set the d_redis(command()) within syslog-ng.conf to do this? Sent from Mail for Windows 10
I would embed json formatted strings as redis list elements. You can format that using $(format-json) On Dec 24, 2016 1:17 AM, <johnsc301@gmail.com> wrote:
I am trying to send information from Syslog-ng to Redis. In /etc/syslog-ng/syslog-ng.conf I added this: destination d_redis { redis( host("127.0.0.1") port(6379) command("RPUSH", "sensor_name", "${sensor_name}")); };
I am trying to create a list of variables, hopefully one being timestamp. Here is an example of my syslog that I am trying to pull: http://pastebin.com/Hx5vW4VA
Here's is syslog-ng.conf, for reference: http://pastebin.com/2VQFBNmK
Those are logs being sent from Snort to Syslog-ng through Snort. I want to connect to Redis.
I saw that the command parameters are: comma-separated list of strings ("<redis-command>", "<first-command-parameter>", "<second-command-parameter>", "<third-command-parameter>") from: https://www.balabit.com/documents/syslog-ng-ose- latest-guides/en/syslog-ng-ose-guide-admin/html/ reference-destination-redis.html
I'm assuming I'd say RPUSH <something>... However, I am unsure of how to find the correct parameters.
Specifically, for now, I want to create a list (RPUSH) of timestamps, IP addresses (to and from), and event type (ICMP, for example).
I did find this list of parameters:
https://www.balabit.com/documents/syslog-ng-ose- latest-guides/en/syslog-ng-ose-guide-admin/html/syslog- ng-parameter-index.html
A good first try, I'd like to make a list of timestamps. How can I set the d_redis(command()) within syslog-ng.conf to do this?
Sent from Mail <https://go.microsoft.com/fwlink/?LinkId=550986> for Windows 10
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/? product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Another option where you can assign name-value pairs yourself: Have you looked at patterndb ? Here are a couple snippets that are working well for me: parser p_proxy { db-parser(file("/usr/local/etc/patterndb.d/proxy.xml")); }; destination d_redis { redis ( host("localhost") command("LPUSH", "logstash", "$(format-json type=proxyproxy_time=${PROXY.TIME} proxy_time_taken=${PROXY.TIME_TAKEN} proxy_c_ip=${PROXY.C_IP} proxy_sc_status=${PROXY.SC_STATUS} proxy_s_action=${PROXY.S_ACTION} proxy_sc_bytes=int64(${PROXY.SC_BYTES}) proxy_cs_bytes=int64(${PROXY.CS_BYTES}) proxy_cs_method=${PROXY.CS_METHOD} proxy_cs_uri_scheme=${PROXY.CS_URI_SCHEME} proxy_cs_host=${PROXY.CS_HOST} proxy_cs_uri_port=${PROXY.CS_URI_PORT} proxy_cs_uri_path=${PROXY.CS_URI_PATH} proxy_cs_uri_equery=${PROXY.CS_URI_EQUERY} proxy_cs_username=${PROXY.CS_USERNAME} proxy_cs_auth_group=${PROXY.CS_AUTH__GROUP} proxy_s_supplier_name=${PROXY.S_SUPPLIER_NAME} proxy_content_type=${PROXY.CONTENT_TYPE} proxy_referrer=${PROXY.REFERRER} proxy_user_agent=${PROXY.USER_AGENT} proxy_filter_result=${PROXY.FILTER_RESULT} proxy_cs_categories=${PROXY.CS_CATEGORIES} proxy_x_virus_id=${PROXY.X_VIRUS_ID} proxy_s_ip=${PROXY.S_IP} proxy_any=${PROXY.ANYREST})\n") ); }; log { source(s_network); parser(p_proxy); destination(d_redis); }; Hope this helps. Jim On Sun, Dec 25, 2016 at 9:27 AM, Scheidler, Balázs < balazs.scheidler@balabit.com> wrote:
I would embed json formatted strings as redis list elements.
You can format that using $(format-json)
On Dec 24, 2016 1:17 AM, <johnsc301@gmail.com> wrote:
I am trying to send information from Syslog-ng to Redis. In /etc/syslog-ng/syslog-ng.conf I added this: destination d_redis { redis( host("127.0.0.1") port(6379) command("RPUSH", "sensor_name", "${sensor_name}")); };
I am trying to create a list of variables, hopefully one being timestamp. Here is an example of my syslog that I am trying to pull: http://pastebin.com/Hx5vW4VA
Here's is syslog-ng.conf, for reference: http://pastebin.com/2VQFBNmK
Those are logs being sent from Snort to Syslog-ng through Snort. I want to connect to Redis.
I saw that the command parameters are: comma-separated list of strings ("<redis-command>", "<first-command-parameter>", "<second-command-parameter>", "<third-command-parameter>") from: https://www.balabit.com/documents/syslog-ng-ose-latest -guides/en/syslog-ng-ose-guide-admin/html/reference- destination-redis.html
I'm assuming I'd say RPUSH <something>... However, I am unsure of how to find the correct parameters.
Specifically, for now, I want to create a list (RPUSH) of timestamps, IP addresses (to and from), and event type (ICMP, for example).
I did find this list of parameters:
https://www.balabit.com/documents/syslog-ng-ose-latest- guides/en/syslog-ng-ose-guide-admin/html/syslog-ng-parameter-index.html
A good first try, I'd like to make a list of timestamps. How can I set the d_redis(command()) within syslog-ng.conf to do this?
Sent from Mail <https://go.microsoft.com/fwlink/?LinkId=550986> for Windows 10
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product= syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/? product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Would there be a string that specific provides timestamp or date?
On Dec 26, 2016, at 8:44 AM, Jim Hendrick <james.r.hendrick@gmail.com> wrote:
Another option where you can assign name-value pairs yourself: Have you looked at patterndb ? Here are a couple snippets that are working well for me:
parser p_proxy { db-parser(file("/usr/local/etc/patterndb.d/proxy.xml")); };
destination d_redis { redis ( host("localhost") command("LPUSH", "logstash", "$(format-json type=proxyproxy_time=${PROXY.TIME} proxy_time_taken=${PROXY.TIME_TAKEN} proxy_c_ip=${PROXY.C_IP} proxy_sc_status=${PROXY.SC_STATUS} proxy_s_action=${PROXY.S_ACTION} proxy_sc_bytes=int64(${PROXY.SC_BYTES}) proxy_cs_bytes=int64(${PROXY.CS_BYTES}) proxy_cs_method=${PROXY.CS_METHOD} proxy_cs_uri_scheme=${PROXY.CS_URI_SCHEME} proxy_cs_host=${PROXY.CS_HOST} proxy_cs_uri_port=${PROXY.CS_URI_PORT} proxy_cs_uri_path=${PROXY.CS_URI_PATH} proxy_cs_uri_equery=${PROXY.CS_URI_EQUERY} proxy_cs_username=${PROXY.CS_USERNAME} proxy_cs_auth_group=${PROXY.CS_AUTH__GROUP} proxy_s_supplier_name=${PROXY.S_SUPPLIER_NAME} proxy_content_type=${PROXY.CONTENT_TYPE} proxy_referrer=${PROXY.REFERRER} proxy_user_agent=${PROXY.USER_AGENT} proxy_filter_result=${PROXY.FILTER_RESULT} proxy_cs_categories=${PROXY.CS_CATEGORIES} proxy_x_virus_id=${PROXY.X_VIRUS_ID} proxy_s_ip=${PROXY.S_IP} proxy_any=${PROXY.ANYREST})\n") ); }; log { source(s_network); parser(p_proxy); destination(d_redis); };
Hope this helps.
Jim
On Sun, Dec 25, 2016 at 9:27 AM, Scheidler, Balázs <balazs.scheidler@balabit.com <mailto:balazs.scheidler@balabit.com>> wrote: I would embed json formatted strings as redis list elements.
You can format that using $(format-json)
On Dec 24, 2016 1:17 AM, <johnsc301@gmail.com <mailto:johnsc301@gmail.com>> wrote: I am trying to send information from Syslog-ng to Redis. In /etc/syslog-ng/syslog-ng.conf I added this: destination d_redis { redis( host("127.0.0.1") port(6379) command("RPUSH", "sensor_name", "${sensor_name}")); };
I am trying to create a list of variables, hopefully one being timestamp. Here is an example of my syslog that I am trying to pull: http://pastebin.com/Hx5vW4VA <http://pastebin.com/Hx5vW4VA> Here's is syslog-ng.conf, for reference: http://pastebin.com/2VQFBNmK <http://pastebin.com/2VQFBNmK> Those are logs being sent from Snort to Syslog-ng through Snort. I want to connect to Redis. I saw that the command parameters are: comma-separated list of strings ("<redis-command>", "<first-command-parameter>", "<second-command-parameter>", "<third-command-parameter>") from: https://www.balabit.com/documents/syslog-ng-ose-latest-guides/en/syslog-ng-o... <https://www.balabit.com/documents/syslog-ng-ose-latest-guides/en/syslog-ng-ose-guide-admin/html/reference-destination-redis.html> I'm assuming I'd say RPUSH <something>... However, I am unsure of how to find the correct parameters.
Specifically, for now, I want to create a list (RPUSH) of timestamps, IP addresses (to and from), and event type (ICMP, for example). I did find this list of parameters: https://www.balabit.com/documents/syslog-ng-ose-latest-guides/en/syslog-ng-o... <https://www.balabit.com/documents/syslog-ng-ose-latest-guides/en/syslog-ng-ose-guide-admin/html/syslog-ng-parameter-index.html> A good first try, I'd like to make a list of timestamps. How can I set the d_redis(command()) within syslog-ng.conf to do this?
Sent from Mail <https://go.microsoft.com/fwlink/?LinkId=550986> for Windows 10
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng <https://lists.balabit.hu/mailman/listinfo/syslog-ng> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng <http://www.balabit.com/support/documentation/?product=syslog-ng> FAQ: http://www.balabit.com/wiki/syslog-ng-faq <http://www.balabit.com/wiki/syslog-ng-faq>
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng <https://lists.balabit.hu/mailman/listinfo/syslog-ng> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng <http://www.balabit.com/support/documentation/?product=syslog-ng> FAQ: http://www.balabit.com/wiki/syslog-ng-faq <http://www.balabit.com/wiki/syslog-ng-faq>
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
participants (4)
-
Jim Hendrick
-
John Cosentino
-
johnsc301＠gmail.com
-
Scheidler, Balázs