Python external program destination and line buffer
Hi, i wrote a program destination (in python) that parse a message and insert data in redis. My code is: import re import fileinput import redis r = redis.Redis(host="127.0.0.1", port=6379, db=0) for line in fileinput.input(): tid = re.search('X-TransactionId: (.+?) ', line) #print tid.group(1) regexp_result = re.findall(" Recipient: ([A-z0-9._%+-]+@[A-z0-9.-]+)", line) for email in regexp_result: r.sadd(tid.group(1)+"_recs", email) I configure my destination in this way: destination python_to_file { program("/usr/bin/python /etc/syslog-ng/python/Ric_acc_dest.py" flush-lines(1)); }; It works only if i send many message. If i send only one message it doesn't insert nothing in redis. Any ideas? --
If i do strace on python program i see only read: read(0, "Oct 21 18:34:12 suse11/localhost"..., 8192) = 611 read(0, "Oct 21 18:34:12 suse11/localhost"..., 4096) = 611 read(0, "Oct 21 18:34:12 suse11/localhost"..., 4096) = 611 read(0, "Oct 21 18:34:12 suse11/localhost"..., 4096) = 611 read(0, "Oct 21 18:34:12 suse11/localhost"..., 4096) = 611 read(0, "Oct 21 18:34:12 suse11/localhost"..., 4096) = 611 read(0, "Oct 21 18:34:12 suse11/localhost"..., 4096) = 611 read(0, "Oct 21 18:34:12 suse11/localhost"..., 4096) = 611 read(0, "Oct 21 18:34:12 suse11/localhost"..., 4096) = 611 read(0, "Oct 21 18:34:12 suse11/localhost"..., 4096) = 611 read(0, "Oct 21 18:34:12 suse11/localhost"..., 4096) = 611 read(0, "Oct 21 18:34:12 suse11/localhost"..., 4096) = 611 read(0, "Oct 21 18:34:12 suse11/localhost"..., 4096) = 611 read(0, "Oct 21 18:34:12 suse11/localhost"..., 4096) = 611 Il 21/10/2015 19:54, Giovanni Mancuso ha scritto:
Hi, i wrote a program destination (in python) that parse a message and insert data in redis.
My code is: import re import fileinput import redis
r = redis.Redis(host="127.0.0.1", port=6379, db=0)
for line in fileinput.input(): tid = re.search('X-TransactionId: (.+?) ', line) #print tid.group(1) regexp_result = re.findall(" Recipient: ([A-z0-9._%+-]+@[A-z0-9.-]+)", line) for email in regexp_result: r.sadd(tid.group(1)+"_recs", email)
I configure my destination in this way: destination python_to_file { program("/usr/bin/python /etc/syslog-ng/python/Ric_acc_dest.py" flush-lines(1)); };
It works only if i send many message. If i send only one message it doesn't insert nothing in redis.
Any ideas?
--
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
-- *Giovanni Mancuso* System Architect *T* 06.9826.9600 *M* +39.340.65.80.739 *F* 06.9826.9680 P.zza S.Benedetto da Norcia, 33 - 00071 Pomezia (RM) Par-Tec S.p.A. <http://www.par-tec.it> Web Site <http://www.par-tec.it> info@par-tec.it <mailto:info@par-tec.it> Pagina Facebook <https://www.facebook.com/ParTecSpA> Profilo Twitter <https://twitter.com/partecspa> Pagina LinkedIn <https://www.linkedin.com/company/par-tec/> Canale YouTube <https://www.youtube.com/user/ParTecSpA> CONFIDENZIALE: Questo messaggio ed i suoi allegati sono di carattere confidenziale per i destinatari in indirizzo. È vietato l'inoltro non autorizzato a destinatari diversi da quelli indicati nel messaggio originale. Se ricevuto per errore, l'uso del contenuto è proibito; si prega di comunicarlo al mittente e cancellarlo immediatamente.
Hi Giovanni, the python fileinput module does have a read buffer, that is very handy under heavy load, but for testing you might be better off with sys.stdin.readline() Best regards, Balint On 10/21/2015 07:54 PM, Giovanni Mancuso wrote:
Hi, i wrote a program destination (in python) that parse a message and insert data in redis.
My code is: import re import fileinput import redis
r = redis.Redis(host="127.0.0.1", port=6379, db=0)
for line in fileinput.input(): tid = re.search('X-TransactionId: (.+?) ', line) #print tid.group(1) regexp_result = re.findall(" Recipient: ([A-z0-9._%+-]+@[A-z0-9.-]+)", line) for email in regexp_result: r.sadd(tid.group(1)+"_recs", email)
I configure my destination in this way: destination python_to_file { program("/usr/bin/python /etc/syslog-ng/python/Ric_acc_dest.py" flush-lines(1)); };
It works only if i send many message. If i send only one message it doesn't insert nothing in redis.
Any ideas?
--
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
participants (2)
-
Balint Kovacs
-
Giovanni Mancuso