- syslog-ng - lists.balabit.hu

Re: [syslog-ng] Unbale to mount syslog-ng.conf to /etc/syslog-ng using docker image
by László Várady 27 May '18

27 May '18

Hi, Thanks. It might be possible that you have SELinux enabled on your system. Give it another try using the :Z flag: docker run -it -v /demo/syslog-ng/syslog-ng.conf:/etc/syslog-ng/syslog-ng.conf*:Z* --name syslog-ng -p 514:514 -p 601:601 balabit/syslog-ng:latest Please note that you are using the latest edge version of Docker with the devicemapper storage engine. It has nothing to do with your issue, but I would consider switching to the stable version (with overlay2). https://docs.docker.com/install/linux/docker-ce/centos/ -- László Várady On Sun, May 27, 2018 at 12:56 PM Mujeeb Baig <baig.mujeeb(a)gmail.com> wrote: > Hi, > > Docker info output > > root@hdata3:/home/smb/code/conf# docker info > Containers: 0 > Running: 0 > Paused: 0 > Stopped: 0 > Images: 0 > Server Version: 18.05.0-ce > Storage Driver: devicemapper > Pool Name: docker-8:1-100670142-pool > Pool Blocksize: 65.54kB > Base Device Size: 10.74GB > Backing Filesystem: xfs > Udev Sync Supported: true > Data file: /dev/loop0 > Metadata file: /dev/loop1 > Data loop file: /var/lib/docker/devicemapper/devicemapper/data > Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata > Data Space Used: 11.73MB > Data Space Total: 107.4GB > Data Space Available: 51.36GB > Metadata Space Used: 581.6kB > Metadata Space Total: 2.147GB > Metadata Space Available: 2.147GB > Thin Pool Minimum Free Space: 10.74GB > Deferred Removal Enabled: true > Deferred Deletion Enabled: true > Deferred Deleted Device Count: 0 > Library Version: 1.02.146-RHEL7 (2018-01-22) > Logging Driver: json-file > Cgroup Driver: cgroupfs > Plugins: > Volume: local > Network: bridge host macvlan null overlay > Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk > syslog > Swarm: inactive > Runtimes: runc > Default Runtime: runc > Init Binary: docker-init > containerd version: 773c489c9c1b21a6d78b5c538cd395416ec50f88 > runc version: 4fc53a81fb7c994640722ac585fa9ca548971871 > init version: 949e6fa > Security Options: > seccomp > Profile: default > Kernel Version: 3.10.0-693.21.1.el7.x86_64 > Operating System: CentOS Linux 7 (Core) > OSType: linux > Architecture: x86_64 > CPUs: 2 > Total Memory: 7.147GiB > Name: api-es-syslog-ng-2 > ID: HOBZ:AVPN:MRVO:JAMW:YZFU:SF5M:LTFF:7H7Y:BKBA:MNV4:OMY3:4IJM > Docker Root Dir: /var/lib/docker > Debug Mode (client): false > Debug Mode (server): false > Registry: https://index.docker.io/v1/ > Labels: > provider=google > Experimental: false > Insecure Registries: > 127.0.0.0/8 > Live Restore Enabled: false > > WARNING: devicemapper: usage of loopback devices is strongly discouraged > for production use. > Use `--storage-opt dm.thinpooldev` to specify a custom block > storage device. > > > On 25 May 2018 at 18:21, László Várady <laszlo.varady(a)balabit.com> wrote: > >> Hi, >> >> Thanks. >> >> Could you share what version of Docker you are using? >> The output of "docker version" and "docker info" might be interesting. >> >> -- >> László Várady >> >> On Fri, May 25, 2018 at 6:41 PM Mujeeb Baig <baig.mujeeb(a)gmail.com> >> wrote: >> >>> Hi Laci >>> >>> Here is the output, this time I created another folder /demo/syslog-ng >>> and copied the syslog-ng.conf file there, but same issue. I had also >>> attached the output of >>> docker volume inspect syslog-ng (see attcahment) >>> By the way thanks for your help so far. >>> >>> root@hdata3:/home/smb/code/conf# docker rmi balabit/syslog-ng:latest >>> Untagged: balabit/syslog-ng:latest >>> Untagged: balabit/syslog-ng@sha256 >>> :55ed29d3f68e6ce5c0494580af0b49bdfbbd83232902d9db0596486dbc2f0d3f >>> Deleted: >>> sha256:aea40146388e85ca11d2b8b4ee7877d2939a71bed41b727431d48b21084967e1 >>> Deleted: >>> sha256:b97aefa7a738770d348b4c7b5ba8d6cfcaada9453e41603db7b57b01a6bf6773 >>> Deleted: >>> sha256:11aa7498ed3edbf74eea2ee5de418fb8789055e278e750d1db5e4fe9057cada4 >>> Deleted: >>> sha256:a15ea7dac472320a77dfffe1066680e7dfd46eea735376a5b8a2040f4d0ca57e >>> Deleted: >>> sha256:f2f7b467bea911acac35d8a24c3b14cf163bb00370209f5214e5950813d45686 >>> Deleted: >>> sha256:42a1952fea491565f3b8fb19d5869576c8d12a9192f867c4b72234349fa9059a >>> Deleted: >>> sha256:e30891f348eea2928d4027527455904b5933426cbbef4be4b9a908c97a09ba68 >>> Deleted: >>> sha256:ffc4c11463ee21b7532b63abd6079393c619a5d0f4b00397a4b9d1cf9efc4d9b >>> root@hdata3:/home/smb/code/conf# >>> root@hdata3:/home/smb/code/conf# docker pull balabit/syslog-ng >>> Using default tag: latest >>> latest: Pulling from balabit/syslog-ng >>> 4176fe04cefe: Pull complete >>> af68cba418f8: Pull complete >>> 9b2b4fba2935: Pull complete >>> 6297af76ec77: Pull complete >>> 7bf2bd8c0da2: Pull complete >>> 85f1f826e469: Pull complete >>> 24a506d1872d: Pull complete >>> Digest: >>> sha256:55ed29d3f68e6ce5c0494580af0b49bdfbbd83232902d9db0596486dbc2f0d3f >>> Status: Downloaded newer image for balabit/syslog-ng:latest >>> root@hdata3:/home/smb/code/conf# >>> root@hdata3:/home/smb/code/conf# >>> root@hdata3:/home/smb/code/conf# docker run -it -v >>> /demo/syslog-ng/syslog-ng.conf:/etc/syslog-ng/syslog-ng.conf --name >>> syslog-ng -p 514:514 -p 601:601 balabit/syslog-ng:latest >>> docker: Error response from daemon: OCI runtime create failed: >>> container_linux.go:348: starting container process caused >>> "process_linux.go:402: container init caused \"rootfs_linux.go:58: mounting >>> \\\"/demo/syslog-ng/syslog-ng.conf\\\" to rootfs >>> \\\"/var/lib/docker/devicemapper/mnt/4751fd8ef1ddb782156a57749429486610b6c3d532a77c6ebf0727ca72ea451a/rootfs\\\" >>> at >>> \\\"/var/lib/docker/devicemapper/mnt/4751fd8ef1ddb782156a57749429486610b6c3d532a77c6ebf0727ca72ea451a/rootfs/etc/syslog-ng/syslog-ng.conf\\\" >>> caused \\\"not a directory\\\"\"": unknown: Are you trying to mount a >>> directory onto a file (or vice-versa)? Check if the specified host path >>> exists and is the expected type. >>> root@hdata3:/home/smb/code/conf# >>> root@hdata3:/home/smb/code/conf# docker volume ls >>> DRIVER VOLUME NAME >>> root@hdata3:/home/smb/code/conf# >>> >>> >>> On 25 May 2018 at 17:01, László Várady <laszlo.varady(a)balabit.com> >>> wrote: >>> >>>> Hi Mujeeb, >>>> >>>> Something weird happened when you executed >>>> >>>> docker run -it -v "$PWD"/syslog-ng.conf:/etc/syslog-ng/syslog-ng.conf >>>> --name syslog-ng -p 514:514 -p 601:601 balabit/syslog-ng:latest >>>> >>>> for the first time, so I would like to ask you to start from a clean >>>> environment. >>>> >>>> - You've made sure that /home/smb/code/conf/syslog-ng.conf exists on >>>> your host and it is a file. >>>> >>>> - In the original balabit/syslog-ng image, >>>> /etc/syslog-ng/syslog-ng.conf is also a file, so the mount should work. >>>> >>>> Please run the following commands: >>>> >>>> 1. docker rm syslog-ng >>>> >>>> 2. docker rmi balabit/syslog-ng >>>> >>>> 3. docker pull balabit/syslog-ng >>>> >>>> And finally, what you actually wanted to run: >>>> >>>> 4. docker run -it -v /home/smb/code/conf/syslog-ng.conf:/etc/syslog-ng/syslog-ng.conf >>>> --name syslog-ng -p 514:514 -p 601:601 balabit/syslog-ng:latest >>>> >>>> If you still experience problems after this, please send us the output >>>> of "docker volume ls". >>>> >>>> One more question: >>>> Is it possible that "/home/smb/code/conf/" is a mounted network >>>> filesystem (NFS, SMB)? >>>> >>>> -- >>>> László Várady >>>> >>>> On Fri, May 25, 2018 at 5:33 PM Mujeeb Baig <baig.mujeeb(a)gmail.com> >>>> wrote: >>>> >>>>> Hi Laci, >>>>> >>>>> It runs fine >>>>> root@hdata3:/home/smb/code/conf# file >>>>> /home/smb/code/conf/syslog-ng.conf >>>>> /home/smb/code/conf/syslog-ng.conf: ASCII text >>>>> >>>>> Regards >>>>> Mujeeb >>>>> >>>>> On 25 May 2018 at 16:20, Szemere, László <laszlo.szemere(a)balabit.com> >>>>> wrote: >>>>> >>>>>> Hello, >>>>>> >>>>>> Thank you for the output. Can you run the >>>>>> >>>>>> *file /home/smb/code/conf/syslog-ng.conf* >>>>>> >>>>>> command on your HOST machine? >>>>>> >>>>>> >>>>>> Br, >>>>>> Laci >>>>>> >>>>>> >>>>>> On Fri, May 25, 2018 at 5:04 PM, Mujeeb Baig <baig.mujeeb(a)gmail.com> >>>>>> wrote: >>>>>> >>>>>>> It is locating the source correctly, this is what I see under >>>>>>> 'Mounts' section of inspect >>>>>>> >>>>>>> "Mounts": [ >>>>>>> { >>>>>>> "Type": "bind", >>>>>>> "Source": "/home/smb/code/conf/syslog-ng.conf", >>>>>>> "Destination": "/1234", >>>>>>> "Mode": "", >>>>>>> "RW": true, >>>>>>> "Propagation": "rprivate" >>>>>>> } >>>>>>> ], >>>>>>> >>>>>>> >>>>>>> On 25 May 2018 at 15:57, Szemere, László <laszlo.szemere(a)balabit.com >>>>>>> > wrote: >>>>>>> >>>>>>>> Hello, >>>>>>>> >>>>>>>> From the Docker documentation: >>>>>>>> https://docs.docker.com/v17.09/engine/admin/volumes/bind-mounts/#difference… >>>>>>>> >>>>>>>> If you use -v or --volume to bind-mount a file or directory that >>>>>>>>> does not yet exist on the Docker host, -v will create the >>>>>>>>> endpoint for you. >>>>>>>>> *It is always created as a directory.*If you use --mount to >>>>>>>>> bind-mount a file or directory that does not yet exist on the Docker host, >>>>>>>>> Docker does *not* automatically create it for you, but generates >>>>>>>>> an error. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> It looks like docker daemon still can not find your file >>>>>>>> *"$PWD"/syslog-ng.conf* and creates a directory instead. >>>>>>>> Instead of heavy debugging, I suggest to run *docker inspect >>>>>>>> syslog-ng* to see what path was actually mounted by the docker >>>>>>>> daemon. You should see something like this in the output: >>>>>>>> >>>>>>>> "Mounts": [ >>>>>>>>> { >>>>>>>>> "Type": "bind", >>>>>>>>> "Source": "XXX/syslog-ng.conf", >>>>>>>>> "Destination": "/1234", >>>>>>>>> "Mode": "", >>>>>>>>> "RW": true, >>>>>>>>> "Propagation": "rprivate" >>>>>>>>> } >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> Once we have the actual mounted path, it will be easier to figure >>>>>>>> out why the docker daemon can not find your original config file. >>>>>>>> >>>>>>>> Br, >>>>>>>> Laci >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On Fri, May 25, 2018 at 4:19 PM, Mujeeb Baig <baig.mujeeb(a)gmail.com >>>>>>>> > wrote: >>>>>>>> >>>>>>>>> Hi Andrew, >>>>>>>>> >>>>>>>>> Thanks for your response. Yes the path is correct >>>>>>>>> >>>>>>>>> When I execute with host location as /1234 it creates the >>>>>>>>> container, but nothing inside /1234 folder >>>>>>>>> >>>>>>>>> root@hdata3:/home/smb/code/conf# docker run -it -v >>>>>>>>> "$PWD"/syslog-ng.conf:/1234 --name syslog-ng -p 514:514 -p 601:601 >>>>>>>>> balabit/syslog-ng:latest >>>>>>>>> syslog-ng: Error setting capabilities, capability management >>>>>>>>> disabled; error='Operation not permitted' >>>>>>>>> [2018-05-25T14:12:15.695588] WARNING: Configuration file format is >>>>>>>>> too old, syslog-ng is running in compatibility mode. Please update it to >>>>>>>>> use the syslog-ng 3.14 format at your time of convenience. To upgrade the >>>>>>>>> configuration, please review the warnings about incompatible changes >>>>>>>>> printed by syslog-ng, and once completed change the @version header at the >>>>>>>>> top of the configuration file.; >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> I can see inside the container that /etc/syslog-ng/ this folder >>>>>>>>> is not updated >>>>>>>>> >>>>>>>>> root@hdata3:/home/smb# docker exec -ti syslog-ng /bin/bash >>>>>>>>> root@d634d32aee4c:/# >>>>>>>>> root@d634d32aee4c:/# ls -l /1234/ >>>>>>>>> total 0 >>>>>>>>> root@d634d32aee4c:/# >>>>>>>>> root@d634d32aee4c:/# ls -l /sy/etcslog-ng/ >>>>>>>>> conf.d/ patterndb.d/ scl.conf syslog-ng.conf >>>>>>>>> root@d634d32aee4c:/# ls -lrt /etc/syslog-ng/ >>>>>>>>> total 12 >>>>>>>>> -rw-r--r--. 1 root root 5910 Feb 23 13:17 syslog-ng.conf >>>>>>>>> drwxr-xr-x. 2 root root 6 Mar 2 12:58 conf.d >>>>>>>>> drwxr-xr-x. 2 root root 6 Mar 2 12:58 patterndb.d >>>>>>>>> -rw-r--r--. 1 root root 1336 Mar 2 12:58 scl.conf >>>>>>>>> root@d634d32aee4c:/# >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> On 25 May 2018 at 13:31, Mitzki, András <andras.mitzki(a)balabit.com >>>>>>>>> > wrote: >>>>>>>>> >>>>>>>>>> Hi Mujeeb, >>>>>>>>>> >>>>>>>>>> Maybe the problem is with the config file path: >>>>>>>>>> "/home/smb/code/conf/syslog-ng.conf" >>>>>>>>>> Could you check that path is correct? >>>>>>>>>> >>>>>>>>>> Regards >>>>>>>>>> Andrew >>>>>>>>>> >>>>>>>>>> On Fri, May 25, 2018 at 1:13 PM, Mujeeb Baig < >>>>>>>>>> baig.mujeeb(a)gmail.com> wrote: >>>>>>>>>> >>>>>>>>>>> Hi >>>>>>>>>>> >>>>>>>>>>> This is my first trial of syslog-ng. I've tried to start >>>>>>>>>>> syslog-ng just like described on the official bolg page >>>>>>>>>>> https://syslog-ng.com/blog/central-log-server-docker/ >>>>>>>>>>> https://syslog-ng.com/blog/collecting-docker-infrastructure-logs-using-sysl… >>>>>>>>>>> >>>>>>>>>>> I am starting docker container as: >>>>>>>>>>> >>>>>>>>>>> docker run -it -v >>>>>>>>>>> "$PWD"/syslog-ng.conf:/etc/syslog-ng/syslog-ng.conf --name syslog-ng -p >>>>>>>>>>> 514:514 -p 601:601 balabit/syslog-ng:latest >>>>>>>>>>> >>>>>>>>>>> But unfortunately it is failing to mount my custom (local) >>>>>>>>>>> configuration file from source to host, complaining about mounting a >>>>>>>>>>> directory onto file >>>>>>>>>>> >>>>>>>>>>> Error: >>>>>>>>>>> >>>>>>>>>>> docker: Error response from daemon: OCI runtime create failed: >>>>>>>>>>> container_linux.go:348: starting container process caused >>>>>>>>>>> "process_linux.go:402: container init caused \"rootfs_linux.go:58: mounting >>>>>>>>>>> \\"/home/smb/code/conf/syslog-ng.conf\\" to rootfs >>>>>>>>>>> \\"/var/lib/docker/devicemapper/mnt/52d32f854a030b396b03e7596ab5d71eb1a18f34a09d5e4997c437568749b259/rootfs\\" >>>>>>>>>>> at >>>>>>>>>>> \\"/var/lib/docker/devicemapper/mnt/52d32f854a030b396b03e7596ab5d71eb1a18f34a09d5e4997c437568749b259/rootfs/etc/syslog-ng/syslog-ng.conf\\" >>>>>>>>>>> caused \\"not a directory\\"\"": unknown: Are you trying to mount a >>>>>>>>>>> directory onto a file (or vice-versa)? Check if the specified host path >>>>>>>>>>> exists and is the expected type. >>>>>>>>>>> >>>>>>>>>>> Am I missing something? >>>>>>>>>>> >>>>>>>>>>> Please let me know >>>>>>>>>>> >>>>>>>>>>> Many Thanks >>>>>>>>>>> >>>>>>>>>>> Mujeeb >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> ______________________________________________________________________________ >>>>>>>>>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng >>>>>>>>>>> Documentation: >>>>>>>>>>> http://www.balabit.com/support/documentation/?product=syslog-ng >>>>>>>>>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> ______________________________________________________________________________ >>>>>>>>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng >>>>>>>>>> Documentation: >>>>>>>>>> http://www.balabit.com/support/documentation/?product=syslog-ng >>>>>>>>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> ______________________________________________________________________________ >>>>>>>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng >>>>>>>>> Documentation: >>>>>>>>> http://www.balabit.com/support/documentation/?product=syslog-ng >>>>>>>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> ______________________________________________________________________________ >>>>>>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng >>>>>>>> Documentation: >>>>>>>> http://www.balabit.com/support/documentation/?product=syslog-ng >>>>>>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> >>>>>>> ______________________________________________________________________________ >>>>>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng >>>>>>> Documentation: >>>>>>> http://www.balabit.com/support/documentation/?product=syslog-ng >>>>>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq >>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> ______________________________________________________________________________ >>>>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng >>>>>> Documentation: >>>>>> http://www.balabit.com/support/documentation/?product=syslog-ng >>>>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq >>>>>> >>>>>> >>>>>> >>>>> >>>>> ______________________________________________________________________________ >>>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng >>>>> Documentation: >>>>> http://www.balabit.com/support/documentation/?product=syslog-ng >>>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq >>>>> >>>>> >>> >

1 0

Re: [syslog-ng] Unbale to mount syslog-ng.conf to /etc/syslog-ng using docker image
by László Várady 25 May '18

25 May '18

Hi, Thanks. Could you share what version of Docker you are using? The output of "docker version" and "docker info" might be interesting. -- László Várady On Fri, May 25, 2018 at 6:41 PM Mujeeb Baig <baig.mujeeb(a)gmail.com> wrote: > Hi Laci > > Here is the output, this time I created another folder /demo/syslog-ng and > copied the syslog-ng.conf file there, but same issue. I had also attached > the output of > docker volume inspect syslog-ng (see attcahment) > By the way thanks for your help so far. > > root@hdata3:/home/smb/code/conf# docker rmi balabit/syslog-ng:latest > Untagged: balabit/syslog-ng:latest > Untagged: balabit/syslog-ng@sha256 > :55ed29d3f68e6ce5c0494580af0b49bdfbbd83232902d9db0596486dbc2f0d3f > Deleted: > sha256:aea40146388e85ca11d2b8b4ee7877d2939a71bed41b727431d48b21084967e1 > Deleted: > sha256:b97aefa7a738770d348b4c7b5ba8d6cfcaada9453e41603db7b57b01a6bf6773 > Deleted: > sha256:11aa7498ed3edbf74eea2ee5de418fb8789055e278e750d1db5e4fe9057cada4 > Deleted: > sha256:a15ea7dac472320a77dfffe1066680e7dfd46eea735376a5b8a2040f4d0ca57e > Deleted: > sha256:f2f7b467bea911acac35d8a24c3b14cf163bb00370209f5214e5950813d45686 > Deleted: > sha256:42a1952fea491565f3b8fb19d5869576c8d12a9192f867c4b72234349fa9059a > Deleted: > sha256:e30891f348eea2928d4027527455904b5933426cbbef4be4b9a908c97a09ba68 > Deleted: > sha256:ffc4c11463ee21b7532b63abd6079393c619a5d0f4b00397a4b9d1cf9efc4d9b > root@hdata3:/home/smb/code/conf# > root@hdata3:/home/smb/code/conf# docker pull balabit/syslog-ng > Using default tag: latest > latest: Pulling from balabit/syslog-ng > 4176fe04cefe: Pull complete > af68cba418f8: Pull complete > 9b2b4fba2935: Pull complete > 6297af76ec77: Pull complete > 7bf2bd8c0da2: Pull complete > 85f1f826e469: Pull complete > 24a506d1872d: Pull complete > Digest: > sha256:55ed29d3f68e6ce5c0494580af0b49bdfbbd83232902d9db0596486dbc2f0d3f > Status: Downloaded newer image for balabit/syslog-ng:latest > root@hdata3:/home/smb/code/conf# > root@hdata3:/home/smb/code/conf# > root@hdata3:/home/smb/code/conf# docker run -it -v > /demo/syslog-ng/syslog-ng.conf:/etc/syslog-ng/syslog-ng.conf --name > syslog-ng -p 514:514 -p 601:601 balabit/syslog-ng:latest > docker: Error response from daemon: OCI runtime create failed: > container_linux.go:348: starting container process caused > "process_linux.go:402: container init caused \"rootfs_linux.go:58: mounting > \\\"/demo/syslog-ng/syslog-ng.conf\\\" to rootfs > \\\"/var/lib/docker/devicemapper/mnt/4751fd8ef1ddb782156a57749429486610b6c3d532a77c6ebf0727ca72ea451a/rootfs\\\" > at > \\\"/var/lib/docker/devicemapper/mnt/4751fd8ef1ddb782156a57749429486610b6c3d532a77c6ebf0727ca72ea451a/rootfs/etc/syslog-ng/syslog-ng.conf\\\" > caused \\\"not a directory\\\"\"": unknown: Are you trying to mount a > directory onto a file (or vice-versa)? Check if the specified host path > exists and is the expected type. > root@hdata3:/home/smb/code/conf# > root@hdata3:/home/smb/code/conf# docker volume ls > DRIVER VOLUME NAME > root@hdata3:/home/smb/code/conf# > > > On 25 May 2018 at 17:01, László Várady <laszlo.varady(a)balabit.com> wrote: > >> Hi Mujeeb, >> >> Something weird happened when you executed >> >> docker run -it -v "$PWD"/syslog-ng.conf:/etc/syslog-ng/syslog-ng.conf >> --name syslog-ng -p 514:514 -p 601:601 balabit/syslog-ng:latest >> >> for the first time, so I would like to ask you to start from a clean >> environment. >> >> - You've made sure that /home/smb/code/conf/syslog-ng.conf exists on your >> host and it is a file. >> >> - In the original balabit/syslog-ng image, /etc/syslog-ng/syslog-ng.conf >> is also a file, so the mount should work. >> >> Please run the following commands: >> >> 1. docker rm syslog-ng >> >> 2. docker rmi balabit/syslog-ng >> >> 3. docker pull balabit/syslog-ng >> >> And finally, what you actually wanted to run: >> >> 4. docker run -it -v /home/smb/code/conf/syslog-ng.conf:/etc/syslog-ng/syslog-ng.conf >> --name syslog-ng -p 514:514 -p 601:601 balabit/syslog-ng:latest >> >> If you still experience problems after this, please send us the output of >> "docker volume ls". >> >> One more question: >> Is it possible that "/home/smb/code/conf/" is a mounted network >> filesystem (NFS, SMB)? >> >> -- >> László Várady >> >> On Fri, May 25, 2018 at 5:33 PM Mujeeb Baig <baig.mujeeb(a)gmail.com> >> wrote: >> >>> Hi Laci, >>> >>> It runs fine >>> root@hdata3:/home/smb/code/conf# file /home/smb/code/conf/syslog-ng.conf >>> /home/smb/code/conf/syslog-ng.conf: ASCII text >>> >>> Regards >>> Mujeeb >>> >>> On 25 May 2018 at 16:20, Szemere, László <laszlo.szemere(a)balabit.com> >>> wrote: >>> >>>> Hello, >>>> >>>> Thank you for the output. Can you run the >>>> >>>> *file /home/smb/code/conf/syslog-ng.conf* >>>> >>>> command on your HOST machine? >>>> >>>> >>>> Br, >>>> Laci >>>> >>>> >>>> On Fri, May 25, 2018 at 5:04 PM, Mujeeb Baig <baig.mujeeb(a)gmail.com> >>>> wrote: >>>> >>>>> It is locating the source correctly, this is what I see under 'Mounts' >>>>> section of inspect >>>>> >>>>> "Mounts": [ >>>>> { >>>>> "Type": "bind", >>>>> "Source": "/home/smb/code/conf/syslog-ng.conf", >>>>> "Destination": "/1234", >>>>> "Mode": "", >>>>> "RW": true, >>>>> "Propagation": "rprivate" >>>>> } >>>>> ], >>>>> >>>>> >>>>> On 25 May 2018 at 15:57, Szemere, László <laszlo.szemere(a)balabit.com> >>>>> wrote: >>>>> >>>>>> Hello, >>>>>> >>>>>> From the Docker documentation: >>>>>> https://docs.docker.com/v17.09/engine/admin/volumes/bind-mounts/#difference… >>>>>> >>>>>> If you use -v or --volume to bind-mount a file or directory that >>>>>>> does not yet exist on the Docker host, -v will create the endpoint >>>>>>> for you. >>>>>>> *It is always created as a directory.*If you use --mount to >>>>>>> bind-mount a file or directory that does not yet exist on the Docker host, >>>>>>> Docker does *not* automatically create it for you, but generates an >>>>>>> error. >>>>>> >>>>>> >>>>>> >>>>>> It looks like docker daemon still can not find your file >>>>>> *"$PWD"/syslog-ng.conf* and creates a directory instead. >>>>>> Instead of heavy debugging, I suggest to run *docker inspect >>>>>> syslog-ng* to see what path was actually mounted by the docker >>>>>> daemon. You should see something like this in the output: >>>>>> >>>>>> "Mounts": [ >>>>>>> { >>>>>>> "Type": "bind", >>>>>>> "Source": "XXX/syslog-ng.conf", >>>>>>> "Destination": "/1234", >>>>>>> "Mode": "", >>>>>>> "RW": true, >>>>>>> "Propagation": "rprivate" >>>>>>> } >>>>>> >>>>>> >>>>>> >>>>>> Once we have the actual mounted path, it will be easier to figure >>>>>> out why the docker daemon can not find your original config file. >>>>>> >>>>>> Br, >>>>>> Laci >>>>>> >>>>>> >>>>>> >>>>>> On Fri, May 25, 2018 at 4:19 PM, Mujeeb Baig <baig.mujeeb(a)gmail.com> >>>>>> wrote: >>>>>> >>>>>>> Hi Andrew, >>>>>>> >>>>>>> Thanks for your response. Yes the path is correct >>>>>>> >>>>>>> When I execute with host location as /1234 it creates the container, >>>>>>> but nothing inside /1234 folder >>>>>>> >>>>>>> root@hdata3:/home/smb/code/conf# docker run -it -v >>>>>>> "$PWD"/syslog-ng.conf:/1234 --name syslog-ng -p 514:514 -p 601:601 >>>>>>> balabit/syslog-ng:latest >>>>>>> syslog-ng: Error setting capabilities, capability management >>>>>>> disabled; error='Operation not permitted' >>>>>>> [2018-05-25T14:12:15.695588] WARNING: Configuration file format is >>>>>>> too old, syslog-ng is running in compatibility mode. Please update it to >>>>>>> use the syslog-ng 3.14 format at your time of convenience. To upgrade the >>>>>>> configuration, please review the warnings about incompatible changes >>>>>>> printed by syslog-ng, and once completed change the @version header at the >>>>>>> top of the configuration file.; >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> I can see inside the container that /etc/syslog-ng/ this folder is >>>>>>> not updated >>>>>>> >>>>>>> root@hdata3:/home/smb# docker exec -ti syslog-ng /bin/bash >>>>>>> root@d634d32aee4c:/# >>>>>>> root@d634d32aee4c:/# ls -l /1234/ >>>>>>> total 0 >>>>>>> root@d634d32aee4c:/# >>>>>>> root@d634d32aee4c:/# ls -l /sy/etcslog-ng/ >>>>>>> conf.d/ patterndb.d/ scl.conf syslog-ng.conf >>>>>>> root@d634d32aee4c:/# ls -lrt /etc/syslog-ng/ >>>>>>> total 12 >>>>>>> -rw-r--r--. 1 root root 5910 Feb 23 13:17 syslog-ng.conf >>>>>>> drwxr-xr-x. 2 root root 6 Mar 2 12:58 conf.d >>>>>>> drwxr-xr-x. 2 root root 6 Mar 2 12:58 patterndb.d >>>>>>> -rw-r--r--. 1 root root 1336 Mar 2 12:58 scl.conf >>>>>>> root@d634d32aee4c:/# >>>>>>> >>>>>>> >>>>>>> >>>>>>> On 25 May 2018 at 13:31, Mitzki, András <andras.mitzki(a)balabit.com> >>>>>>> wrote: >>>>>>> >>>>>>>> Hi Mujeeb, >>>>>>>> >>>>>>>> Maybe the problem is with the config file path: >>>>>>>> "/home/smb/code/conf/syslog-ng.conf" >>>>>>>> Could you check that path is correct? >>>>>>>> >>>>>>>> Regards >>>>>>>> Andrew >>>>>>>> >>>>>>>> On Fri, May 25, 2018 at 1:13 PM, Mujeeb Baig <baig.mujeeb(a)gmail.com >>>>>>>> > wrote: >>>>>>>> >>>>>>>>> Hi >>>>>>>>> >>>>>>>>> This is my first trial of syslog-ng. I've tried to start syslog-ng >>>>>>>>> just like described on the official bolg page >>>>>>>>> https://syslog-ng.com/blog/central-log-server-docker/ >>>>>>>>> https://syslog-ng.com/blog/collecting-docker-infrastructure-logs-using-sysl… >>>>>>>>> >>>>>>>>> I am starting docker container as: >>>>>>>>> >>>>>>>>> docker run -it -v >>>>>>>>> "$PWD"/syslog-ng.conf:/etc/syslog-ng/syslog-ng.conf --name syslog-ng -p >>>>>>>>> 514:514 -p 601:601 balabit/syslog-ng:latest >>>>>>>>> >>>>>>>>> But unfortunately it is failing to mount my custom (local) >>>>>>>>> configuration file from source to host, complaining about mounting a >>>>>>>>> directory onto file >>>>>>>>> >>>>>>>>> Error: >>>>>>>>> >>>>>>>>> docker: Error response from daemon: OCI runtime create failed: >>>>>>>>> container_linux.go:348: starting container process caused >>>>>>>>> "process_linux.go:402: container init caused \"rootfs_linux.go:58: mounting >>>>>>>>> \\"/home/smb/code/conf/syslog-ng.conf\\" to rootfs >>>>>>>>> \\"/var/lib/docker/devicemapper/mnt/52d32f854a030b396b03e7596ab5d71eb1a18f34a09d5e4997c437568749b259/rootfs\\" >>>>>>>>> at >>>>>>>>> \\"/var/lib/docker/devicemapper/mnt/52d32f854a030b396b03e7596ab5d71eb1a18f34a09d5e4997c437568749b259/rootfs/etc/syslog-ng/syslog-ng.conf\\" >>>>>>>>> caused \\"not a directory\\"\"": unknown: Are you trying to mount a >>>>>>>>> directory onto a file (or vice-versa)? Check if the specified host path >>>>>>>>> exists and is the expected type. >>>>>>>>> >>>>>>>>> Am I missing something? >>>>>>>>> >>>>>>>>> Please let me know >>>>>>>>> >>>>>>>>> Many Thanks >>>>>>>>> >>>>>>>>> Mujeeb >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> ______________________________________________________________________________ >>>>>>>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng >>>>>>>>> Documentation: >>>>>>>>> http://www.balabit.com/support/documentation/?product=syslog-ng >>>>>>>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> ______________________________________________________________________________ >>>>>>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng >>>>>>>> Documentation: >>>>>>>> http://www.balabit.com/support/documentation/?product=syslog-ng >>>>>>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> >>>>>>> ______________________________________________________________________________ >>>>>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng >>>>>>> Documentation: >>>>>>> http://www.balabit.com/support/documentation/?product=syslog-ng >>>>>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq >>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> ______________________________________________________________________________ >>>>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng >>>>>> Documentation: >>>>>> http://www.balabit.com/support/documentation/?product=syslog-ng >>>>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq >>>>>> >>>>>> >>>>>> >>>>> >>>>> >>>>> ______________________________________________________________________________ >>>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng >>>>> Documentation: >>>>> http://www.balabit.com/support/documentation/?product=syslog-ng >>>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq >>>>> >>>>> >>>>> >>>> >>>> >>>> ______________________________________________________________________________ >>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng >>>> Documentation: >>>> http://www.balabit.com/support/documentation/?product=syslog-ng >>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq >>>> >>>> >>>> >>> >>> ______________________________________________________________________________ >>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng >>> Documentation: >>> http://www.balabit.com/support/documentation/?product=syslog-ng >>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq >>> >>> >

1 0

Unbale to mount syslog-ng.conf to /etc/syslog-ng using docker image
by Mujeeb Baig 25 May '18

25 May '18

Hi This is my first trial of syslog-ng. I've tried to start syslog-ng just like described on the official bolg page https://syslog-ng.com/blog/ central-log-server-docker/ https://syslog-ng.com/blog/collecting-docker- infrastructure-logs-using-syslog-ng/ I am starting docker container as: docker run -it -v "$PWD"/syslog-ng.conf:/etc/syslog-ng/syslog-ng.conf --name syslog-ng -p 514:514 -p 601:601 balabit/syslog-ng:latest But unfortunately it is failing to mount my custom (local) configuration file from source to host, complaining about mounting a directory onto file Error: docker: Error response from daemon: OCI runtime create failed: container_linux.go:348: starting container process caused "process_linux.go:402: container init caused \"rootfs_linux.go:58: mounting \\"/home/smb/code/conf/syslog-ng.conf\\" to rootfs \\"/var/lib/docker/ devicemapper/mnt/52d32f854a030b396b03e7596ab5d7 1eb1a18f34a09d5e4997c437568749b259/rootfs\\" at \\"/var/lib/docker/ devicemapper/mnt/52d32f854a030b396b03e7596ab5d7 1eb1a18f34a09d5e4997c437568749b259/rootfs/etc/syslog-ng/syslog-ng.conf\\" caused \\"not a directory\\"\"": unknown: Are you trying to mount a directory onto a file (or vice-versa)? Check if the specified host path exists and is the expected type. Am I missing something? Please let me know Many Thanks Mujeeb

4 7

exclude empty pairs from json-parser()
by Nik Ambrosch 23 May '18

23 May '18

Hi - I'm trying to figure out how to instruct json-parser() to exclude empty (or nonexistent) json pairs from being passed to it's template. This is possible with the csv parser using null("") but not json. It seems that if i specify all my json-parser() pairs using --pair example=\"${MYMACRO}\" the desired behavior is achieved, however using --scope nv-pairs instead will log the empty values.

2 2

syslog-ng Open Source Edition official binary repository
by Gergely Bodnár 23 May '18

23 May '18

Dear syslog-ng users, Last autumn, we asked for your feedback ( https://syslog-ng.com/blog/new-website-and-official-repo-coming-up-tell-us-…) on our plan to introduce official binary repositories for syslog-ng Open Source Edition. The overwhelming majority of those who got back to us agreed that it is fair to provide this service tied to a simple registration. We are happy to announce that although a bit late compared to our originally planned schedule, we are launching this service, first as a test run with Debian Jessie packages. We will always host the latest release, and these packages will be built and tested by us, so you won't have to worry about dependencies - provided you are running a suitable Linux distribution. We are planning to add an RPM repository as well in the future, knowing well that many of our open source users require that. How to use repository? ------------------------------- To use the newly introduced syslog-ng Open Source Edition Debian Jessie repository, you only have to complete a few quick and easy steps. 1. Create a MyBalabit account (https://my.balabit.com/signup) Just click on MyBalabit in the top right corner of the page, and then choose Sign Up (under OS X, the sign up option will be hiding under the hamburger menu). At the top of the form, there will be a drop down menu where you have to specify why are you requesting access to this portal. Choose syslog-ng OSE Binary Repository Access, then fill out the rest of the form, read and agree to our privacy policy, then hit create you account.If you already happen to have a MyBalabit account, simply log in and look for Request Access in the top menu. Once in there, select syslog-ng OSE Binary Repository Access and click on request access. 2. Once you finished with the initial steps, look for the Access tokens option at the bottom of the left menu bar. In there, you will find your token to be used with the syslog-ng OSE DEB repository. If for some reason you need a new token, use the regenerate button. 3. Now you should be all set to configure your compatible Linux distribution to access the repository. # echo "deb http://JXJiI9Znyt:@apt.syslog-ng.com/syslog-ng-ose jessie main" >/etc/apt/sources.list.d/oserepo.list # apt-get update # apt-get install --no-install-recommends -y curl # curl http://JXJiI9Znyt@apt.syslog-ng.com/syslog-ng-ose/pubkey.gpg | apt-key add - # apt-get update # apt-cache policy syslog-ng # apt-get install syslog-ng And that's it, you should be all set! Please, don't forget that this is still very much a trial for us to gauge the interest in such a service. Bugs might surface, the repository might be inaccessible due to technical glitches, etc. We encourage you to share your feedback with us. It would be much appreciated! Regards, Gergely Bodnar -- *Gergely Bodnar* | syslog-ng Product Owner gergely.bodnar(a)balabit.com follow Balabit on: web <http://go.balabit.com/F9ZZ500bh0U013LX000VAE5> | blog <http://go.balabit.com/VZZi000AL90V300U55X1cE0> | linkedin <http://go.balabit.com/CZ100j0d50LU0EAZ3X09V05>| facebook <http://go.balabit.com/I090eUZ0551kEA3X0L0ZV00> | twitter <http://go.balabit.com/c0503X1U05EZ09AVl0ZfL00> *© BalaBit Corp.* Weiser Mazars 135 West 50th Street New York, NY 10020 USA

5 7

Forwarding flat files to a remote syslog-ng server
by Song, Young 22 May '18

22 May '18

Hi, I'm trying to send various flat files to a central syslog-ng server. So, I have the following setups but somehow using the "local1" and "local2" facilities don't work, although no errors when restarting syslog services on both client & server. Any advice would be appreciated much! ======================================================================= My syslog-ng CLIENT (running syslog-ng 3.6.2): source s_file1 { file("/var/log/syslog/file1.log" follow-freq(1) flags(no-parse) program_override("audit")); }; filter f_file1 { facility(local1); }; destination d_file1 { syslog("10.10.10.10" transport("udp") port(514)); }; log { source(s_file1); filter(f_file1); destination(d_file1); }; source s_file2 { file("/var/log/syslog/file2.log" follow-freq(1) flags(no-parse) program_override("audit")); }; filter f_file2 { facility(local2); }; destination d_file2 { syslog("10.10.10.10" transport("udp") port(514)); }; log { source(s_file2); filter(f_file2); destination(d_file2); }; ======================================================================== My syslog-ng SERVER (10.10.10.10 - running syslog-ng 2.0.9): destination d_file1 { file ("/var/log/syslogs/file1-$YEAR$MONTH$DAY.log" owner(root) group(root) perm(0644) dir_perm(0755) create_dirs(yes)); }; filter f_file1 { facility(local1); }; log { source(remote); filter(f_file1); destination(d_file1); }; destination d_file2 { file ("/var/log/syslogs/file2-$YEAR$MONTH$DAY.log" owner(root) group(root) perm(0644) dir_perm(0755) create_dirs(yes)); }; filter f_file2 { facility(local2); }; log { source(remote); filter(f_file2); destination(d_file2); }; Thanks. - Young

3 3

Error binding socket;
by vinod samant 14 May '18

14 May '18

Hi I am getting below error while restarting server . [root@localhost logs]# /etc/init.d/syslog-ng restart Stopping syslog-ng: [FAILED] syslog-ng is stopped Starting syslog-ng: [2018-05-11T16:41:55.756615] Error binding socket; addr='AF_INET(0.0.0.0:514)', error='Address already in use (98)' [2018-05-11T16:41:55.756684] Error initializing message pipeline; plugin name='network', location='/etc/syslog-ng/syslog-ng.conf:24:3' [FAILED] My requirement :- OS:- centos 6.5 1. I want to use wildcard to send all apache custom logs from client to server . Configuration of both client and server :- >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>client conf<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< @version:3.13 @include "scl.conf" @define allow-config-dups 1 source s_local { system(); internal(); }; destination d_network {udp("192.168.122.67" port(514)); }; destination d_local { file("/var/log/messages"); }; log{ source(s_local); destination(d_network); destination(d_local); }; #################### source s_local{ wildcard-file( base-dir("/usr/local/apache/logs") filename-pattern("*.log") recursive(no) follow-freq(1)); }; destination d_network {tcp("192.168.122.67" port(514)); }; log{ source(s_local); destination(d_network); }; >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>server conf<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< @version:3.13 @include "scl.conf" source s_local{system(); internal();}; source s_network{ udp(); tcp(); }; destination d_local{file("/var/log/messages");}; destination d_from_net{file("/var/log/from_net");}; log{ source(s_local); destination(d_local); }; log { source(s_network); destination(d_from_net); }; source s_net { network(flags(no-parse)); }; parser p_apache { csv-parser( columns("apache.FILE_NAME", "apache.MESSAGE") flags(greedy) ); }; destination d_apache { file("/usr/local/apache/logs/${apache.FILE_NAME}" template("${apache.MESSAGE}\n")); }; log{ source (s_net); parser (p_apache); destination(d_apache); }; What should be changed on configuration file on both side ? *Thanks & Regards :-* *VINOD SINGH SAUD* *(M):- 09718663552* *(W):-09997645597* *(E) :- vinod.samant.123(a)gmail.com <vinod.samant.123(a)gmail.com>*

3 6

How to configure client and host server
by vinod samant 11 May '18

11 May '18

Hi , I am beginner in linux and first time i am going to installation and configure syslog-ng .So, I hope ,will get more help from your side because i am being confuse while reading documentation.first of all i want to mention my requirements. OS:- centos 6.9 My Requirements:- 1. we have 20 apache webserver ,we are writing apache custom logs or php applications logs in a particular directory ,But directory location is different different for both apache log and php application logs. For example we are writing log in "/usr/local/apache/logs " and apache log format or php application log format are mentioned below . *apche custom logs Fromta:- * abc_access_2018-05-09.log ,abc_error_2018-05-09.log and xyz_access_2018-05-09.log etc.. *Php application log format:- * hello_vidoes_20180509.txt , world_music_20180509.txt 2. I want to store these logs of 20 web servers on a centralised server at real time and we do not want to change logs file and format . Is this possible? 3. I want to make 20 web server as client server and centralised server (as host server). Here ,I can not understand from documentation ,So Can you provide the configuration for both side client and host side ? Client ip:- 192.168.122.184 server IP(centralised):- 192.168.122.61 *Thanks & Regards :-* *VINOD SINGH SAUD* *(M):- 09718663552* *(W):-09997645597* *(E) :- vinod.samant.123(a)gmail.com <vinod.samant.123(a)gmail.com>*

3 9

Best way to pipe "application" logs to central syslog-ng server.
by Delon Lee Di Lun 09 May '18

09 May '18

Hi, I am working on a mini project that requires the receipt of "application" logs using rsyslog client to pipe it to a syslog-ng central server. An example of the "application" logs im referring to would be for instance Apache HTTPD logs, I want to separate the "application" logs, in this example, the apache logs, and the "OS" logs into different directories. I am thinking of certain crude way, using regex to filter the messages at my server end ,to do it which might not be as clean. Would like to ask the floor if anybody had experience with working in this environment. Would change my rsyslog client and my syslog-ng server to use RFC5424, IETF's compare to the legacy BSD syslog protocol help? Comparing the headers there is a "app-name" variable in the IETF's syslog protocol I can use? Yours Sincerely, Delon Lee

5 12

Disk based queuing with rsyslog?
by Delon Lee Di Lun 09 May '18

09 May '18

Hi, I read that syslog-ng can be setup to use disk based with flow control with the client to prevent lost of logs. However I was just thinking, how does this mechanism works? Does the client have to be using syslog-ng? Will it work if the client is using rsyslog? Native appliance that purely support syslog? Yours Sincerely, Delon Lee

4 9

Difference in version
by Delon Lee Di Lun 08 May '18

08 May '18

Hi, I am trying to download the source code of syslog-ng 3.15.1 from the link below: https://github.com/balabit/syslog-ng/releases May I know what is the difference between the file with "orig" and the one without? Yours Sincerely, Delon

6 13

hostname not appearing correctly when receiving logs from switches
by Clayton Dukes 07 May '18

07 May '18

Interesting! We’ve been getting a lot of support tickets for this very problem. I can easily recreate the issue. Balabit Team: is this a new bug? [cid:image001.png@01D306E3.0FEBC990] Clayton Dukes Founder & CEO LogZilla Corporation 2900 N. Quinlan Park Rd, B240-341 Austin, TX, 78732 Tel: 936-4NetOps (463-8677) Web: www.logzilla.net<http://www.logzilla.net/> [cid:image002.png@01D306E3.0FEBC990]<https://twitter.com/logzilla>[cid:image003.png@01D306E3.0FEBC990]<https://youtu.be/drg5wv_mgfA>[cid:image004.png@01D306E3.0FEBC990]<https://www.linkedin.com/in/lzcdukes/> For NetOps, By NetOps! From: syslog-ng <syslog-ng-bounces(a)lists.balabit.hu> on behalf of Joshua <aces621(a)yahoo.com> Reply-To: Joshua <aces621(a)yahoo.com>, Syslog-ng users' and developers' mailing list <syslog-ng(a)lists.balabit.hu> Date: Monday, April 30, 2018 at 7:09 PM To: "syslog-ng(a)lists.balabit.hu" <syslog-ng(a)lists.balabit.hu> Subject: [Suspected Spam] [syslog-ng] hostname not appearing correctly when receiving logs from switches Hi All, I am pretty new to syslog-ng but do have some basic knowledge. I have deployed syslog-ng v3.14 on a newly deployed Linux server because syslog-ng v3.5 is working very well on another syslog server. On this new deployment, the syslogs received from most of the servers are able to show IP/host, however, the syslogs from our switches contains IP/host showing as ":" (colons). I copied the current working custom build .conf from another syslog server into our new server. Can someone help me figure out what I am missing? It is working for some components but not for switches. I tested the same switch by sending syslog to another syslog server and the hostname is appearing but just not appearing on the new syslog server. The only difference between the two server is that one uses v3.5 (the working one) and the other uses syslog-ng v3.14. I have set: "keep_hostname (yes)" but it still doesn't work. Can someone please help? Am I missing something here? Thanks Joshua Lai

5 5

save Syslog-ng daemon logs locally
by Hirbod Moriani 05 May '18

05 May '18

Hi everyone, I want to save log from syslog-ng daemon itself. For example "start a TLS session"/"end of a TLS session" or something like that should be logged. It's possible in syslog-ng CE? and how to config syslog-ng to save daemon logs locally?

2 1

Difference in 3.15.1 versions?
by Delon Lee Di Lun 05 May '18

05 May '18

Hi, I am trying to download the source code of syslog-ng 3.15.1 from the link below: https://github.com/balabit/syslog-ng/releases May I know what is the difference between the file with "orig" and the one without? Yours Sincerely, Delon

1 0

hostname not appearing correctly when receiving logs from switches
by Joshua 01 May '18

01 May '18

Hi All, I am pretty new to syslog-ng but do have some basic knowledge. I have deployed syslog-ng v3.14 on a newly deployed Linux server because syslog-ng v3.5 is working very well on another syslog server. On this new deployment, the syslogs received from most of the servers are able to show IP/host, however, the syslogs from our switches contains IP/host showing as ":" (colons). I copied the current working custom build .conf from another syslog server into our new server. Can someone help me figure out what I am missing? It is working for some components but not for switches. I tested the same switch by sending syslog to another syslog server and the hostname is appearing but just not appearing on the new syslog server. The only difference between the two server is that one uses v3.5 (the working one) and the other uses syslog-ng v3.14. I have set: "keep_hostname (yes)" but it still doesn't work. Can someone please help? Am I missing something here? ThanksJoshua Lai

1 0

Destination program input
by Scappatura Rocco 30 Apr '18

30 Apr '18

Hello. I have written a C program that read the successful POP3 access log lines to my POP3 server. After get one line, it get the time and the POP3 account that logged in. These data are then stored in the account MySQL database. The program (named 'pop3_access') works fine: echo "Apr 10 12:00:17 mail1 pop3d: LOGIN, user= xxx(a)domain.tld, ip=[::ffff:xxx.yyy.zzz.vvv], port=[56196]"|./pop3_access Infact quering DB, I get: username pop3_last_access xxx(a)domain.tld 2018-04-10 12:00:17 I have configured syslog: filter f_pop3_access { facility(local7) and match ('LOGIN,'); }; destination d_pop3_last_access { program("/usr/local/bin/pop3_access >/dev/null" ); }; log { source(src); filter(f_pop3_access); destination(d_pop3_last_access); }; But no rows are inserted in the database.. >From log of destination program 'pop3_access' I see the it remains in a while loop waiting for an 'EOF' that never arrives: while (fgets(line, MAXSTR, stdin)) { if (sscanf(line, "%s %d %s %s %s %s %s %s %s", mese, &giorno, ora, s, proto, comando, u, ip, porta) != 9){ syslog (LOG_NOTICE, "Errore scanf"); exit(-1); } } How I have to do so that my program succeed in taking the syslog row correctly, and then returns after process the data as expected? Regards, Rocco Scappatura

3 5

insert timezone into log
by Asif Iqbal 26 Apr '18

26 Apr '18

How do I change the time in the log to append a timezone. Timestamp should be based on generateTime=1523620861 which is in the log So today the time in the shows like this Apr 25 16:46:51 host.example.net .., generateTime=1524674663, ... I like to change it like below 2018-04-25T16:46:55+0000 host.example.net ...,generateTime=1524674663, .... {"PROGRAM":"alarmLog,","PRIORITY":"notice","MESSAGE":"applianceName=KING-MER-50-PRI, tenantName=king, alarmType=vrrp-v3-proto-error, alarmKey=0|vni-0/3.0, generateTime=1523620861, applianceId=1, vsnId=0, tenantId=4, alarmCause=causeOther, alarmClearable=no, alarmClass=new, alarmKind=root, alarmEventType=equipmentAlarm, alarmSeverity=indeterminate, alarmOwner=tenant, alarmSeqNo=36657, alarmText=\"vni-0/3.0\", siteName=","HOST":"host.example.net","FACILITY":"user","DATE":"Apr 13 04:01:22"} -- Asif Iqbal PGP Key: 0xE62693C5 KeyServer: pgp.mit.edu A: Because it messes up the order in which people normally read text. Q: Why is top-posting such a bad thing?

2 4

GSoC 2018 participant: Queue Interface
by Mehul Prajapati 25 Apr '18

25 Apr '18

Hi All, I'm Mehul Prajapati, a prospective student in Masters of Applied Computer Science graduate program at Concordia University. In my spare time, I do competitive programming on HackerRank, online computer science courses (edx, Coursera, Udacity) and open source projects to improve myself as a software developer. I am going to work on the Queue Interface project using C. The aim of this project is to develop a Redis based queue interface which could be an alternative to the disk queue. Many thanks to Mr. Kókai Péter for selecting me. I am excited to be a part of this community. Looking forward to work on this project. -- Regards, Mehul Prajapati

1 0

GSoC 2018 participant : ElasticSearch Destination Client
by Videet Singhai 25 Apr '18

25 Apr '18

Hello all, I am Videet Singhai from India, currently pursuing Bachelor of Engineering in Information Technology. I am going to work on the ElasticSearch Destination REST Client using C. This client should be able to send the logs to ElasticSearch without depending on other languages. Thanks to Mr. Laszlo Budai for selecting me. Looking forward to having an amazing summer. Thank you

1 0

my syslog-ng-githead repo now depends on Java 1.8
by Czanik, Péter 25 Apr '18

25 Apr '18

Hi, Recently I updated my experimental syslog-ng repo built from git head for RHEL/CentOS 7 & Fedora to use openjdk 1.8 instead of 1.7, as it might support Oracle Java packages better (no need to install both openjdk and Oracle java). @Evan: please give it a try, if it fixes your problem: https://copr.fedorainfracloud.org/coprs/czanik/syslog-ng-githead/ @everyone: would you mind, if I update the dependency also for my unofficial packages for the next release? (3.15) As far as I can see, it's not openjdk 1.7 but 1.8 which is installed by default on CentOS 7. Bye, Peter Czanik (CzP) <peter.czanik(a)balabit.com> Balabit / syslog-ng upstream https://syslog-ng.com/blog/author/peterczanik/ https://twitter.com/PCzanik

1 0

Bug report: syslog-ng requests too many capabilities
by Russenberger Dominik 24 Apr '18

24 Apr '18

Hi List, I recently noticed something very strange: although I run syslog-ng as an unprivileged user (with -u log -g log), newly created logfiles were owned by root. syslog-ng shows up running as user log in ps, as expected. The culprit turned out to be linux capabilities. No problem I thought, I just add a "--caps 'CAP_NET_BIND_SERVICE=epi'" command line parameter (binding ports <1024 is the only capability I need). But then I was getting flooded by messages like this: syslog-ng[25657]: Error managing capability set, cap_set_proc returned an error; caps='= cap_net_bind_service+eip cap_fowner+e', error='Operation not permitted (1)' In my opinion, there are 2 bugs in syslog-ng: * if I tell a daemon to run as unprivileged user I do not expect it to write files as user root. What syslog-ng is doing basically is faking being an unprivileged user, while retaining capabilities which are equivalent to full root permissions. Syslog-ng should imho either run as root, with capabilities; OR as unprivileged user without capabilities (except those explicitly given in --caps) * syslog-ng drops to the capabilities it gets told in --caps, but later g_process_cap_modify() ignores what was specified. Some notes about my environment: - the OS is an Ubuntu remix. I'm recompiling syslog-ng 3.11.1 from Ubuntu Artful, with some changed configure flags (see below) - there are multiple syslog-ng instances running. One for the system itself (as root), and multiple other syslogs receiving incoming logs over udp/tcp (running as user log) My syslog-ng version: syslog-ng 3 (3.11.1) Installer-Version: 3.11.1 Revision: Compile-Date: Apr 10 2017 14:06:00 Module-Directory: /usr/lib/syslog-ng/3.11 Module-Path: /usr/lib/syslog-ng/3.11 Available-Modules: disk-buffer,afsocket,afprog,system-source,kvformat,cef,affile,date,cryptofuncs,basicfuncs,confgen,pseudofile,csvparser,afuser,syslogformat,dbparser,linux-kmsg-format Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-IPv6: on Enable-Spoof-Source: off Enable-TCP-Wrapper: off Enable-Linux-Caps: on Enable-Systemd: off Thanks, Dominik

4 4

Need clarification on different behavior of forwarding rules in syslog-ng
by Kavita Mohite (kmohite) 18 Apr '18

18 Apr '18

Hello Support Team, In the syslog-ng configuration, we have 2 forward rules as follows: 1. One rule reads from source (port 514) and writes syslogs to the file for all the syslogs greater than debug. 2. Second rule reads from source (port 514) and forwards the the syslog to the destination if a set of filter rules match. These set of filter rules are called whitelist filter rules are filter rules containing match filter and priority filter in each rule. As there are multiple filter rules, they are ORed to create a main filter rule. Now we have syslogs coming in at a rate of 400 eps. Following is our observation: A. If the syslog matches the filter in point 2 above, then following happens · It is forwarded to the destination provided in point 2. · Also at the same time it is written to a file as per rule in point 1. B. If the syslog does not match the filter in point 2 above, then following happens · It is NOT forwarded to the destination provided in point 2 as there is no match. · It is suppose to still write to file as the rule in 1 matches. But this does not happen. We are seeing that if the filter in point 2 matches then both the forward rules get executed immediately and we see the log written to file also immediately. But in case of B, if the syslog does not match we are seeing that the write to file is happening g in bulk for 30 minutes. Why is this ? Why does it not write to file immediately in case of (B) but does it in case of A ? Below is the snap of the config values we are using in the syslog-ng.conf options { flush_lines (100); log_fetch_limit(1000); log_iw_size(10000); time_reopen (10); log_fifo_size (10000); chain_hostnames (off); use_dns (no); use_fqdn (no); create_dirs (yes); keep_hostname (yes); threaded(yes); }; Thanks Kavita

2 1

Need clarification on different behavior of forwarding rules in syslog-ng
by Kavita Salve 18 Apr '18

18 Apr '18

Hello Support Team, In the syslog-ng configuration, we have 2 forward rules as follows: 1. One rule reads from source (port 514) and writes syslogs to the file for all the syslogs greater than debug. 2. Second rule reads from source (port 514) and forwards the the syslog to the destination if a set of filter rules match. These set of filter rules are called whitelist filter rules are filter rules containing match filter and priority filter in each rule. As there are multiple filter rules, they are ORed to create a main filter rule. Now we have syslogs coming in at a rate of 400 eps. Following is our observation: A. If the syslog matches the filter in point 2 above, then following happens - It is forwarded to the destination provided in point 2. - Also at the same time it is written to a file as per rule in point 1. B. If the syslog does not match the filter in point 2 above, then following happens - It is NOT forwarded to the destination provided in point 2 as there is no match. - It is suppose to still write to file as the rule in 1 matches. *But this does not happen.* We are seeing that if the filter in point 2 matches then both the forward rules get executed immediately and we see the log written to file also immediately. But in case of B, if the syslog does not match we are seeing that the write to file is happening g in bulk for 30 minutes. *Why is this ? Why does it not write to file immediately in case of (B) but does it in case of A ?* Below is the snap of the config values we are using in the syslog-ng.conf options { flush_lines (100); log_fetch_limit(1000); log_iw_size(10000); time_reopen (10); log_fifo_size (10000); chain_hostnames (off); use_dns (no); use_fqdn (no); create_dirs (yes); keep_hostname (yes); threaded(yes); }; Thanks Kavita Mohite

1 1

Re: [syslog-ng] Support for Open Source Syslog-ng
by Naveen Bhalla (nbhalla) 14 Apr '18

14 Apr '18

Team, Could you pls help us on below P1 situation for us? Regards, [http://wwwin.cisco.com/c/dam/cec/organizations/gmcc/services-tools/signatur…] Naveen Bhalla | Manager.Technical Support CMS Platform Operations Cell: +91-9880362157 Desk: +91-80-44260795 From: Naveen Bhalla (nbhalla) Sent: 13 April 2018 09:14 PM To: 'support(a)balabit.com' <support(a)balabit.com<mailto:support@balabit.com>> Subject: Support for Open Source Syslog-ng Hello Support Team, We have a situation in our platform where syslog-ng is dropping part of syslog traffic coming into our server. The syslog-ng has around 750 match rules in its configuration. Based on these rules the syslogs are forwarded to the destinations. Also, there is one rule to write all the received syslogs to disk. We are receiving syslogs at the rate of 300 eps. The issue is that we are seeing that syslog-ng is not able to process the syslogs and forward them to the destinations. It is not writing to the disk also. We are seeing that there is a big delay after which some syslogs are getting written to the disk. We are seeing loss of UDP packets. The UDP buffer size is big enough. net.ipv4.tcp_rmem = 4096 4194304 16777216 net.ipv4.tcp_wmem = 98304 4194304 16777216 net.core.rmem_default = 234217728 net.core.wmem_default = 234217728 net.core.rmem_max = 234217728 net.core.wmem_max = 234217728 net.ipv4.tcp_window_scaling = 1 net.ipv4.ip_local_port_range = 32768 61000 fs.file-max = 2097152 net.core.optmem_max = 40960 net.core.netdev_max_backlog = 50000 net.ipv4.udp_rmem_min = 8192 net.ipv4.udp_wmem_min = 8192 We need help to resolve this issue. We are using open-source syslog-ng in our setup. Regards, [http://wwwin.cisco.com/c/dam/cec/organizations/gmcc/services-tools/signatur…] Naveen Bhalla | Manager.Technical Support CMS Platform Operations Cell: +91-9880362157 Desk: +91-80-44260795

3 2

Insider 2018-04: RHEL6; Windows Event Log; Patched RPM; HEC; Conferences;
by Czanik, Péter 12 Apr '18

12 Apr '18

Dear syslog-ng users, This is the 66th issue of syslog-ng Insider, a monthly newsletter that brings you syslog-ng-related news. NEWS syslog-ng is available for RHEL 6 / CentOS 6 again -------------------------------------------------- After more than a year, the latest version of syslog-ng compiles again on Red Hat Enterprise Linux version 6. This is thanks to a patch from Balázs Scheidler which resolves compatibility problems with older glib releases by implementing the missing functions within syslog-ng. https://syslog-ng.com/blog/latest-syslog-ng-available-rhel-6-centos-6/ Agentless Windows logging ------------------------- With syslog-ng Premium Edition, you can collect Windows event logs without installing any third-party application on your Windows-based computer. Learn more about it from our blog at https://syslog-ng.com/blog/how-to-collect-windows-event-logs-with-syslog-ng… or join our webinar where you can also ask questions: https://www.brighttalk.com/webcast/16207/314789?utm_source=OSE/ Build your own syslog-ng RPM ---------------------------- When a user reports a problem with syslog-ng, developers create code to fix it. The fix is called a patch or a pull request (PR) in Git terminology. Often this code does not enter the main source code automatically but developers ask the user to test the fix before the code is merged. A new challenge starts here: figuring out how to test a proposed fix. This blog shows you how to download and patch the syslog-ng source code and build easy-to-install (and remove) RPM packages from it. https://syslog-ng.com/blog/build-syslog-ng-rpm-patched-git-sources-rhel-cen… While this blog introduces you to package building in depth, a lot simpler method is also available for Ubuntu packages and should be working soon again for RHEL / CentOS as well: https://github.com/balabit/syslog-ng/tree/master/dbld Sending logs to Splunk HEC -------------------------- Recently Splunk started to recommend the use of the HTTP Event Collector (HEC) instead of forwarders. syslog-ng supports this in multiple ways. In a previous blog, we showed you how to use the http() destination of syslog-ng. This time, we introduce you to another possibility: using an external python script to send logs to HEC. Read our blog at https://syslog-ng.com/blog/splunk-hec-sending-logs-using-program-destinatio… or watch the webinar at https://www.brighttalk.com/webcast/16207/308061 Conferences: CentOS Dojo, FOSDEM and SCALE ------------------------------------------ Recently syslog-ng was presented at some popular open source conferences. Read about the presentations and experiences in related blogs: CentOS Dojo & FOSDEM: https://syslog-ng.com/blog/centos-dojo-fosdem-2018/ SCALE: https://syslog-ng.com/blog/syslog-ng-scale-2018/ UPCOMING EVENTS LOADays: http://loadays.org/ (Antwerp) Red Hat Summit: https://www.redhat.com/en/summit/2018 (San Francisco) Your feedback and news, or tips about the next issue are welcome at documentation(a)balabit.com. To read this newsletter online, visit: https://syslog-ng.com/blog/ Peter Czanik (CzP) <peter.czanik(a)balabit.com> Balabit / syslog-ng upstream https://syslog-ng.com/blog/author/peterczanik/ https://twitter.com/PCzanik

1 0