[syslog-ng] Processing orphaned disk queue files

László Várady annonameus at gmail.com
Mon Mar 6 21:31:20 UTC 2023


You are probably looking for the `dqtool assign` command in syslog-ng OSE.
Please note that this command has a bug that makes it inoperable, so you 
can only use it properly if you upgrade to the upcoming syslog-ng 4.1.0 

Let me share a quick tutorial.

1. Prepare a destination with the appropriate disk-buffer type 

destination {
     network("localhost" port(4444) disk-buffer(reliable(yes) 
     http(url("https://test.tld/logs") workers(3) 
disk-buffer(reliable(yes) disk-buf-size(10G)));

2. Start and stop syslog-ng to generate persist entries for the your 

$ bin/persist-tool dump var/syslog-ng.persist

http(https://test.tld/logs,).1.queue = { "value": "..." }
http(https://test.tld/logs,).2.queue = { "value": "..." }
http(https://test.tld/logs,) = { "value": "..." }
afsocket_dd_qfile(stream,localhost:4444) = { "value": "..." }

3. Find the name of the destination into which you want to inject the 
orphaned queue file. For example: "afsocket_dd_qfile(stream,localhost:4444)"

4. Inject the queue file:

$ bin/dqtool assign -p var/syslog-ng.persist -n 
"afsocket_dd_qfile(stream,localhost:4444)" orphaned/syslog-ng-00001.rqf

You can now start syslog-ng and monitor its "queued" counters with 
`syslog-ng-ctl stats` to make sure the queue file is drained.


Alternatively, you can use a source called "example-diskq-source", if 
you are okay with trying experimental features.
In that case, you need to remove your disk-buffer declaration from your 
destination, and declare the diskq-source right before
the destionation:

source {
destination {
     network("localhost" port(4444));


The third and the less user-friendly alternative is using `persist-tool 

$ bin/persist-tool dump var/syslog-ng.persist > dump.txt

You can edit the dump file manually (the hex values represent the 
disk-buffer file name in ASCII, you can replace them),
and then you can reimport the edited dump:

$ bin/persist-tool add dump.txt -o var/

László Várady

On 2023. 03. 06. 18:31, Steve Bernacki wrote:
> Hello,
> I am using syslog-ng OSE 3.29. I have two destinations that I recently 
> redefined in order to spread load out across a number of receivers. 
> However, having done this, I now have orphaned queue files of 
> considerable size that I'd like to process through.
> I found one reference on how to do this with syslog-ng PE which is 
> documented at 
> https://support-public.cfm.quest.com/46246_syslog-ng-pe-diskbuffer-recovery.pdf, 
> but I'm finding the OSE tools are acting slightly differently, for 
> example persist-tool dump prints its output as hex instead of ASCII, 
> as is documented in the PDF I linked to above. In addition, the 
> process is fairly involved, which makes me nervous to experiment with 
> this.
> Looking at the command output for "dqtool --help", I see there are a 
> number of commands listed that are not documented anywhere, like 
> "relocate" and "assign", which look like they may be helpful to me. 
> Does anyone know if these can be used to help me process through these 
> logs?
> Thanks
> Steve
> ______________________________________________________________________________ 
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation: 
> http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq

More information about the syslog-ng mailing list